READ 180 Universal
Study: Hamilton et al. (2011)

Describe how students were selected to participate in the study:

For students to be eligible, they had to be enrolled in grades 6, 7, or 8 in one of the 19 participating middle schools in Newark NJ. In addition, students had to be identified as a struggling reader. This identification was based on the students’ spring (pretest) reading score of the state assessment, the New Jersey Assessment of Skills and Knowledge (NJASK). Students with scores one standard deviation or more below the norm were eligible. (Citation; pages 5-7.)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students scoring below 1 standard deviation of the district norm on the reading subtest of the state assessment were identified as being at risk for academic failure. This criterion resulted in cutoff scores that ensured that all students were identified by the state as “partially proficient”, which is the lowest category possible on the state assessment.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Read 180 (Enterprise Edition) is the treatment for this evaluation.

Specify which condition is the control condition:

The control condition implemented the “business as usual” district language arts curriculum. The control condition used the New Jersey Core Content Curriculum Standards for literacy instruction. (Citation: pages 1-3)

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

The schools eligible to participate in the Striving Readers program were randomly assigned to either the intervention or a control condition in May 2006 (see Section 1.C.1.2 for school eligibility requirements). The targeted evaluation is, therefore, a randomized cluster design; no classroom- or student-level random assignment was involved. Although randomly assigning students would be the most statistically efficient design, it was not feasible for this study. One of the main constraints was the cost of implementation, which is largely determined by the number of participating schools. Additionally, there are contamination and spillover effects associated with student-level randomization. For example, teachers are likely to be aware that a colleague is delivering a special intervention, and this awareness may influence their behavior. Additionally, intervention and nonintervention students would interact, possibly closing the gap between their differences. In either case, the impact estimates would be biased toward zero.

What was the unit of assignment?

Schools

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Teachers were provided with training on the use of the curriculum. In addition, teachers received ongoing coaching and feedback, related to fidelity and quality of implementation, by trained district resource teacher coordinators.

Describe when and how fidelity of treatment information was obtained.

No classroom observations were conducted. Extant data were used to measure fidelity of implementation. Four components were measured: (1) teacher attendance at training, (2) class size, (3) the frequency of ongoing assessment (number of SRI administrations), and (4) the duration and frequency of use of the instructional software. Data for the first two components were obtained from district records. Data for the last two components were obtained from software usage reports. Because these data were extant, the calculation of inter-rater reliability is impossible (i.e. multiple raters were not used). Moreover the use of extant data provides objective and reliable indicators of fidelity.

What were the results on the fidelity-of-treatment implementation measure?

Each treatment classroom (n=21) receives a fidelity rating for each of the four components described above. The fidelity ratings range from 1 (low fidelity) to 4 (high fidelity). These components are then combined into an overall school-level fidelity score. Data for each fidelity component is provided below. 1. Teacher attendance at training- 40% of new teachers in year 4 were categorized as having ‘full participation’. The remaining 60% of teachers had ‘moderate participation’ (Note: there are only 3 categories for fidelity in this component due to the maximum number of training days) 2. 100% of treatment classrooms had fewer than 21 students, resulting in a fidelity level of ‘high’ for this component for all classrooms. 3. 100% of treatment classrooms administered a minimum of 3 SRI assessments during year 4, resulting in a fidelity level of ‘high’ in this component for all classrooms. 4. The duration of use of instructional software has two sub-components; the number of sessions per week, and the duration (in minutes) of each session. Software must be used a minimum of three times a week for 15 minutes each time for a classroom to receive a ‘high’ fidelity rating. 71% of classrooms received a ‘low’ rating for this component. 19% received a ‘low-to-moderate’ rating. 1% received a ‘moderate-to-high’ rating. No classroom received a ‘high’ fidelity rating. For the school-level overall fidelity scores, 40% of schools received a ‘high’ rating and the rest received ‘moderate-to-high’.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

To determine the effect of READ 180, an intent-to-treat (ITT) analysis was conducted based on repeated cross-sectional data, using a multilevel software package (HLM). A linear two-level model was used, with student and school as the two levels. Achievement for students within schools was predicted by a series of student and school characteristics. Student covariates were fixed across schools with no interactions (p. 53)