READ 180 Universal
Study: Sprague et al. (2011)

Describe how students were selected to participate in the study:

Within the five high schools participating in the study, all incoming ninth-grade students identified as struggling readers—as measured by the Scholastic Reading Inventory—were included in the pool for random assignment.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The Scholastic Reading Inventory (SRI) was used to identify students as “at risk for academic failure.” The SRI has overlapping Lexile levels and, as a result, the range for identifying eligible students had to be established (therefore, the 50th Normal Curve Equivalency or NCE was used as the benchmark).

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

READ 180, an intensive literacy curriculum developed for struggling readers in grades 4 through 12, is the treatment program.

Specify which condition is the control condition:

In addition to READ 180, there was one additional targeted intervention, Strategic Instruction Model (SIM) Xtreme Reading, and one whole-school intervention, Strategic Instruction Model Content Enhancement Routines for Teachers (SIM-CERT). The Xtreme Reading Program was developed for adolescents who struggle with reading and writing. SIM-CERT provides reading strategies to improve literacy instruction across all disciplines. The two components of “business as usual” and therefore considered the control condition included (1) the supplemental services ordinarily available to students in need of additional reading support and (2) the standard ELA courses for all students inclusive of any normally provided reading instruction.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

All incoming ninth-grade students identified as struggling readers based on the SRI were included in the pool for random assignment. Pre-randomization blocking of students (by special education and ELL status) was employed where numbers permitted, to ensure the similarity of students across groups on observable characteristics relevant to the outcome and to increase the precision of impact estimates. Following the receipt of SRI scores, evaluators randomly assigned students to one of the targeted interventions or the control group.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

In Year 4, READ 180 teachers had on average, 10 years of teaching experience. They had worked at their current school for three years, on average. READ 180 teachers were supposed to receive a variety of professional development opportunities and support, ranging from trainings, seminars, in-classroom support, web-based instructional support, and online RED courses focused on aspects of reading instruction. All READ 180 teachers, except one, received all professional development as defined by the developer.

Describe when and how fidelity of treatment information was obtained.

Classroom observations were conducted by evaluators twice during the school year—in February and May—in order to collect data on classroom-level implementation. Two observations were used to increase reliability (an over 85% rate of item-level agreement). Secondary data and extant documents provided by districts to document their implementation efforts were collected for analysis by evaluators. Miscellaneous documents reviewed included developer materials, professional development agendas, meeting minutes, memoranda, written curricula, and course syllabi. In addition, many meetings were held with the districts as well as clarifications made after meetings and receipt of data. Documents were used to corroborate findings and triangulate data for reporting verification.

What were the results on the fidelity-of-treatment implementation measure?

An overall implementation rating involved compiling the primary component ratings by teacher and indicating the numbers of teachers achieving the highest level (adequacy). The overall ratings for inputs consisted of three primary components: (1) professional development participation, (2) provision of materials/technology/assessments, and (3) classroom organization/structure. The overall classroom model rating, as a primary component itself, consisted of the four subcomponents: (1) instructional practices including use of structured content, research-based instructional methods, and responsive teaching; (2) dosage, including use of rotations, pacing for the year, and amount of instructional time; (3) use of materials and/or technology; and (4) use of assessments to inform instruction. Summary input and classroom model ratings were created by averaging to calculate overall implementation percentages and associated implementation levels: 1 = no evidence (0–24%); 2 = low (25–49%); 3 = moderate (50–74%); and 4 = adequate or high (75–100%). For the inputs, four of the five READ 180 teachers received ratings of adequate or high, indicating that the professional development, materials, and classroom structure required for implementation had been provided for the majority of teachers. The remaining teacher received a rating of moderate, consistent with the prior year, indicating that most but not all inputs were provided. All teachers indicated they had enough teacher materials and were provided with the required 90 minute daily class period; the teacher with the moderate score, according to district records, did not receive the prescribed amount of professional development. For the classroom model, two of the five READ 180 teachers received a rating of adequate or high, indicating fidelity of implementation as defined was achieved. The remaining READ 180 teachers (three of the five) were implementing with moderate fidelity. Teachers received moderate scores rather than adequate because they were observed to be behind schedule as per the pacing calendar and did not devote the full 90 minute class period to READ 180 instruction.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

The analysis was designed to estimate the impact of the intervention by comparing the achievement scores of the treatment group on average to that of the control group. Using SDRT-4 scores from the spring of the ninth-grade year, student performance in reading for the treatment group was compared with the control group. ANCOVA models were fit for these analyses allowing the effects of participation in the intervention to be assessed in the model. Treatment effect size estimates were calculated and average achievement across schools. The model for this cross-sectional analysis of the impact of the targeted intervention was specified with fixed effects for schools. In other words, the overall impact of each targeted intervention was estimated as a treatment effect averaged across schools.