Reading Recovery
Study: May et al. (2016)

Describe how students were selected to participate in the study:

Students in the 1,254 schools that participated in the RCT were identified to be part of the experiment via the following screening process: Reading Recovery teachers first screened a pool of candidates for Reading Recovery intervention using the Observation Survey of Early Literacy Achievement (OS) (Clay, 2013). Candidates included 1st-grade students who were identified by school staff—including kindergarten, 1st-grade, and intervention teachers—as struggling readers.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Once schools had identified eight 1st-grade students with the lowest OS scores, Reading Recovery teachers entered the names of the selected students into an online random assignment tool, noting their English language learner (ELL) status and their baseline OS Text Reading Level (TRL) subtest scores. The tool then matched them into pairs by first matching any students with ELL designations, then matching the student with the lowest TRL subtest score with the next-lowest student, and so on. Once the students were matched, a randomizing algorithm then randomly assigned one student in each pair to the treatment group and the other to the control group. The result was recorded in IDEC, and the tool was locked so that randomization in that school could not be redone.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

0.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Reading Recovery

Specify which condition is the control condition:

The study was designed as a delayed-treatment RCT; control students began receiving their Reading Recovery lessons after the posttests were administered to both treatment and control students in each matched pair. The ‘‘blocking’’ of students in matched pairs was a central part of the study design that produced, essentially, four mini-experiments per school.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

The study was designed as a delayed-treatment RCT, control students began receiving their Reading Recovery lessons after the posttests were administered to both treatment and control students in each matched pair. The ‘‘blocking’’ of students in matched pairs was a central part of the study design that produced, essentially, four mini-experiments per school. A total of 4,892 students were randomized to treatment, and 4,892 to control. Both pretest and posttest data were available for 7,855 of these students (4,136 treatment; 3,719 control). Pairs in which either student was missing assessment data were dropped from the RCT, leaving a total of 6,888 students who were able to be matched into pairs with complete data (3,444 matched pairs in 1,122 schools).

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Teachers who deliver the Reading Recovery intervention must complete an intensive, year-long graduate-level training course taught by a literacy coach called a teacher leader. Through this training and ongoing professional development, teachers develop expertise at analyzing students’ literacy behaviors, identifying learning needs, and delivering responsive instruction. Reading Recovery theory asserts that the role of the teacher is to identify students’ strengths and needs, and to facilitate his or her learning by providing appropriate opportunities to acquire and practice new skills. Other key roles in Reading Recovery implementation include teacher leaders, who train and support Reading Recovery teachers, and Reading Recovery trainers, who are faculty members at OSU and its 19 partner universities around the U.S. Each partner university operates a University Training Center (UTC) that serves as a regional hub of Reading Recovery training and activity. University-based trainers support and train teacher leaders, and oversee the program’s operations within their areas.

Describe when and how fidelity of treatment information was obtained.

Within the evaluation report, "Chapter 4, Reading Recovery Implementation Fidelity: Scale-up Successes and Challenges" focuses on the results for implementation fidelity under the scale-up, and describes the methods we used to obtain them. We report that, with very few exceptions, strong fidelity to the program model was seen in all four years of the evaluation (see pages 69-82).

What were the results on the fidelity-of-treatment implementation measure?

CPRE/CRESP’s evaluation of implementation fidelity represents a rigorous, sustained, systematic, and successful attempt to measure and assess wide-ranging implementation activities. While our measurement was done via statistical analysis of survey data, the qualitative inquiry served both to corroborate the survey findings and to shed light on the few departures from fidelity that were observed. A consistent finding is that Reading Recovery was implemented with high fidelity to the program model over the course of the i3 scale-up. We found that the intervention was delivered as designed to the students in the scale-up, and that teachers delivering Reading Recovery lessons were properly trained. All in all, this finding suggests that the impacts we observed were, indeed, the result of faithful implementation of the intervention, and that the i3 scale-up successfully replicated Reading Recovery in the schools involved in the expansion. (p. 82)

Was the fidelity measure also used in control classrooms?

CPRE/CRESP collected survey data from 1st-grade teachers in the schools participating in the scale-up in order to understand the services provided to control students while the treatment students received Reading Recovery. In addition, we surveyed Reading Recovery teachers in order to assess the extent of their instructional contact with control students during the experiment. This is especially important in this study because Reading Recovery teachers spend half of their time engaging in instructional activities outside of their Reading Recovery lessons (e.g., small-group work, co-teaching, push-in support). Control group students were allowed to receive any available supports or intervention services other than Reading Recovery, therefore we wanted to understand the extent to which control students were instructed by Reading Recovery teachers during the experiment. Survey data were collected from each Reading Recovery teacher in an RCT school on the precise types of interaction (whole-class, small-group, individualized), frequency of interaction (from never to daily), and the timing of interaction they had with the control group students (during or after the experiment). We were able to collect information on 1,177 (54 percent) of all 2,171 students assigned to the control group in Years Two and Three of the RCT. We found that 8 percent of the control group students had some exposure to their Reading Recovery teacher in a whole-class setting; 34 percent had exposure in a small-group setting; and 10 percent of the control group students received individualized support from the Reading Recovery teacher during the experiment. Individualized instructional interaction with a Reading Recovery teacher has the highest potential for imitating the intervention. It does not represent contamination since it does not involve one-to-one Reading Recovery lessons. Although the rate of one-to-one instructional contact is low, overall, the fact that nearly 40 percent of the control students were taught by a Reading Recovery teacher during the experiment may have caused some attenuation of the estimated impacts. (p. 36)

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

The RCT addressed the following research objectives: 1. Estimate Reading Recovery’s impact on the early literacy competence of 1st-grade students, as measured by the ITBS; 2. Estimate the impacts of Reading Recovery on subscales of the ITBS as well as the OS, an aligned instrument used for Reading Recovery screening and monitoring; and 3. Estimate the impacts of Reading Recovery on two subgroups of particular interest Impacts on student reading performance were estimated in all analyses by comparing immediate post-intervention reading achievement of students randomly assigned to participate in Reading Recovery at the beginning of first grade to students randomly assigned to the control condition. We used a three-level hierarchical linear model (HLM) (Raudenbush & Bryk, 2002) with students nested within matched pairs, and matched pairs nested within schools. The multi-site, matched-pairs design of this random assignment study means that each school and each pair is an independent mini-experiment, and that the estimated causal impact is less prone to problems associated with attrition through student and school non-participation. Differences in the posttest performance of the treatment and control students were estimated after controlling for pretest performance. For the confirmatory analysis of Reading Recovery’s impacts on the ITBS Reading Total scores for the full sample as well exploratory analyses of ELL and rural subgroup impacts and impacts on ITBS subscales, this HLM included each TRL score as a covariate. For exploratory analyses of impacts on the OS, the OS Total score was used as the covariate. All models also included a binary indicator of treatment condition, a four-level fixed effect for year, an interaction effect for treatment by year, a random effect for matched pair, a random effect for overall school performance (i.e., school intercepts), and a random effect for the impact of Reading Recovery (i.e., school treatment effects). A completely general (unstructured) covariance matrix was used, which included a correlation between random effects for school-level intercept and slope. Additionally, a grouped residual variance was included to account for differences in dispersion of outcome scores within the treatment versus control groups. Models were estimated using PROC MIXED in SAS 9.3 via Restricted Maximum Likelihood (REML), with model-based standard errors and degrees of freedom based on within- and between-cluster sample sizes. (pp. 37-38)