Reading Recovery

Study: May, Sirinides, Gray, & Goldsworthy (2016)

May, H., Sirinides, P., Gray, A., & Goldsworthy, H. (2016). Reading Recovery: An evaluation of the four-year i3 scale-up. Retrieved from: http://www.cpre.org/reading-recovery-evaluation-four-year-i3-scale
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

Reading Recovery is a highly effective, short-term intervention of daily one-to-one lessons that supplement good classroom teaching for the lowest-achieving first graders. The goal is to dramatically reduce the number of first- grade children with extreme difficulty learning to read and write and to reduce the cost of these learners to educational systems.

The lowest literacy achievers in Grade 1 receive a half-hour lesson each school day for 12-20 weeks with a specially trained Reading Recovery teacher. Daily and weekly records provide rich information for progress monitoring. As soon as children meet grade-level expectations and can continue to learn in the classroom, their lessons are discontinued, and new students begin individual instruction. Just as in the intent of RTI, two positive outcomes are possible: (a) responding to the intervention by meeting grade-level literacy expectations (approximately 75% of the children complete interventions) and (b) recommended for further evaluation and future support.

The intent of Reading Recovery lessons is to promote accelerated learning so that students can catch up with their peers and benefit from classroom instruction without supplemental help. Each lesson is individually designed and individually delivered by specially trained teachers and includes: reading familiar books, reading yesterday’s new book while the teacher takes a running record, working with letters and/or words using magnetic letters, composing a written message, assembling a cut-up story, and reading a new book.

Extensive training is required for the responsive, contingent teaching in Reading Recovery. Three levels of training form a network of expertise and support: (a) university-based trainers of teacher leaders; (b) site-based teacher leaders who train teachers; and (c) school-based teachers who deliver the intervention on their campuses.

Reading Recovery is intended for use in first grade. It is designed for any student at risk of academic failure. The academic area of focus is reading (including phonological awareness, phonics/word study, comprehension, fluency, and vocabulary) and writing (including spelling, sentence construction, phonemic awareness, letter-sound relationships, phonological and orthographic awareness, analogy, and writing vocabulary).

Reading Recovery has been implemented in the United States for 25 years. During the 2008-2009 school year, 1,909 school systems in 47 states were implementing Reading Recovery and 34 school systems in 8 states implemented Descubriendo la Lectura.

Where to obtain:
Reading Recovery Council of North America
500 W. Wilson Bridge Rd., Suite 250
Worthington, Ohio 43085-5218
Phone: 614-310-7323
Website: www.readingrecovery.org

Also available at University Training Centers: http://www.readingrecovery.org/development/centers/index.asp
          
Cost:
Initial per teacher costs for a new implementation in a school would include the cost of the year-long training (costs vary according to universities granting credit), the initial purchase of non-consumable children’s books to be used with students (about $2,500), professional books, materials, and supplies (approximately $250), and data processing fees (about $57).

Replacement cost per student for subsequent use: The cost of Reading Recovery per trained teacher is dramatically reduced after the training year. The only replacement costs are optional additions to the non-consumable set of books used with children and possible non-consumable professional books for teachers. Expenditures to the district/site include ongoing professional development, data processing fees, university affiliation fees if applicable, and miscellaneous supplies.

Reading Recovery is designed for use with individual students.

Reading Recovery takes 30 minutes per session with a recommended five sessions per week for 15-20 weeks.

The program includes a highly specified manual of teaching procedures from which teachers select to meet needs of each individual child. Technology is not required for the intervention with children. However, required data are sent electronically to the International Data Evaluation Center (IDEC).

An academic year of training with university credit is required for instructors.

For an academic year, teachers meet weekly with a registered Reading Recovery teacher leader who has completed a full year of training at a university training center to prepare for training teachers. While in training, teachers are also working with Reading Recovery students. Teachers learn to observe and discuss lessons behind a one-way mirror/glass.

Concurrent with the training, these teachers work with children in one-to-one lessons for part of the school day. Teacher leaders also make school visits and support implementation in schools.

Instructors must be professionals. When selecting Reading Recovery teachers, schools are encouraged to consider teaching experience, preferably with primary-grade students.

The training manuals have been trialed with thousands of children across the years and subsequent revisions followed. Books used for training Reading Recovery professionals also have evolved after trials with children.

Ongoing professional development is a requirement for all Reading Recovery professionals. Several professional development sessions are scheduled yearly and include continued observation of one-to-one teaching sessions. Reading Recovery teacher leaders also provide ongoing support to teachers at their schools.

 

Participants: Convincing Evidence

Sample size: Students in the 1,254 schools that participated in the RCT were identified to be part of the experiment via the following screening process: Reading Recovery teachers first screened a pool of candidates for Reading Recovery intervention using the Observation Survey of Early Literacy Achievement (OS) (Clay, 2013). Candidates included 1st-grade students who were identified by school staff—including kindergarten, 1st-grade, and intervention teachers—as struggling readers.

Risk Status: Once schools had identified eight 1st-grade students with the lowest OS scores, Reading Recovery teachers entered the names of the selected students into an online random assignment tool, noting their English language learner (ELL) status and their baseline OS Text Reading Level (TRL) subtest scores. The tool then matched them into pairs by first matching any students with ELL designations, then matching the student with the lowest TRL subtest score with the next-lowest student, and so on. Once the students were matched, a randomizing algorithm then randomly assigned one student in each pair to the treatment group and the other to the control group. The result was recorded in IDEC, and the tool was locked so that randomization in that school could not be redone.

Demographics:

 

Program

Control

Cox Index

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

3444

100

3444

100

0

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

 

12

 

13

0.06

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

 

 

 

 

 

  Hispanic

 

20

 

19

0.04

  White

 

42

 

44

0.05

  Other

 

26

 

24

0.06

Socioeconomic status

  Subsidized lunch

 

 

 

 

 

  No subsidized lunch

 

 

 

 

 

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

 

 

 

 

 

ELL status

  English language learner

664

26.1

639

25.8

0.00

  Not English language learner

2780

73.9

2805

74.2

0.00

Gender

Female

 

40

 

39

0.03

Male

 

60

 

61

0.03

Training of Instructors: Teachers who deliver the Reading Recovery intervention must complete an intensive, year-long graduate-level training course taught by a literacy coach called a teacher leader. Through this training and ongoing professional development, teachers develop expertise at analyzing students’ literacy behaviors, identifying learning needs, and delivering responsive instruction. Reading Recovery theory asserts that the role of the teacher is to identify students’ strengths and needs, and to facilitate his or her learning by providing appropriate opportunities to acquire and practice new skills. Other key roles in Reading Recovery implementation include teacher leaders, who train and support Reading Recovery teachers, and Reading Recovery trainers, who are faculty members at OSU and its 19 partner universities around the U.S. Each partner university operates a University Training Center (UTC) that serves as a regional hub of Reading Recovery training and activity. University-based trainers support and train teacher leaders, and oversee the program’s operations within their areas.

Design: Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: N/A

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: N/A

Were the program and control groups demographically comparable at pretreatment?: Yes

Was there attrition bias1 ?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

 

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Fidelity of Implementation: Unconvincing Evidence

Describe when and how fidelity of treatment information was obtained: Chapter 4, Reading Recovery Implementation Fidelity: Scale-up Successes and Challenges, focuses on the results for implementation fidelity under the scale-up, and describes the methods we used to obtain them. We report that, with very few exceptions, strong fidelity to the program model was seen in all four years of the evaluation (p. 8, see pages 69-82).

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: CPRE/CRESP’s evaluation of implementation fidelity represents a rigorous, sustained, systematic, and successful attempt to measure and assess wide-ranging implementation activities. While our measurement was done via statistical analysis of survey data, the qualitative inquiry served both to corroborate the survey findings and to shed light on the few departures from fidelity that were observed. A consistent finding is that Reading Recovery was implemented with high fidelity to the program model over the course of the i3 scale-up. We found that the intervention was delivered as designed to the students in the scale-up, and that teachers delivering Reading Recovery lessons were properly trained. All in all, this finding suggests that the impacts we observed were, indeed, the result of faithful implementation of the intervention, and that the i3 scale-up successfully replicated Reading Recovery in the schools involved in the expansion (p. 82).

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Clay’s Observation Survey of Early Literacy Achievement Total Score (0S)

Cronbach’s Alpha = 0.87

Split-Half = 0.89

These measures are relevant to early literacy learning in classroom and intervention settings and predictive of end of first grade reading difficulty with a cut score of 425 with 90% sensitivity.

These measures apply equally well to classroom literacy instruction.

Broader  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Iowa Test of Basic Skills (ITBS) Total score

Information regarding the technical characteristics of the ITBS was obtained through the Guide to Research and Development (GRD Manual), the ITBS technical manual. The GRD Manual contains multiple reliability coefficients (internal consistency, equivalent forms, test-retest), most of which range between the middle 0.80s to low 0.90s (p. 35).

The ITBS Reading Total served as the confirmatory outcome measure for the RCT. The ITBS is a well-regarded, group-administered, norm- and criterion-referenced, standardized assessment designed to “assess the extent to which a child is cognitively ready to begin work in the academic aspects of the curriculum” (Hoover et al., 1994, as cited in Tang & Gómez-Bellengé, 2007), and to “measure growth in fundamental areas of school achievement” (Hoover et al., 2003, p.1).

Applies equally to literacy learning in a classroom or small group setting.

Iowa Test of Basic Skills (ITBS) Reading Words sub-scale

As Above.

 

 

Iowa Test of Basic Skills (ITBS)

Comprehension sub-scale

As Above.

 

 

 

Number of Outcome Measures: 4 Reading

Mean ES - Targeted: 0.93*

Mean ES - Broader: 0.42*

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Reading

Clay’s Observation Survey of Early Literacy Achievement Total Score (0S)

0.93***

Broader Measures

Construct

Measure

Effect Size

Reading

ITBS Total score

0.44***

Reading

ITBS Reading Words sub-scale

0.40***

Reading

ITBS Comprehension sub-scale

0.41***

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: Yes

Administration Group Size: Individual

Duration of Intervention: 30 minutes, 5 times a week, 12-20 weeks

Minimum Interventionist Requirements: Professional, An academic year of training with university credit

Reviewed by WWC or E-ESSA: WWC & E-ESSA

What Works Clearinghouse Review

Beginning Readers Protocol

EffectivenessReading Recovery® was found to have positive effects on alphabetics and general reading achievement and potentially positive effects on fluency and comprehension.

Studies Reviewed: 3 studies meet standards out of 79 studies total

Full Report

English Language Learners Protocol

Effectiveness: No studies of Reading Recovery® that fall within the scope of the English Language Learners (ELL) review protocol meet What Works Clearinghouse (WWC) evidence standards. The lack of studies meeting WWC evidence standards means that, at this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of Reading Recovery® on ELL.

Studies Reviewed: N/A

Full Report

 

Evidence for ESSA

Program Outcomes: Reading Recovery has been evaluated in four qualifying studies (many other studies did not qualify because they only reported outcomes for successful students, or used developer-made measures). In comparison to control groups, the average effect size across the four studies was +0.43 on measures such as ITBS, CAT, Woodcock, and Gates. These outcomes qualify Reading Recovery for the ESSA “Strong” category, and for the “Solid Outcomes” rating (at least two studies with effect sizes of at least +0.20).

Number of Studies: 4

Average Effect Size: 0.43

Full Report

Other Research: Potentially Eligible for NCII Review: 3 studies