QuickReads
Study: Vadasy & Sanders (2008)

Describe how students were selected to participate in the study:

Referral and screening. In the fall of the academic year, 40 fourth- and fifth-grade teachers in 12 public elementary schools in a large northwestern city were asked to refer students who (a) had never been retained, (b) had low rates of reading fluency or comprehension, and (c) would particularly benefit from a fluency-oriented intervention (i.e., teachers were asked to recommend students with adequate word reading skills who would benefit most from fluency instruction and who could be pulled out for instruction). Once active parent consents were obtained, referred students were screened for eligibility. Students were considered eligible for participation if they demonstrated at-risk performance on the average of three gradelevel reading passages from the Oral Reading Fluency (ORF) subtest of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS; Good & Kaminski, 2002); fourth-grade at-risk performance was defined as scoring below 93 words correct per minute (WCPM) on fourthgrade passages, and fifth-grade at-risk performance was defined as scoring below 104 WCPM on fifth-grade passages. Of those screened, one fifth grader was recommended for an alternative intervention, as the student was able to read only 11 WCPM (far lower than bottom 10th percentile performance of 61 WCPM for fifth graders; see Hasbrouck & Tindal, 2006). Students eligible for participation were administered the full pretest battery. (p. 237)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students screened on the average of 3 passages drawn from DIBELS Oral Reading Fluency. Students who performed below grade level norms (“at risk” or “at some risk” according to the DIBELS norms) were considered eligible.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Quick Reads fluency pull-out for pairs of students was the treatment condition.

Specify which condition is the control condition:

Business-as-usual classroom instruction was the control condition. Important: random assignment occurred within schools, not by classrooms.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

RNG-uniform distribution used: students randomly assigned to dyads within school, and dyads randomly assigned to treatment or control conditions within schools.

What was the unit of assignment?

Other

If other, please specify:

Dyads

Please describe the unit of assignment:

Students were randomly assigned to dyads (pairs) and then dyads randomly assigned to conditions. Treatment effect tested at dyad-level.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:
Dyads

Please describe the unit(s) used for primary data analysis:

Treatment effect tested at dyad-level.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Twenty tutors were recruited from their school communities. Tutors’ educational levels, general tutoring experience, and experience working with fourth and fifth graders varied. Two tutors were employees of the district and served regularly as instructional assistants (IAs), and 3 were hourly employees. Prior to the study, tutors ranged from 0 to 11 years of tutoring experience and averaged 15 years of education: Two had master’s degrees, 11 had bachelor’s degrees (3 with teaching certificates), 2 had associate’s degrees, and 5 had attended some college. The average educational attainment of tutors in this study matches the paraeducator competency requirements under the No Child Left Behind Act of 2001. Tutor training: Tutors participated in one initial 4-hour training by project staff. Training included an overview of reading fluency development and the repeated reading method. Research staff then modeled the use of Quick Reads materials and vocabulary instruction. The tutors practiced and received feedback before they began work with students. Following initial training, coaches visited tutors weekly to provide coaching and modeling and to collect observation data on tutor instruction and management. Midyear, tutors attended a 3-hour workshop provided by research staff to reinforce tutoring strategies and effective student management. The workshop addressed specific tutor skills for successful Quick Reads lessons and included a demonstration of a Quick Reads lesson with students in study. Tutor coaching: Throughout the 20-week intervention, research staff supported and conducted observations of the tutors. Specific researcher coaches were assigned to a set of tutors, and for each tutor, a minimum of eight observations were conducted (of which there were at least two observations per dyad). Coaches met monthly to discuss tutoring implementation progress.

Describe when and how fidelity of treatment information was obtained.

To monitor treatment implementation fidelity, data were collected via observation forms, including (a) tutors’ adherence to scripted Quick Reads protocols, (b) tutor behavior in terms of both organization and responsiveness to students’ needs, and (c) student progress in terms of the amount of time spent actively engaged in reading passages. Tutors’ fidelity to scripted protocols was measured using a dichotomous (yes–no) implementation checklist that included two to five criteria for each of the Quick Reads instructional steps (the percentage of observed correct criteria across all steps was calculated per observation). Tutors’ behavior was measured using a 5-point rating scale of 0 (never) to 4 (always) on eight criteria, including “organizational materials,” “tutoring time spent on instruction,” “full tutoring time used,” “smooth transitions,” “corrections match error and skill,” “use of specific praise,” “quick pace,” and “keeps students engaged.” Student progress was measured by recording the amount of time (in seconds) students were actively reading text. Across all three measurements (protocols, behaviors, and student progress), only components actually observed were recorded (i.e., if the beginning of the tutoring session was the only component observed, then tutor behavior and student progress were not recorded).

What were the results on the fidelity-of-treatment implementation measure?

A total of 54 paired observations from five pairs of raters (one researcher-rater was used as a baseline for comparison with the other five) were used to obtain interobserver reliability. Adherence to scripted Quick Reads protocols reliability ranged from r = 0.53 to r = 0.91 and averaged r = 0.76. Reliabilities averaged r = 0.81 for tutor behavior ratings and r = 0.92 for the amount of time students spent on passage reading. Across 254 observations (approximately 13 per tutor), adherence to protocols averaged M = 90% (SD = 11.5%); across 248 observations (approximately 12 per tutor), tutor behaviors averaged M = 3.7 (SD = 0.55), and across 206 observations, each student (within the dyad) spent an average of M = 7.8 minutes per session (SD = 4.42) actively engaged in orally reading Quick Reads text.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

we have excluded the attention and RAN measures as they were only measured at pretest for the purpose of describing the sample (no treatment-control differences at pretest: see p. 243)

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Multilevel modeling with students at Level 1 and dyads at Level 2; although not stated in the publication, we tested whether within-dyad variance was heterogeneous due to the pseudo-dyad grouping of the control group; no significant heterogeneous Level 1 variance was detected, and no substantive difference in outcomes observed with incorporation of heterogeneous Level 1 variance was found. Hence the simpler multilevel model assuming homogeneous Level 1 variance was reported in the publication.