Number Rockets
Study: Fuchs et al. (2005)

Describe how students were selected to participate in the study:

All first-grade teachers (n = 41) in six Title 1 and four non-Title 1 schools in a southeastern metropolitan school district agreed to participate. From these 41 classrooms, we identified not-at-risk (NAR) and at-risk (AR) students in September using the following procedure. In whole-class format, we tested the 667 (89% of) students for whom we received parent consent. The measures were Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, and CBM Concept/Applications. Based on a factor score computed across these measures, we identified the 308 lowest-scoring students for individual testing, all of whom failed the local benchmark for designating risk status in math using the CBM Computation measure. Staff shared the names of these students with teachers, who nominated 11 additional students as potentially AR. Staff administered an individual battery to these 319 children. Then, based on the Week 4 CBM score, we identified the lowest 139 performing students as AR (i.e., 21% of the consented students); scores for all 139 students fell below the CBM benchmark for risk in math. These 139 students were randomly assigned to control or tutoring conditions, blocking by classrooms to ensure comparable distribution of AR students in the control and tutoring conditions within classrooms. This created two pools of students relevant to the TRC review: 69 AR control students and 70 AR tutored students (a comparison group of NAR students was also followed, but is not relevant to the TRC review). Because some students moved to other schools, the size of these two groups decreased from September to May, respectively, to 63 and 64. See Table 1 for student demographics and pretreatment intelligence, reading, and math standard scores by group for these 63 and 64 students. The demographics of the groups were comparable. As expected, on the intelligence and academic measures, the NAR students performed higher than the AR students, but the AR control and AR tutored groups performed comparably. NOTE: Standard scores on the Woodcock-Johnson III (WJ III; Woodcock, McGrew, & Mather, 2001) achievement measures placed the AR students near average (i.e., 93-98). By contrast, on measures for which standard scores are relative to classmates, performance fell substantially lower (85-93). (Note that the high scores on the Woodcock measures, relative to locally-normed measures, are surprising because our sample was drawn from an urban district, including six high-poverty schools, which can be expected to perform lower than a nationally representative sample. So, it is important to consider three points. First, other researchers [e.g., Speece & Case, 2001; Vellutino, Scanlon, Small, & Fanuele, in press] have also documented Woodcock measures’ inadequate sensitivity to reading difficulty in first grade. Second and relatedly, performance floors on the Woodcock measures make discrimination among low-performing first graders difficult. At the beginning of first grade on WJ III Calculation, for example, a raw score of 2 (e.g., Make the number 1; Make the number 3) corresponds to a standard score of 87. With one additional correct item (e.g., 1+1=), the standard score increases to 93; with one more correct item [e.g., 2+2=], to 99. Third, research at third grade [Fletcher, Cirino, & Fuchs, 2004] has also documented a pattern of high WJ III Calculation standard scores for low-performing students relative to another norm-referenced measure [i.e., WJ III Calculation scores were 8-10 points higher than Wide Range Achievement Test – Arithmetic scores]. All of this raises questions about the structure and/or norming of the Woodcock tests and lends credence to the locally normed figures. Moreover, with respect to the generalizability of findings, it is important to consider that many response-to-intervention models of prevention and identification and the studies underpinning those models [e.g., McMaster, Fuchs, Fuchs, & Compton, in press; Speece & Case, 2001] incorporate local normative frameworks to designate risk, as was done in this study.)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

In whole-class format, we tested the 667 (89% of) students for whom we received parent consent. The measures were Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, and CBM Concept/Applications. Based on a factor score computed across these measures, we identified the 308 lowest-scoring students for individual testing, all of whom failed the local benchmark for designating risk status in math using the CBM Computation measure. Staff shared the names of these students with teachers, who nominated 11 additional students as potentially AR. Staff administered an individual battery to these 319 children. Then, based on the Week 4 CBM score, we identified the lowest 139 performing students as AR (i.e., 21% of the consented students); scores for all 139 students fell below the CBM benchmark for risk in math. These 139 students were randomly assigned to control or tutoring conditions, blocking by classrooms to ensure comparable distribution of AR students in the control and tutoring conditions within classrooms. The at-risk sample was at the 21st percentile of a representative sample on pretest (Week 4 was before intervention, but having provided students with sufficient time to acclimate to the assessment) Curriculum-Based Measurement-Calculations (a reliable assessment of overall math competence at beginning of first grade). The term “representative sample” is used in the research design sense, i.e., representing the full range of performance (e.g., not among a sample of students selected low or high performing). In the case of this study/sample, students were in a metropolitan area with a high proportion of subsidized lunch students. So in terms of a national sample, it is safe to assume the samples are below the 25th percentile of a nationally representative sample in the demographic sense.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

“AR tutored” who received the Number Rockets program

Specify which condition is the control condition:

“AR control” who received business-as-usual (i.e., no tutoring)

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

These 139 students were randomly assigned to control or tutoring conditions, blocking by classrooms to ensure comparable distribution of AR students in the control and tutoring conditions within classrooms.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

None of the tutors was a certified teacher; only one tutor had previous experience tutoring. Training occurred as follow. In a 1-day training session for tutors, (a) an overview of the tutoring program, goals, and topics was presented, and (b) the tutoring procedures were explained for each activity in the first four tutoring topics. After presentation of each activity, tutors practiced the activity with a partner, with more practice completed in the next two weeks. One week later, in a second session, tutors learned to use the drill/practice math fact activities. At the end of that week, a review session was held. Tutoring began one week later. Also, tutors attended weekly meetings to learn about and practice upcoming tutoring topics. In these weekly sessions, tutors also discussed difficulties they faced. Supervisors facilitated the weekly meetings and helped tutors problem solve.

Describe when and how fidelity of treatment information was obtained.

All tutoring sessions were audiotaped. Tutors did not know which audiotapes would be checked for fidelity. We checked tapes for all 27 tutoring groups for Topic 4 (Day 1 or 2) and Topic 16 (Day 1) using a checklist that corresponded to the steps included in the lesson’s script, with 9-19 items (mean: 12) per checklist. Each checklist item was marked as observed, not observed, or not applicable. Fidelity was indexed as percentage of items implemented (observed divided by the sum of observed + not observed). A second coder re-checked fidelity for a random sample of 25% of the audiotapes. Agreement between coders was 88.3%.

What were the results on the fidelity-of-treatment implementation measure?

Across tutors and sessions, the percentage of fidelity for the first check was 95.6; for the second check, 93.5.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

One-way analyses of variance (ANOVAs) were applied to pretest, posttest, and improvement scores on the seven mathematics dependent variables, using condition (NAR vs. AR control vs. AR tutored) as the factor. In Table 2, we report raw score means and SDs by condition, as well as omnibus F values, Fisher LSD post hoc follow-up tests (Seaman, Levin, & Serlin, 1991) for significant effects, and effect sizes (ESs) comparing the conditions. To compute ESs for pre- and posttreatment scores, we subtracted the difference between means and divided by the pooled SD (Hedges & Olkin, 1985). For improvement scores, we corrected for the correlation between the pre- and posttest: difference between improvement means, divided by the pooled SD of improvement/square root of 2(1-rxy) (Glass, McGaw, & Smith, l981).