Fraction Face-Off! (previously Fraction Challenge)
Study: Fuchs et al. (2012)

Describe how students were selected to participate in the study:

The sample comprised fourth-grade students from 53 classrooms in 13 schools, 290 of whom were identified as AR. We randomly assigned the AR students, at the individual level, stratifying by classroom and risk severity, to fraction intervention or control groups. Another 292 low-risk classmates served as a comparison group for interpreting AR progress in response to the same classroom fraction instruction and for gauging the extent to which AR students closed the fraction achievement gap. We sampled 2-8 AR and low-risk students per classroom. When screening yielded more students than could be accommodated in the study, we randomly selected students for participation. We defined risk as performance on a broad-based calculations assessment (Wide Range Achievement Test–4 or WRAT-4; Wilkinson, 2004) below the 35th percentile, and purposely sampled half the AR sample from below the 17th percentile (more severe) and the other half from between the 18th and 34th percentiles (less severe). In the beginning-of-fourth-grade range of performance, the WRAT almost entirely samples whole-number items. Because this study was not about intellectual disability, we excluded 18 students with T-scores below the 9th percentile on both subtests of the Wechsler Abbreviated Scales of Intelligence (WASI; Psychological Corporation, 1999), as per the study protocol. IN THIS TRC REVIEW, IT IS IMPORTANT TO NOTE THAT WE ANALYZED EFFECTS AS A FUNCTION OF MOVE SEVERE (<17TH PERCENTILE) VS. LESS SEVERE RISK (BETWEEN 18TH AND 34TH PERCENTILES). THERE WAS NO INTERACTION BETWEEN RISK SEVERE AND INTERVENTION CONDITION, INDICATING FRACTION CHALLENGE WAS COMPARABLY EFFECTIVE FOR STUDENTS WITH MORE AND LESS SEVERE RISK. WE ALSO REPORT EFFECT SIZES SEPARATELY FOR MORE AND LESS SEVERE RISK IN THE BODY OF THE PAPER. IN THISI WAY, WE BELIEVE THE REPORT MEETS THE TRC CRITERION ON PARTICIPANTS. Of the 290 AR students, 22 moved (10 tutored; 12 controls) before the end of the study, and another nine had at least one piece of missing data. These 31 students did not differ from the remaining students on pretest measures, and Little's (l988) MCAR test indicated data were missing at random (i.e., there was no identifiable pattern to the missing data). We therefore omitted these 31 cases, leaving 259 students in the final AR sample: 129 tutored students (60 more severe; 69 less severe) and 130 control students (66 more severe; 64 less severe). In the comparison group of 292 low-risk students, 10 moved prior to the end of the study. These 10 students did not differ from remaining students on pretest measures, and Little's MCAR test indicated data were missing at random. We therefore omitted these 10 cases. See Table 1 (in attached report) for pretest WRAT performance and demographic data for each study condition. Because WRAT was the defining variable for risk severity (more severe vs. less severe), the latter group outperformed the former, but there was no difference between conditions (tutoring vs. control) and no revealed no interaction between severity and condition. Inclusion of low-risk students revealed that these students performed reliably higher than each of the four AR groups. Demographic data (not collected for low-risk students) was comparable for the AR groups, as a function of risk severity and condition.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Risk was defined as performance on a broad-based calculations assessment (Wide Range Achievement Test–4 or WRAT-4; Wilkinson, 2004) below the 35th percentile, and purposely sampled half the at-risk sample from below the 17th percentile (more severe) and the other half from between the 18th and 34th percentiles (less severe).

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

In report, referred to as "intervention" group.

Specify which condition is the control condition:

In report, referred to as "control" group.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

: To investigate effects of intervention focused primarily on understanding of fractions and whether effects are moderated by child characteristics, we identified AR students and randomly assigned them to tutoring and control conditions at the individual student level, while stratifying by classroom and severity of students’ risk status.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Tutors were licensed teachers or non-licensed personnel, each of whom was responsible for two to four tutoring groups. Tutors were trained in a 2-day workshop, with bi-weekly 1 hour meetings providing additional updates on upcoming tutoring topics and problem solving concerning challenging students.

Describe when and how fidelity of treatment information was obtained.

Every intervention session was audiotaped. 20% of recordings (n = 293) were randomly sampled such that tutor, student, and lesson were sampled comparably. A research assistant listened to each sampled tape, while completing a checklist to identify the essential points the tutor conducted. Two research assistants independently listened to 20% (n = 58) of the 293 recordings to assess concordance. The mean difference in score was 1.74% (SD = 2.81).

What were the results on the fidelity-of-treatment implementation measure?

The mean percentage of points addressed was 97.69 (SD = 3.39).

Was the fidelity measure also used in control classrooms?

No

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

We assessed pretreatment comparability with a 2-way analysis of variance (ANOVA) on pretest performance for each measure. In these analyses, treatment condition (intervention vs. control) and risk severity (more severe vs. less severe) were the factors. At the start of intervention, there were no significant effects on any measure for treatment condition or for the interaction between treatment condition and risk severity. On whole-number calculations and NAEP, there was a significant main effect for risk severity (across tutored and control groups), which does not threaten the study’s internal validity. See Table 3 (in attached report) for means and standard deviations. We assessed posttest AR students’ fraction performance as a function of treatment condition and risk severity using parallel ANOVAs. Intervention enhanced the fraction performance of AR learners, and this effect was not moderated by students’ risk severity. See Table 3 (in attached report) for means and standard deviations. Note that for each measure, we report the intra-class correlation (ICC) for the effect of classrooms (and teachers) in which children received their core instruction. These ICCs were negligible or small and, in no case, did their inclusion in the model affect findings.