Hot Math Tutoring
Study: Fuchs et al. (2008)

Describe how students were selected to participate in the study:

This study was conducted across four years, with 30 classrooms participating each year, for a total of 120 third-grade classrooms. We refer to each year’s sample as a “cohort.” Stratifying so that each condition was represented approximately equally in each school, we randomly assigned 40 classrooms to control (i.e., teacher-designed math problem-solving instruction) and 80 classrooms to Hot Math SBI (i.e., researcher-designed SBI). One Hot Math SBI teacher dropped out due to personal reasons during the first month of the study. To obtain a representative sample, we screened 2,023 students on whom we had consent. That is, in the 119 third-grade classrooms, we randomly sampled 1,200 students for participation, blocking within classroom and within three strata: (a) 25% of students with scores 1 SD below the mean of the entire distribution on the Test of Computational Fluency (Fuchs, Hamlett, & Fuchs, 1990); (b) 50% of students with scores within 1 SD of the mean of the entire distribution on the Test of Computational Fluency; and (c) 25% of students with scores 1 SD above the mean of the entire distribution on the Test of Computational Fluency. Of these 1,200 students, 59 moved prior to posttesting (including 45 AR students). The 59 children who moved prior to posttesting were demographically comparable to the pupils who remained in the study. Among these students, we identified 288 students as AR of poor problem-solving outcomes. To derive a parsimonious equation for predicting problem-solving outcomes, we conducted regression analyses on a previous database (Fuchs, Fuchs, Prentice, Hamlett, Finelli, & Courey, 2004) of third-grade students who had received Hot Math SBI. The final prediction equation included pretest performance on the immediate transfer problem-solving measure (see Measures) and pretest performance on the Test of Computational Fluency (Fuchs et al., 1990). For each cohort, we rank ordered students on the predicted score and selected the lowest 72 students in that year’s sample. All of these students scored below the district criterion designating risk for math learning disabilities on the Test of Computation Fluency. These students were randomly assigned to tutoring conditions, while stratifying by classroom condition. In this way, some AR students received neither classroom nor tutoring Hot Math SBI; some received classroom but not tutoring Hot Math SBI; some received tutoring but not classroom Hot Math SBI; and some received classroom and tutoring Hot Math SBI. Of the 288 AR students, 45 moved prior to posttesting. On demographic and pretest performance variables, the 45 children who moved prior to posttesting were comparable to the 243 pupils who remained in the study, and there were no significant interactions between AR students’ tutoring condition and attrition status. See Table 1 for student demographics and pretreatment intelligence, reading, and math standard scores by group for the “program” and “control” groups. The demographics of the two groups relevant to the TRC review were comparable. (NOTE: In the research report, classrooms were randomly assigned to control (teacher-designed word-problem instruction; one-third of classrooms) or Classroom Hot Math (two-thirds of classrooms). Within these whole-class conditions, at-risk students were randomly assigned to control (no tutoring; one-third of students within each classroom condition) or Hot Math Tutoring (two-thirds of students within each classroom condition). In this way, AR students received one of four conditions: (1) no Classroom Hot Math and no Hot Math Tutoring (we refer to this as “control” in this protocol), (2) Classroom Hot Math and no Hot Math Tutoring (not addressed in this protocol), (3) no Classroom Hot Math with Hot Math Tutoring (referred to as the “program” condition in this protocol), and (4) Classroom Hot Math and Hot Math Tutoring (not addressed in this protocol). In the tables in the attached research report, look under “Class-level Condition: Control: Tutoring-level condition” (i.e., left side of tables). Then look at “AR tutor” for the program condition (second column) and at “AR control” for the control condition (third column). This contrasts AR students who did not receive Hot Math Tutoring against AR students who did receive Hot Math Tutoring. None of the students in these two conditions received Classroom Hot Math.)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All of these students scored below the district criterion designating risk for math learning disabilities on the Test of Computation Fluency. The at-risk sample was at the 24th percentile (lowest 72 of each cohort’s 300 students). The 300 students were a representative sample on a combination of the pretest immediate transfer measure of math problem solving (a reliable index that correlates well with commercial measures of math problem solving) and pretest performance on the Test of Computational Fluency, a reliable and widely used measure of mathematics skill. I use the term “representative sample” in the research design sense, i.e., representing the full range of performance (e.g., not among a sample of students selected low or high performing). In the case of this study/sample, students were in a metropolitan area with a high proportion of subsidized lunch students. So in terms of a national sample, it is safe to assume the samples are below the 25th percentile of a nationally representative sample in the demographic sense.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

In the research report, classrooms were randomly assigned to control (teacher-designed word-problem instruction; one-third of classrooms) or Classroom Hot Math (two-thirds of classrooms). Within these whole-class conditions, at-risk students were randomly assigned to control (no tutoring; one-third of students within each classroom condition) or Hot Math Tutoring (two-thirds of students within each classroom condition). In this way, AR students received one of four conditions: (1) no Classroom Hot Math and no Hot Math Tutoring (we refer to this as “control” in this protocol), (2) Classroom Hot Math and no Hot Math Tutoring (not addressed in this protocol), (3) no Classroom Hot Math with Hot Math Tutoring (referred to as the “program” condition in this protocol), and (4) Classroom Hot Math and Hot Math Tutoring (not addressed in this protocol). In the tables in the attached research report, look under “Class-level Condition: Control: Tutoring-level condition” (i.e., left side of tables). Then look at “AR tutor” for the program condition (second column) and at “AR control” for the control condition (third column). This contrasts AR students who did not receive Hot Math Tutoring against AR students who did receive Hot Math Tutoring. None of the students in these two conditions received Classroom Hot Math

Specify which condition is the control condition:

In the research report, classrooms were randomly assigned to control (teacher-designed word-problem instruction; one-third of classrooms) or Classroom Hot Math (two-thirds of classrooms). Within these whole-class conditions, at-risk students were randomly assigned to control (no tutoring; one-third of students within each classroom condition) or Hot Math Tutoring (two-thirds of students within each classroom condition). In this way, AR students received one of four conditions: (1) no Classroom Hot Math and no Hot Math Tutoring (we refer to this as “control” in this protocol), (2) Classroom Hot Math and no Hot Math Tutoring (not addressed in this protocol), (3) no Classroom Hot Math with Hot Math Tutoring (referred to as the “program” condition in this protocol), and (4) Classroom Hot Math and Hot Math Tutoring (not addressed in this protocol). In the tables in the attached research report, look under “Class-level Condition: Control: Tutoring-level condition” (i.e., left side of tables). Then look at “AR tutor” for the program condition (second column) and at “AR control” for the control condition (third column). This contrasts AR students who did not receive Hot Math Tutoring against AR students who did receive Hot Math Tutoring. None of the students in these two conditions received Classroom Hot Math.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

In a southeastern metropolitan school district, 120 third-grade classrooms participated in this study. Stratifying so that each condition was represented approximately comparably in each school, we randomly assigned 40 classrooms to the control condition (i.e., 3 weeks of researcher-designed general math problem-solving instruction) and assigned 80 classrooms to the Hot Math SBI condition (i.e., 3 weeks of researcher-designed general math problem solving plus 13 weeks of researcher-designed SBI). The study occurred over 4 school years during a time when the school district was relatively stable. One quarter of the sample entered the study each year. During the first 3 years, SBI classrooms were randomly assigned to Hot Math SBI or to a variant designed to strengthen Hot Math SBI. In the first three cohorts, the effects of the two SBI conditions were not statistically significantly different, but both were reliably better than control. Therefore, we did not test a variant in Cohort 4 (all Cohort 4 teachers were randomly assigned to control or to the standard version of SBI). We considered all student in the SBI classrooms to have participated in one SBI condition; however, to assess SBI variants, we included cohort effects in the analytic model. One Cohort 3 classroom in the SBI condition left the study during the first month of participation because of the classroom teacher’s personal reasons.

What was the unit of assignment?

Teachers

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

None of the tutors was a certified teacher; only one tutor had previous experience tutoring. Tutors were trained in one full-day session. Tutors were introduced to the program and its goals and provided instruction, demonstrations, and scripted materials. They were paired to practice the program. Then, they condcuted one lesson for a trainer and were judged on a point-by point system for fidelity to treatment. A tutor who achieved 95% fidelity was considered reliable. A tutor who scored lower than 95% fidelity was coached on points he/she missed, asked to practice more, and then re-rated at a later time on another lesson. At weekly meetings, tutors met with a trainer to solve problems that arose. At the beginning of each unit, a 3-hour training session was conducted to orient tutors and distribute supporting materials. Across the four years of the study, the typical tutor was one to two years beyond undergraduate education, studying for a graduated degree in education, special education, counseling, or education policy. The majority of tutors worked for the project one year, with three tutors working for more than one year. Each year of the study, two full-time project coordinators, typically with bachelor's or master's level degrees typically outside of education, also tutored. Each year, five or six tutors were needed. (None of the tutors conducted Classroom Hot Math and Hot Math Tutoring).

Describe when and how fidelity of treatment information was obtained.

Each tutoring session was audiotaped. At the study’s end, four research assistants independently listened to tapes while completing a checklist to identify the percentage of points addressed. We sampled tapes so that, within conditions, tutors, groups, and session numbers were sampled equitably. For each of 64 tutoring small groups, 20% of sessions were sampled (7-8 tapes distributed equally across the four units). Intercoder agreement, calculated on 20% of the sampled tapes, was 96.4%.

What were the results on the fidelity-of-treatment implementation measure?

The mean percentage of points addressed across all units was 98.12 (SD = 1.28).

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

We converted scores on the three problem-solving measures to percentage correct so performance on the three measures could be compared. To examine how much of the total variance in improvement on the three problem-solving measures was explained by the clustering of children in classrooms and in tutoring groups, we estimated variance components with SAS PROC MIXED (Littell, 2006). The resulting intraclass correlations showed that the effect for classroom clustering explained 16.10% of the variance (p <.001) and the effect for tutoring-group clustering explained 4.10% of the variance (p = .006). We therefore incorporated each as a random effect into our model, which also included four fixed effects: one within-subjects factor (problem-solving measure) and three between-subjects factors (classroom condition, tutoring condition, and cohort). To assess pretreatment comparability, we fit a full model that included all main effects, 2-way, 3-way, and 4-way interactions as well as estimated the impact of classroom as a random effects factor. To index learning as a function of study condition, we used improvement on the three problem-solving measures. (Fitting a model using improvement scores produces identical effects as would considering the interaction between test occasion [pre vs. posttest] and study conditions. We opted for improvement scores because their interpretation is more straightforward.) In this full model, the variance component for tutoring group decreased to zero (indicating that all of the variance associated with tutoring group clusters was explained in the model). So we fixed the random effects of tutoring group to zero and also eliminated from the final model all higher-order interactions that were not statistically significant. To follow-up significant effects, we Bonferroni-corrected p-values by the number of follow-up tests we ran for that significant effect. To compute effect sizes (ESs), we subtracted the difference between improvement means and then divided by the pooled standard deviation of the improvement/square root of 2(1-rxy) (Glass, McGaw, & Smith, 1981). To compute ESs for pre- and posttreatment scores, we subtracted the difference between means and divided by the pooled SD (Hedges & Olkin, 1985). For improvement scores, we corrected for the correlation between the pre- and posttest: difference between improvement means, divided by the pooled SD of improvement/square root of 2(1-rxy) (Glass, McGaw, & Smith, l981).