ROOTS (Whole Number Foundations Level K)
Study: Clarke et al. (2011)

Describe how students were selected to participate in the study:

Students were nested within classrooms, which were randomly assigned to treatment (ROOTS) or control.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Participating teachers (n=29) were asked to, “please select 5 students whom you think would benefit from a small group math intervention program.” These 143 nominated students comprised our student sample.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

ELM + ROOTS is the treatment group.

Specify which condition is the control condition:

ELM-only is the control group.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Full day kindergarten classrooms who participated in the ELM study were randomly assigned to treatment or control conditions, blocking on teachers’ ELM experience (one year or none) and school. Fourteen classrooms were in the treatment condition (ELM+Roots) and 15 classrooms were in the control condition (ELM-only).

What was the unit of assignment?

Teachers

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Fourteen instructional assistants (IAs) participated in the study, of which thirteen were female and all identified themselves as White. Three of the IAs had college degrees and two held current teacher certifications. In this sample, nine of the IAs had four or more years experience as an instructional assistant and all but four of the IAs had completed college level coursework in mathematics. IAs taught the ROOTS program. Participating IAs attended three PD workshops focused on the ROOTS curriculum. The initial PD workshop focused on the instructional objectives related to Lessons 1-25, the critical content of kindergarten mathematics, small-group management techniques, and the instructional practices that have been empirically validated to increase student math achievement (e.g., teacher provided academic feedback). In the second and third workshops the same format was followed as in workshop 1 but with a focus on the second half of the ROOTS curriculum, Lessons 26-50. Workshops were 4 hours in length.

Describe when and how fidelity of treatment information was obtained.

Online logs completed by the 14 instructional assistants (IAs) who delivered the ROOTS intervention. Research staff observed ROOTS instruction 1-3 times over the course of the study and rated fidelity of implementation using a 3-point rating scale where 3 represented “full implementation,” 2 represented “partial implementation,” and 1 was “not taught”.

What were the results on the fidelity-of-treatment implementation measure?

Online logs revealed that groups generally completed all 50 ROOTS lessons during the year. IAs demonstrated high fidelity scores (M = 2.92, SD = 0.06) for the activities prescribed. Seventy five percent implementation would correspond to a 2.5 on our scales thus the average score of 2.92 represents high levels of fidelity above the 75% criteria. Intraclass correlation coefficients (ICC) representing inter-rater reliability showed substantial agreement between observers (ICCs = 0.67).

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

We assessed intervention effects on each of the primary outcomes with a mixed model (multilevel) time by condition analysis (Murray, 1998). This tests differences between conditions on change in outcomes from the fall of kindergarten (T1) to the spring (T2). The specific model tests time coded 0 at T1 and 1 at T2, condition coded 0 for control and 1 for ROOTS, and the interaction between the two. With 29 schools, tests of time by condition used 27 degrees of freedom. The analyses included students identified as at risk for math difficulties across both conditions. The analysis included all available data—whether or not students’ scores were present at both time points—to estimate differences between assessment times and between conditions, which minimizes the potential for bias due to missing data. The nested time by condition analysis accounts the intraclass correlation associated with multiple students nested within the same schools. As a test of net differences, it also provides an unbiased and straightforward interpretation of the results (Cribbie & Jamieson, 2000, Fitzmaurice, Laird, & Ware, 2004). Model estimation. We fit models to our data with SAS PROC MIXED version 9.1 (SAS Institute, 2009) using restricted maximum likelihood (REML), generally recommended for multilevel models (Hox, 2002). From each model, we estimated fixed effects and variance components. Maximum likelihood estimation for the time by condition analysis allows the use of all available data and provides potentially less biased results even in the face of substantial attrition, provided the missing data were missing at random (Schafer & Graham, 2002). In the present study, we did not believe that attrition or other missing data represented a meaningful departure from the missing at random assumption, meaning that missing data did not likely depend on unobserved determinants of the outcomes of interest (Little & Rubin, 2002). The models assume independent and normally distributed observations. We addressed the first, more important assumption (van Belle, 2008) by explicitly modeling the multilevel nature of the data. Regression methods have also been found quite robust to violations of normality and outliers have a limited influence on the results in a variety of multilevel modeling scenarios (Bloom, Bos, & Lee, 1999; Donner & Klar, 1996; Fitzmaurice et al., 2004; Maas & Hox, 2004; 2004b; Murray et al., 2006). This feature of multilevel models also eases concerns about the use of different scoring methods used for different measures in the analyses (e.g., raw scores, scaled scores, standard scores). Effect sizes. To ease interpretation of effects we computed an effect size, Hedges’ g (Hedges, 1981), for each fixed effect. Hedges’ g represents an individual-level effect size comparable to Cohen’s d (Rosenthal, Rosnow, & Rubin, 2000), except that Cohen’s d uses the sample standard deviation while Hedges’ g uses the population standard deviation (Rosenthal & Rosnow, 2008).