ROOTS (Whole Number Foundations Level K)
Study: Clarke et al. (2020)

Describe how students were selected to participate in the study:

A total of 138 kindergarten classrooms participated in the efficacy trial. Within the 138 participating classrooms, all kindergarten students with parental consent were screened in late fall of their kindergarten school year.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All students with parental consent from the 138 classrooms were screened in the late fall of their kindergarten year. The screening process included the Assessing Student Proficiency in Early Number Sense (ASPENS; Clarke et al., 2011) and the Number Sense Brief (NSB; Jordan et al., 2010), which are standardized measures of early mathematics proficiency. Students were eligible for the ROOTS intervention and thus considered at risk for mathematics difficulty (MD) if they received an NSB score of 20 or less and a composite ASPENS score in the strategic or intensive range. After being determined eligible for the ROOTS intervention, students’ NSB and ASPENS scores were separately converted into standard scores and then combined to form an overall composite score for each at-risk student. All data management was conducted by the project’s independent evaluator. Composite scores within each classroom were then rank ordered, and the 10 ROOTS-eligible students with the lowest composite scores were randomly assigned to one of three conditions: (a) 2: 1 ROOTS group, (b) 5: 1 ROOTS group, or (c) no-treatment control condition. The 10 ROOTS-eligible students were then pre-tested using the Test of Early Mathematics Ability (TEMA-3). The average TEMA-3 raw score at pretest was 17.0 for the sample, which corresponds to the 19th percentile according to TEMA-3 scoring documents.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

40.2%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

ROOTS

Specify which condition is the control condition:

BAU

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

After being determined eligible for the ROOTS intervention, students’ NSB and ASPENS scores were separately converted into standard scores and then combined to form an overall composite score for each at-risk student. All data management was conducted by the project’s independent evaluator. Composite scores within each classroom were then rank ordered, and the 10 ROOTS-eligible students with the lowest composite scores were randomly assigned to one of three conditions: (a) 2: 1 ROOTS group, (b) 5: 1 ROOTS group, or (c) no-treatment control condition.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The ROOTS intervention was delivered by district employees and interventionists hired specifically for the efficacy trial. The majority of interventionists (93.5%) identified as female (93.5%) and White (76.1%), with 12.0% identifying as Hispanic. The remaining 11.9% identified as another race/ethnicity or declined to respond. Almost all interventionists (92.3%) had previous experience providing small group instruction, and 60.5% had a bachelor’s degree or higher. About half of interventionists (56.5%) had taken an algebra course at the college or graduate level. On average, interventionists had 10.4 years of teaching experience (SD p 8.6), and 22.0% had a current teaching license or certification. The ROOTS interventionists participated in two 5-hr professional development workshops that were delivered by project staff with a background in mathematics education, including one of the curriculum developers. The initial workshop focused on mathematics content covered through Lesson 25, effective instructional practices (e.g., eliciting group and individual responses, providing academic feedback), and strategies for small group management (e.g., instructional pacing, setting group expectations). The second workshop covered content from Lesson 26 to 50 and reviewed instructional and management strategies. During each workshop, project staff modeled lesson delivery and provided time for interventionists to practice lessons and receive feedback on their use of instructional practices. All interventionists also received between two and four coaching visits from ROOTS coaches during intervention implementation to boost implementation fidelity and enhance instructional quality. The coaching visits consisted of direct observations of lesson delivery followed by feedback on instructional quality (e.g., use of effective instructional practices and group management strategies) and fidelity of intervention implementation (e.g., correctly using mathematical models, following the teacher scripting).

Describe when and how fidelity of treatment information was obtained.

Fidelity of ROOTS implementation was measured via direct observations by trained research staff. Each ROOTS group was observed three times during the course of the intervention. On a 4-point scale (4 = all, 3 = most, 2 = some, 1 = none), observers rated the extent to which the interventionist (a) met the lesson’s instructional objectives, (b) followed the provided teacher scripting, and (c) used the prescribed mathematical models for that lesson. Observers also recorded whether the interventionist taught the number of activities prescribed in the lesson.

What were the results on the fidelity-of-treatment implementation measure?

Interventionists were observed to meet instructional objectives (M = 3.49, SD = 0.69), follow scripting (M = 3.31, SD = 0.75), and use prescribed models (M = 3.61, SD = 0.64). Interventionists also taught the majority of prescribed activities (M = 4.14 out of 5 activities per lesson, SD = 0.77). Interclass correlation coefficients (ICCs) for these fidelity ratings across observers were as follows: .82 for number of activities taught, .70 for meeting instructional objectives, .75 for following teacher scripting, and .70 for using prescribed mathematical models. According to guidelines proposed by Landis and Koch (1977), these ICCs indicate substantial agreement across observers. There were no differences in ICCs by group size.

Was the fidelity measure also used in control classrooms?

No. The ROOTS intervention occurred outside of and in addition to core mathematics instruction. To document the instructional practices and mathematics content employed in the control condition, research staff administered two surveys and conducted one direct observation of core mathematics instruction during the intervention time period. No evidence of treatment diffusion or contamination of ROOTS instruction was found in the core mathematics instructional settings.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

SAT 10 and SESAT were not administered pre and post and were used to investigate secondary research questions (e.g. closing the achievement gap with not-at risk peers).

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

We assessed intervention effects on each of the primary outcomes with a mixed model (multilevel) time × condition analysis (Murray, 1998) designed to account for students partially nested within small groups (Baldwin, Bauer, Stice, & Rohde, 2011; Bauer, Sterba, & Hallfors, 2008).