NumberShire Level 1
Study: Fien et al. (2015); Smith et al. (2016)

Describe how students were selected to participate in the study:

After obtaining parent consent and student assent, first-grade students in 26 classrooms* were screened using the math screening measure easyCBM-CCSS. After screening, the 10 lowest students in each class were rank ordered according to their easyCBM-CCSS scores then matched in pairs. Each member of the pair was then randomly assigned to either the intervention or the control group. *One classroom dropped out of the study.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Prior to randomization, all participating first grade students completed a math screening assessment, the EasyCBM-CCSS fall benchmark. Based on performance on this assessment, the 10 lowest students in each class were identified as “NumberShire eligible.” On average, students in the intervention and control groups performed at the 20th percentile on the EasyCBM-CCSS fall benchmark relative to all students screened.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

The NumberShire group utilized the submitted program, NumberShire Level 1.

Specify which condition is the control condition:

The wait-list control group continued with their typical instruction: “Business as usual”.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Students. Eligibility for the NS1 intervention used a three-stage process. First, 632 consented first grade students enrolled in the 26 participating classrooms were screened in fall 2013. Students were screened using the fall benchmark of the EasyCBM – Common Core State Standards (EasyCBM-CCSS; Alonzo et al., 2006) assessment. In each classroom, the 10 students with the lowest scores on the EasyCBM-CCSS were identified as NS1-eligible and then matched in pairs according to their scores. Each participating classroom had five pairs of NS1-eligible students. For example, the two lowest performers on the EasyCBM-CCSS formed the first pair, while the students who were rank ordered in the ninth and tenth positions formed the fifth pair. A total of 250 students were determined eligible for the intervention.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Interventionists were 9 district-employed instructional assistants and one parent volunteer. The interventionists were predominantly female, and just over half were White, while the remaining were Hispanic or Latino. They had an average of 6.9 years of experience working in schools, and 3.3 years providing interventions for students at risk for mathematics difficulties. The majority did not have formal training in mathematics or education. They had varying levels of experience with technology. One-third of interventionists reported using technology-based math interventions prior to participating in the study.

Describe when and how fidelity of treatment information was obtained.

Project staff observed one NumberShire session in each classroom in the final two weeks of implementation. In addition, research staff tracked student progress through game sessions on a weekly basis. Metrics gathered during NS1 gameplay served as an additional measure of fidelity of implementation, including number of sessions completed, number of items completed, and latency and accuracy in responding.

What were the results on the fidelity-of-treatment implementation measure?

Project staff directly observed each NS1 intervention group once during the eight-week study using the Technology Observation Tool (TOT: Nelson & Doabler, 2013). The TOT is a researcher-developed, standardized protocol designed to assess fidelity of implementation of the NS1 intervention. Project staff observed and rated each of the intervention session sites (e.g., computer labs) on six items of implementation fidelity: (a) use of effective procedures at start of gameplay, (b) students use of headphones during gameplay, (c) student engagement, (d) active monitoring and classroom management, (e) troubleshooting of technological issues, (f) use of effective procedures at end of gameplay. All items were rated on a 4-point scale (1 = not present, 4 = highly present) and were averaged to compute an overall implementation fidelity score. The average fidelity ratings for interventionists’ use of effective procedures at the start of the session, active monitoring during student gameplay, and use of effective procedures at the conclusion of the session were 3.5 (SD = 1.1), 3.6 (SD = 0.7), and 3.2 (SD = 1.2), respectively. Interventionists also received an average rating of 2.6 (SD = 1.1) for troubleshooting technology issues during sessions. Observers rated students’ engagement during gameplay and use of headphones, a critical component of NS1, as 3.7 (SD = 0.7) and 3.2 (SD = 0.8), respectively. The average overall fidelity score was 3.3 (SD = 0.8), indicating moderate overall fidelity with substantial variability between NS1 groups. Metrics gathered during NS1 gameplay served as an additional measure of fidelity of implementation, including number of sessions completed, number of items completed, and latency and accuracy in responding. Between pretest and posttest treatment students, on average, completed 18.6 game sessions or 4.7 weeks of game play (SD = 8.1 sessions, range = 2 to 33 sessions) and repeated 12.8 game sessions (SD = 5.9, range = 2 to 24). During game play treatment students completed an average of 499.8 practice opportunities (SD = 269, range = 41 to 1,156) and completed 69% of the practice opportunities correctly (SD = 12%, range = 37% to 90%). Project staff used gameplay metrics to track student progress through game sessions on a weekly basis, and corresponded regularly with interventionists to provide support when needed (e.g., when a student’s gameplay progression deviated from the standard schedule of four sessions per week)

Was the fidelity measure also used in control classrooms?

No. Because NumberShire was a specific technology-based intervention that was not used in the control classrooms, the fidelity of measure was not applicable.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Two-week mastery tests were not administered pre-post.

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Univariate effects of intervention condition on posttest outcome measures were examined using between subjects analysis of covariance (ANCOVA) adjusting for pretest scores. Intervention effects on the two-, four-, and six-week interim mastery tests were also evaluated using ANCOVAs adjusting for pretest proximal NumberShire (PN) total score as a covariate. Pearson’s r correlation coefficients were used to explore associations between number of sessions completed and change in outcomes from pretest to posttest among students assigned to the treatment condition. Non-nested analyses were appropriate for this study given that students were both the unit of randomization and the unit of analysis. All analyses were conducted with SPSS 21 and alpha was set to p < .05, two-tailed, for all tests. Hedges’ g was reported as a metric of intervention effect size (What Works Clearinghouse, 2008; .2, .5, and .8 are considered small, medium, and large effects). Hedges’ g was computed as the difference between the covariate adjusted means of the two groups at posttest divided by the posttest pooled standard deviation of the outcome.