Reflex
Study: Rudel (2016)

Describe how students were selected to participate in the study:

Teachers were asked to volunteer for the study from a school in Florida. The participating students were from the classes of these teachers. Students who were identified before the study began as not being ready for math fact acquisition were not included. (The concern was that these students would be unlikely to receive math fact fluency instruction in the treatment condition, leading to an unfair comparison.)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students who had fewer than 10 digits correct per minute at pretest (across 3 1-minute probes) were identified as at-risk for failure in mathematics. This is the threshold for M-CBM screening cut-off provided by the National Institute on Response to Intervention (Fuchs and Fuchs). PLEASE NOTE: The school did not provide data on at-risk students or percentiles on normed achievement tests, so we are only using the above criterion when reporting percentage of at-risk students in the population. However, given the large percentage of traditionally under-served populations in the study, we expect the percentage of students who would be deemed "at-risk" based on achievement test percentiles would be far higher than the percentage tendered below. We are providing sub-population analyses on Hispanic, LEP, and SES students to show the effect of Reflex on these populations.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

8.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Students in classes where there was any use of Reflex.

Specify which condition is the control condition:

Students in classes that had no use of Reflex.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Random numbers were generated by a Python program by a blinded analyst. NOTE: This study was intended as an RCT but was compromised (see page 6 of full text ) in the case of 1 of the teachers.

What was the unit of assignment?

Classes

If other, please specify:

Please describe the unit of assignment:

There were 8 participating classes. Random assignments were made based on the names of the homeroom teachers provided to project personnel. However, due to departmentalization, two of the classes (1 treatment and 1 comparison) had the same math teacher.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

Each student was analyzed individually as part of a 2-level HLM model with clusters as the higher level unit.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

4 of the 7 teachers had master's degrees (3 classes in treatment, 1 class in comparison). The other 3 teachers had bachelor's degrees. The number of years of teaching experience among treatment classes were {21, 13, 2, "15 or more"}. The number of years of teaching experience across comparison groups were {5, 15 or more, 15 or more, 15 or more}. (We relied on testimony from the math coach for those teachers who did not return information on background, and several teachers were at the school prior to her being there, so "15 or more" was all the information we could get for those teachers.) All teachers in the program condition received training on using Reflex.

Describe when and how fidelity of treatment information was obtained.

All work in Reflex occurs within the Reflex system, so session logs were consulted to derive information on usage.

What were the results on the fidelity-of-treatment implementation measure?

One class had poor usage in general (average 1.5 days/week), one class had reasonable usage (2.7 days/week), and two had good usage (3.3 and 3.4 days/week). However, these averages can be misleading since some students elected to play outside of school and had many days of usage, skewing the average. Overall, approximately 30% of the students in the study had poor usage (<1.6 days/week), 15% had reasonable-to-good usage (1.7-2.5 days/week) and the rest had near-model to model usage. (>2.5 days/week).

Was the fidelity measure also used in control classrooms?

No.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

An HLM analysis was conducted to determine the effects of binary and continuous variables as well as their interactions for the full population. The only covariates with statistically significant effects were age and pretest scores. The comparison and treatment groups were statistically similar with regard to these variables. The values we have reported are those from the full model. For demographic sub-analyses, all statistically significant covariates were kept. These are detailed in the addendum. Due to the smaller numbers of students in these populations, a simple ANCOVA was used rather than an HLM.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

In addition to the pretest and posttest, the study included an interim test taken approximately 60% of the way through the study. There were 8 participants absent from the posttest (5 from the treatment, 3 from the comparison group). Of these 8 students, 7 were present for the interim assessment. An ordinary least squares regression was used to impute posttest scores based on interim and pretest scores as well as demographic covariates. (See page 16 of full text.) The full text also provides a no-imputation analysis that does not consider students absent at posttest.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

An HLM model was constructed with clusters (classes) at the upper level and individual variables at the lower level. Statistical significance and effect size were based on the size and t-value for the intervention's coefficient within that analysis. For subpopulation analyses, a simply ANCOVA was used, and the strength and statistical significance of the effect was measured based on the coefficient of the intervention categorical variable.