Spring Math
Study: Codding et al. (2016)

Describe how students were selected to participate in the study:

All students in grades 2, 3, and 4 in the participating school were eligible (N = 236). Eligible students were screened for mathematics difficulties using standard CBM screening procedures for mathematics and 141 of the screened students met at-risk criteria (less than 40 digits correct per two minutes for grade 2 and 3 students and less than 80 digits correct per two minutes for students in grade 4) for inclusion in study procedures. Because the study used a fluency-building intervention to examine dosage effects, follow-up assessments were conducted with all 141 students to verify that a fluency-building intervention would be a good match with their learning needs. Specifically, students were excluded if (a) they scored in the frustrational range (less than 20 digits correct per two minutes for grades 2 and 3 and less than 40 digits correct for grade 4) on all grade-level probes (this sequence is described below); (b) they could not represent a computation problem using drawings; or (c) they were receiving special education services. The final sample was 101 students in 10 classrooms from grade 2 (n = 39), grade 3 (n = 46), and grade 4 (n = 16).

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All students in grades 2, 3, and 4 were screened using standard CBM procedures. The grade 2 screening was addition and subtraction fact families 0-20; Grade 3 screening was mixed multi-digit addition and subtraction with and without regrouping; Grade 4 was multiplication and division fact families 0-12. Any student who scored below the mastery criterion for these measures at their grade level, participated in a follow-up survey-level assessment that was administered individually by researchers. In Grade 2, this involved administering subtraction with minuends to 20, subtraction with minuends to 9, subtraction with minuends to 5, addition with sums to 20, addition with sums to 12, and addition with sums to 6 in that order. Grade 3 skills assessed were subtraction of 2-digit by 2-digit with and without regrouping, subtraction with minuends to 20, subtraction with minuends to 9, subtraction with minuends to 5, addition of 2-digit by 2-digit with and without regrouping, sums to 20, sums to 12, and sums to 6. Grade 4 skills assessed were division 0-12, division 0-5, multiplication 0-12, multiplication 0-5, addition & subtraction fact families 0-20, subtraction 0-20, and sums to 20. Assessment progressed until the child scored in the instructional range (grades 2-3- 20-39 digits correct in two minutes and grade 4- 40-79 digits correct per two minutes). For the instructional level skill, the student was asked to draw a solution to the problem to assess the child’s mathematical thinking, reasoning, and to verify conceptual understanding of the task (i.e., understanding multiplication as repeated addition can be demonstrated by the child drawing factor sets such as 5 x 3 drawn as three sets of five or five sets of three).

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

100.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

4 days per week intervention using the classwide math intervention protocol with small groups.

Specify which condition is the control condition:

Weekly progress monitoring only, which we refer to as the control condition in the LDRP article.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

We had two additional conditions: once weekly intervention and twice weekly intervention.

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Participants were assigned to one of four conditions (i.e., four times weekly, twice weekly, once weekly, control) using stratified random sampling, controlling for skill level and grade so that each condition consisted of approximately equivalent numbers of students from each grade and skill level. Six groups of 3–5 students were arranged per condition (i.e., 24 total groups).

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Eight school psychology graduate students collected Screening and progress monitoring data (3 doctoral, 5 specialist-level). Six of these students (3 doctoral, 3 specialist-level) also led the intervention groups for each condition. Each graduate student was trained to use curriculum-based assessment and implement academic interventions as part of their program coursework. Graduate students participated in two training sessions that were specific to the procedures employed in the current study. First, graduate students viewed a 20-min video, in which the first author modeled the treatment protocol with three graduate students (not otherwise serving in the study) acting as participants. Graduate students observed the video in pairs and practiced implementing the treatment protocol with one another. During the second session, the first author reviewed all screening materials and observed test administration to correct errors and provide feedback. Scripted protocols guided all assessment and intervention activities.

Describe when and how fidelity of treatment information was obtained.

Procedural fidelity of intervention steps was evaluated using a checklist that accounted for 18 distinct interventionist behaviors representing one 12-min treatment session. All intervention sessions for each group were recorded using handheld audiotape recorders. Two specialist-level school psychology graduate students were trained in the intervention procedures and acted as independent reviewers. These graduate students were not otherwise involved with the study. The observers listened to 54 audio recordings (i.e., 12-min sessions) representing between 2 and 6 days of treatment for each of the six interventionists (57% of treatment sessions administered). We also observed and quantified child engagement during 12-18% of intervention sessions to ensure equivalence across experimental conditions.

What were the results on the fidelity-of-treatment implementation measure?

Mean treatment adherence was 93% (60% - 100%) as measured across all six interventionists. There were no commission or omission errors. Errors reflected wording alterations when intervention activities were explained. Mean treatment adherence was (a) 100% and 86% (66%-100%) for the four-times-weekly interventionists, (b) 97% (89% - 100%) and 87% (60% - 100%) for the twice-weekly interventionists, and (c) 100%, respectively, for both once-weekly interventionists. Mean observed student engagement was 98% (94-100%) on the Behavioral Observation of Students in Schools (BOSS; Shapiro, 2004).

Was the fidelity measure also used in control classrooms?

No, not for the intervention procedures because control students were only exposed to weekly progress monitoring assessment. Interscorer agreement data were collected for assessments for students in both control and treatment conditions.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Multilevel modeling was used to examine predictors of slope and final score across all measures. SAS PROC MIXED was used for the analyses due to the longitudinal nature of the data and our interest in monitoring individual progress of each student. Two-level models were analyzed where students served as the level-2 unit and repeated observations (1 pretest screening + 4 weeks of consecutive progress monitoring) served as the level-1 unit. Across all models, the restricted maximum likelihood (REML) estimation method and unstructured covariance were employed. Predictors for the models were grade and treatment assignment (i.e., four times weekly, twice weekly, once weekly, or control). Treatment assignment was coded (1 to 4) so that variables representing group membership could be used to predict differences in students’ final scores and growth rates over time. The control group and grade 4 students were used as the reference groups. For parsimony, the 1st-level equation of students’ progress over time was combined with the 2nd level equation of differences in students’ performance as a function of a treatment assignment and grade (Singer & Willett, 2003). Progress monitoring scores were centered at the end of treatment (final session at end of five weeks), resulting in negative numbers when growth was positive. Table 3 (p. 9) displays the unconditional and final models across measures. The unconditional model was fit in order to determine whether students’ performances varied over time across each measure. The random effects parameters demonstrated that across measures, students’ scores significantly varied around the mean, and there were significant differences between each student’s observed and predicted scores over time. Final models were determined to be a better fit than the unconditional models because AIC and BIC values were lower (Singer & Willett, 2003).