Spring Math

Study: VanDerHeyden, McLaughlin, Algina, & Snyder (2012)

VanDerHeyden, A. M., McLaughlin, T., Algina, J., & Snyder, P. (2012). Randomized evaluation of a supplemental grade-wide mathematics intervention. American Education Research Journal, 49, 1251-1284. http://aer.sagepub.com/cgi/reprint/49/6/1251?ijkey=CHbWMLJp8/kRc&keytype=ref&siteid=spaer

Descriptive Information

Usage

Acquisition and Cost

Program Specifications and Requirements

Training

Spring Math is a web-based RTI system for mathematics.

 

Please note: As an RTI system, Spring Math include screening, progress monitoring, and intervention however, NCII has only reviewed the intervention component for the purposes of the Academic Intervention Tools Chart.

Spring Math is intended for use in Kindergarten through eighth grade. The program is intended for use with any student at risk of academic failure. The academic area of focus is math (including computation, concepts, word problems, algebra, problem solving, decimals, and fractions).

Where to Obtain: 
1667 Snelling Ave. N.
St. Paul, MN 55108
Phone: 651-999-6000
Web Site: www.springmath.com

Cost: The initial cost for implementing Spring Math is $7.00 per student, and the replacement cost per student for subsequent use is $7.00.

There are no additional costs or requirements for implementation. The annual per student cost includes initial setup, universal screening and progress monitoring assessments, interventions, and related reporting.

Support documentation is also included.

It is recommended that Spring Math is used with individual students or small groups 15-20 minutes per session, five days a week, for 15-32 weeks.

The program includes a highly specified teacher’s manual

Teachers and administrators interact with Spring Math online. When the teacher logs in to his or her account, the teacher dashboard provides a screening packet for the teacher to print. The packet contains assessments with standardized directions printed on each page in the form of a script for the teacher. The teacher administers and scores the screening measures using answer keys that are provided by the system and then enters the scores into the dashboard. Spring Math provides graphs of students’ performance and if the criterion is met (median score in the risk range on the first two screening measures), recommends a classwide intervention. The teacher dashboard now directs classwide intervention, providing a printable intervention packet that contains the intervention protocol and all materials needed to conduct the classwide intervention for one week. Each week, the teacher enters scores from the 5th session, Spring Math graphs the performance of students, and advances the class to the next target skill or stays on the current skill level. Student progress monitoring data are tracked and used to recommend individual students for more intensive (Tier 3) intervention where needed and the system provides all necessary materials to conduct Tier 3 intervention.

Training is required for the instructor and will take less than one hour to complete.

Training is provided by video tutorials and an online user manual. Topics covered in the tutorial and manual include how to conduct the screening and how to conduct classwide and individual intervention. The research basis for the assessments and interventions are detailed in the online manual. An FAQ section is included.  Implementation support includes: system interpretation of data, provision of all needed assessment and intervention materials, and a coach dashboard that tracks intervention use schoolwide and student learning gains and then directs the coach to support intervention where needed for better results.

 

 

Participants: Convincing Evidence

This article uses two samples in grades 4 and 5. Sample a includes all students with proximal measure scores in the intervention year for intervention and control students. Sample b includes students with two consecutive years of state accountability test scores in mathematics

Sample size: All fourth and fifth grade students attending any of the 7 schools in a southeastern U.S. city were eligible for participation. Inclusion criteria were (1) enrolled in the system at the time of spring testing in the spring of the intervention year (spring score available), (2) not categorized as limited English proficient according to state criteria, and (3) participating in general education mathematics instruction. From the original sample, 254 fifth graders and 283 fourth graders met inclusion criteria. Among these included students, 186 fifth graders and 188 fourth graders had a preceding year’s spring test score available (pre-test score) on the year-end state test. Thus the final sample for the distal measure outcomes was 186 5th graders and 188 4th graders. The final sample for the proximal outcome measures was 254 fifth graders and 283 fourth graders.

Risk Status: Classes for which the median score was in the at-risk range on the fall occasion proximal measures were considered as eligible for intervention. 100% of classes screened met the risk criterion in both the control and intervention groups.

Demographics:

 

Program

Control

Cox Index

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

168a

105b

53a

50b

115a

83b

52a

51b

0.02

0.02

  Grade 5

148a

107b

47a

50b

106a

79b

48a

49b

0.02

0.02

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

123a

79b

39a

37b

79a

53b

36a

33b

0.08

0.11

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

16a

12b

5a

6b

14a

11b

5a

7b

0.00

0.10

  Hispanic

20a

12b

6a

6b

6a

5b

3a

3b

0.44

0.44

  White

157a

109b

50a

51b

122a

93b

55a

57b

0.12

0.15

  Other

 

 

 

 

 

Socioeconomic status

  Subsidized lunch

182a

121b

58a

57b

124a

90b

56a

56b

0.05

0.02

  No subsidized lunch

132a

91b

42a

43b

97a

72b

44a

44b

0.05

0.02

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

33a

20b

10a

9b

29a

20b

13a

12b

0.18

0.19

  Not identified with a disability

282a

191b

89a

90b

192a

142b

87a

88b

0.12

0.12

ELL status

  English language learner

 

 

 

 

 

  Not English language learner

 

 

 

 

 

Gender

Female

156a

100b

49a

47b

108a

 76b

49a

47b

0.00

0.00

Male

160a

112b

51a

53b

113a

86b

51a

53b

0.00

0.00

aFull Sample which was used for the CBM or targeted measures. bSample included students with a pre-test state accountability test score in math and a post-test state accountability test score in math (two consecutive years).

Training of Instructors: Teachers, RtI coordinators, and administrators were trained to implement the intervention using a combination of antecedent and live coaching strategies. Following a series of trainings specific to principals, the first author traveled to each school to conduct a 1-hour training with teachers whose classes were assigned to the intervention condition. Additionally, each school had an RtI coordinator and that person was charged with receiving and organizing weekly data to provide to the first author via an electronic spreadsheet program designed to organize the data and present graphs of class progress each week during the intervention. In the didactic training session, an overview of the rationale for the intervention program was shared with teachers using the district’s data reflecting low mathematics achievement. Details of the intervention were provided, including sharing the intervention protocol, describing how the intervention would progress based on student mastery of skills within a pre-established hierarchy of skills, showing effects on mathematics achievement obtained in other districts using the same intervention, and showing short video clips of the intervention being implemented in classrooms in other districts. An opportunity to discuss and troubleshoot intervention implementation was provided to teachers at this time. Teachers were provided all materials needed to implement the intervention each week by the on-site RtI coordinators. School principals agreed to conduct implementation integrity checks via direct observation as part of the intervention plan (described in greater detail in the next section). The consultant organized feedback on district progress for district administrators and school principals bimonthly during the year. Graphed feedback on each class’s progress with the intervention was provided to principals and district administrators. The consultant met in person with the district leaders and principals, reporting the number of skills mastered by teacher and identifying implementation errors. Additionally, the consultant communicated directly with principals and RtI coordinators providing a list of teachers whose classes were growing at a slower pace relative to other classes in the same school and encouraged an intervention integrity check in those classes. Finally, on a bimonthly basis (four total occasions), the consultant conducted integrity observations in classrooms with each principal and modeled for school principals how to troubleshoot intervention implementation with the classroom teacher. School principals agreed to conduct implementation integrity checks via direct observation as part of the intervention plan. Principals or on-site RtI coordinators agreed to conduct four integrity observations each week with approximately half of those occurring during regular mathematics instruction within control classrooms in an attempt to capture contamination between control and intervention conditions. The intervention integrity checklist listed each step of the intervention in observable terms and administrators were trained to observe and note the occurrence of each step of the intervention. The trained observer used the scripted intervention protocol to note correctly and independently completed steps of the intervention. Where deviations from the protocol were observed in intervention classrooms, principals and/or RtI coordinators provided corrective feedback on implementation and assisted the teacher to troubleshoot barriers to effective implementation.

Design: Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: N/A

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Analysis corrected for pretest differences.

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: N/A

Were the program and control groups demographically comparable at pretreatment?: Yes

Was there attrition bias1 ?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: We examined fidelity in three ways. (1) We conducted a survey at the midpoint of the intervention of all treatment and control teachers. The survey documented by teacher report characteristics of core instruction in mathematics (number of minutes, type of supplemental intervention if any, number of minutes supplemental intervention was provided, etc.). These findings are reported on p. 1257 of the manuscript under the heading, “Instructional Context” and in Table 2 on p. 1258. Contamination was documented with two teachers in the control group reporting that they had implemented the classwide intervention.

(2) We also examined use of the intervention in treatment classrooms using permanent products. We examined the number of skills mastered during intervention by class since classes were equivalent at the start of intervention and the intervention used a standard protocol with classes advancing through a fixed sequence of skills as a class to a new skill based upon the median score reaching a mastery criterion associated with the skill. Because a weekly score was recorded for all students in all treatment classes and the class was expected to advance based upon a specific decision rule, we also computed number of deviations from the treatment plan during the 29 weeks of the study (intervention score recorded, class advanced or did not advance as prescribed). (3) Trained observers (coach or principal) conducted a total of 406 direct observations of the intervention across all classrooms (treatment and control) during math lessons and documented occurrence of the steps in the standard protocol for classwide intervention. Each step was coded as having been implemented or not and the total number of steps implemented was divided by the total number of intervention steps in the protocol and the quotient was multiplied by 100%. 281 observations were conducted in treatment classrooms representing about 8% of opportunities for intervention among intervention teachers (26 weeks of intervention completed on average times 5 sessions per week times 26 intervention teachers). 125 observations were conducted in control classrooms. Average integrity in treatment classrooms was 96.69% (83.5%-100%) and estimated average implementation in control classrooms was 2.69% (0-20%). No feedback was provided to teachers following completion of the observation checklist. What we learned from direct observation data was that the most common integrity error was failing to use the intervention at all, which we suspected direct observation was less sensitive to given that the presence of the observer likely cued or reminded the teacher to implement the intervention. When the intervention was used, teachers tended to implement all of the steps of the intervention correctly. Thus, in our two published articles, we emphasized permanent product estimates of integrity including weeks of intervention and skill progression (which we thought of as trials to criterion data).

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: Teachers completed on average 26.1 weeks of intervention (range, 20-29). Intervention teachers deviated from the intervention plan on average 1.4 times during the intervention (range, 0-5). On average, correct decisions (advance a skill level or not) were made for 94% of decision-making occasions (range, 80-100%). These permanent product indicators indicated that the intervention was sufficiently used to measure its effect on average although between teacher differences were apparent. Results on p. 1272 of the article under the heading “Intervention Integrity Effects” indicate that fourth grade classes successfully mastered more skills compared to fifth grade classes. There was a positive, but not statistically significant relationship between integrity estimates and post-intervention year-end state accountability scores. There was a positive and statistically significant relationship between integrity estimates and CBM growth in the intervention group.

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

Alternate form correlations for students in the control group from fall to winter was r = 0.77 (n = 106) and r = 0.78 for winter to spring (n = 105).

Intervention targets (in addition to other skills) simple and complex operations with addition and subtraction. Mastery of this skill reflects a child’s understanding of addition, subtraction, and place value.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 4th grade students are expected to have mastered.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

Multiplication 0-12 (Grade 4)

Alternate form correlations for students in the control group from fall to winter was r = 0.64 (n = 104) and r = 0.78 (n = 105) for winter to spring.

Intervention targets (in addition to other skills) simple and complex operations with multiplication. Mastery of this skill reflects a child’s understanding of multiplicative reasoning and rapid identification of factors.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 4th grade students are expected to master.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

Fact Families Multiplication & Division 0-12 (Grade 4)

Alternate form correlations for students in the control group from fall to winter was r = 0.61 (n = 104) and r = 0.83 (n = 105) for winter to spring.

Intervention targets (in addition to other skills) simple and complex operations with multiplication and division. Mastery of this skill reflects a child’s understanding of the conceptual relationship of division and multiplication, specifically division as solving for an unknown factor, which is an important precursor to proportional reasoning.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 4th grade students are expected to master.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

Fact Families Multiplication & Division 0-12 (Grade 5)

Alternate form correlations for students in the control group was r = 0.70 (n = 99) for fall to winter and r = 0.76 (n = 99) for winter to spring.

Intervention targets (in addition to other skills) simple and complex operations with multiplication and division. Mastery of this skill reflects a child’s understanding of the conceptual relationship of division and multiplication, specifically division as solving for an unknown factor, which is an important precursor to proportional reasoning.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 5th grade students are expected to have mastered.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

Multiply 2 by 3 Digits without Regrouping (Grade 5)

Alternate form correlations for students in the control group was r = 0.44 for fall to winter (n = 100) and r = 0.74 for winter to spring (n = 99).

Intervention targets (in addition to other skills) complex operations with multiplication. Mastery of this skill reflects a child’s understanding of multiplication and place value.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 5th grade students are expected to have mastered.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

Simplify Fractions (Grade 5)

Alternate form correlations for students in the control group was r = 0.83 for fall to winter (n = 96) and r = 0.71 for winter to spring (n = 97).

Intervention targets (in addition to other skills) operations with fractions and conversion of fraction quantities. Mastery of this skill reflects a child’s emerging understanding of proportion and creating equivalent proportions.

Teachers are responsible for teaching mathematical content reflected in their state standards. This skill is a skill that 5th grade students are expected to have mastered.

All students (treatment and control) were exposed to this measure at fall, winter, and spring screening.

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

IOA = 98.33% (98%-99%)*

Same as noted above.

Same as noted above.

Multiplication 0-12 (Grade 4)

IOA = 97.33% (97%-98%)*

Same as noted above.

Same as noted above.

Fact Families Multiplication & Division 0-12 (Grade 4)

IOA = 97.33% (97%-98%)*

Same as noted above.

Same as noted above.

Fact Families Multiplication & Division 0-12 (Grade 5)

IOA = 98.67% (98%-99%)*

Same as noted above.

Same as noted above.

Multiply 2 by 3 Digits without Regrouping (Grade 5)

IOA = 93% (88%-96%)*

Same as noted above.

Same as noted above.

Simplify Fractions (Grade 5)

IOA = 98.67% (98%-99%)*

Same as noted above.

Same as noted above.

Broader  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009)

Cronbach’s Alpha = 0.88 for students in Grades 4 and 5

Program instructional content narrowly samples grade level standards and guides students to mastery. Thus, instructional content is included in this measure, but the measure is broader reflecting the full content of the state standards and requiring application of learned skills to successfully complete items.

This is the state-adopted annual accountability assessment that assesses the extent to which children have learned the content and skills that are reflected in the state standards.

 

All students received this measure.

 

Number of Outcome Measures: 6 Math

Mean ES - Targeted: 0.68*

Mean ES - Broader: 0.10

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Math

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

0.61***

Math

Multiplication 0-12 (Grade 4)

0.68***

Math

Fact Families Multiplication & Division 0-12 (Grade 4)

0.73***

Math

Fact Families Multiplication & Division 0-12 (Grade 5)

0.81***

Math

Multiply 2-digit by 3-digits without Regrouping (Grade 5)

0.68***

Math

Simplify Fractions (Grade 5)

0.58***

Broader Measures

Construct

Measure

Effect Size

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 4)

0.22

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 5)

-0.02

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: Yes

Children on IEP

Targeted Measures

Construct

Measure

Effect Size

Math

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

0.28

Math

Multiplication 0-12 (Grade 4)

0.67

Math

Fact Families Multiplication & Division 0-12 (Grade 4)

0.56

Math

Fact Families Multiplication & Division 0-12 (Grade 5)

0.30

Math

Multiply 2-digit by 3-digits without Regrouping (Grade 5)

0.73

Math

Simplify Fractions (Grade 5)

0.19

Broader Measures

Construct

Measure

Effect Size

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 4)

0.35

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 5)

0.01

Children who received free or reduced-price lunch

Targeted Measures

Construct

Measure

Effect Size

Math

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

0.60***

Math

Multiplication 0-12 (Grade 4)

0.76***

Math

Fact Families Multiplication & Division 0-12 (Grade 4)

0.66***

Math

Fact Families Multiplication & Division 0-12 (Grade 5)

0.69***

Math

Multiply 2-digit by 3-digits without Regrouping (Grade 5)

0.55**

Math

Simplify Fractions (Grade 5)

0.51**

Broader Measures

Construct

Measure

Effect Size

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 4)

0.30

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 5)

0.11

Children who were not in the racial majority category (i.e., white/Caucasian)

Targeted Measures

Construct

Measure

Effect Size

Math

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

0.58**

Math

Multiplication 0-12 (Grade 4)

0.58**

Math

Fact Families Multiplication & Division 0-12 (Grade 4)

0.62***

Math

Fact Families Multiplication & Division 0-12 (Grade 5)

0.69***

Math

Multiply 2-digit by 3-digits without Regrouping (Grade 5)

0.62**

Math

Simplify Fractions (Grade 5)

0.55**

Broader Measures

Construct

Measure

Effect Size

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 4)

0.27

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 5)

0.00

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Disaggregated Data for <20th Percentile: Yes

Low-achieving students, falling at or below the 20th percentile on pre-test measures of achievement

Targeted Measures

Construct

Measure

Effect Size

Math

3-Digit Addition & Subtraction with & without Regrouping (Grade 4)

0.56

Math

Multiplication 0-12 (Grade 4)

1.11**

Math

Fact Families Multiplication & Division 0-12 (Grade 4)

0.90*

Math

Fact Families Multiplication & Division 0-12 (Grade 5)

0.63

Math

Multiply 2-digit by 3-digits without Regrouping (Grade 5)

0.66

Math

Simplify Fractions (Grade 5)

0.98**

Broader Measures

Construct

Measure

Effect Size

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 4)

0.79*

Math

Mississippi Curriculum Test, 2nd Edition (Pearson, 2009) (Grade 5)

-0.05

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Administration Group Size: Individual, Small Group, (n=4-30)

Duration of Intervention: 15-20 minutes, 5 times a week, 15-32 weeks

Minimum Interventionist Requirements: Paraprofessional, Less than 1 hour of training

Reviewed by WWC or E-ESSA: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

Other Research: Potentially Eligible for NCII Review: 0 studies