MATH 180
Study: HMH (2014)

Describe how students were selected to participate in the study:

Students were selected from four distinctive geographic districts. There were 10 schools using the MATH 180 program and a matched set of 10 additional schools that served as comparisons. Student grade level ranged from 6th through 8th. All students in the study were identified by their district/school as being at risk in the area of mathematics based on factors including previous year performance on state assessments, grades NWEA MAP and the Math Inventory. As a result these students were selected in both the MATH 180 and comparison schools for additional intervention math support.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All students in the study were identified by their district/school as being at risk in the area of mathematics based on factors including not passing their state assessment or previous math course, NWEA MAP and the Math Inventory. Students were on average below the 25th percentile at the start of the study. Students in the study were in the 6th through 8th grade. The 25th percentile for the 6th grade is 645Q, 7th and 8th grade are even higher at 665Q and 730Q respectively.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

MATH 180 Course I

Specify which condition is the control condition:

Business as usual locally derived intervention. Consisted of ad hoc use of programs such as Math Navigator, Math lab, Triumphs, 1XL and teacher developed materials.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

This evaluation utilized a quasi-experimental design to estimate the impact of the MATH 180 program on math achievement in Grades 6-8. The MATH 180 program was implemented during the 2013-2014 school year in four districts across four states in the 2014-2015 school year. A total of 10 schools from these districts implemented MATH 180 in their schools for all eligible students. Eligible comparison school students were identified or matched within district on the same criteria used to identify students in the MATH 180 schools. To the extent possible, schools were also matched on the percentage of English Language Learners (ELL), students with disabilities, and students eligible for Free or Reduced Lunch (FRL) program to ensure that groups were comparable on these demographic variables.

What was the unit of assignment?

Other

If other, please specify:

Schools and teachers were randomly assigned

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Math 180 teachers received 3 days of training on the program and periodic coaching to reinforce the quality of the program’s implementation. The background of the teachers is summarized below: Gender: Female: MATH 180 Schools Percent= 74%; Comparison Schools Percent= 20% Male: MATH 180 Schools Percent= 26%; Comparison Schools Percent= 80% Employment Status: Classroom Teacher: MATH 180 Schools Percent= 96%; Comparison Schools Percent= 100% Special Ed Teacher: MATH 180 Schools Percent= 4%; Comparison Schools= 0% Teaching Experience: Total years of experience: MATH 180 Schools M= 8.88, SD=10.04; Comparison Schools M=6.20, SD=5.54 Years in this district: MATH 180 Schools M=6.33, SD=6.83; Comparison Schools M=2.40, SD=2.61 Years at current school: MATH 180 Schools M=3.20, SD=3.16; Comparison Schools M=1.40, SD=0.55 Education (Highest Degree): Bachelors: MATH 180 Schools Percent=50%; Comparison Schools Percent= 20% Masters: MATH 180 Schools Percent= 50%; Comparison Schools Percent= 80% Certifications: Standard or Professional: MATH 180 Percent= 74%; Comparison Schools Percent= 20% Initial or Provisional: MATH 180 Percent= 26%; Comparison Schools Percent= 80% Grade(s) Teaching Core Math: 6: MATH 180 Percent= 37%; Comparison Schools Percent= 60% 7: MATH 180 Percent= 44%; Comparison Schools Percent= 60% 8: MATH 180 Percent= 26%; Comparison Schools Percent= 20% Grade(s) Teaching Math Intervention This Year: 6: MATH 180 Percent= 41%; Comparison Schools Percent= 80% 7: MATH 180 Percent= 41%; Comparison Schools Percent= 80% 8: MATH 180 Percent= 22%; Comparison Schools Percent= 40% Grade(s) Teaching Math Intervention Last Year: 6: MATH 180 Percent= 22%; Comparison Schools Percent= 0% 7: MATH 180 Percent= 19%; Comparison Schools Percent= 0% 8: MATH 180 Percent= 15%; Comparison Schools Percent= 0%

Describe when and how fidelity of treatment information was obtained.

Levels of program implementation fidelity were tracked in multiple ways. The research team utilized three methods for triangulating the level of program implementation; 1) student software activity (which accounted for use of software and teacher progress via mSkills assessments), 2) classroom observations and 3) a teacher survey Program fidelity was monitored through an examination of the student’s software data which captures their number of sessions, time spent, activity and performance in each of the program’s zones. In addition, mSkills assessments which occur at the end of each teacher led instructional block were captured through the software. This data yielded how many instructional blocks the teacher completed and how each student performed on the assessment. In addition, teacher surveys, interviews with math coaches and periodic classroom observations were conducted. Each school also received 6-8 visits by Math Solutions coaches throughout the study year. These coaches provided regular guidance to teachers and administrators on implementing the program with fidelity.

What were the results on the fidelity-of-treatment implementation measure?

As reported in the research update “Early Outcome Effects of a Blended Learning Model for Math Intervention Instruction With Special Population Students” average progress through the blocks for SPED, EL and FRL was 3.43, 4.34, and 3.88 respectively. These are within the normal and expected ranges for MATH 180. Teacher facilitated instruction progressed to block 5 which was also within the normal and expected range of progress for the program. Teachers were also observed at three separate times throughout the study year although there was no second observer so intercoder agreement was not applicable.

Was the fidelity measure also used in control classrooms?

Yes, the surveys, interviews and observations.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

See attached study pages 3 - 6