focusMATH Intensive Intervention
Study: Styers & Baird-Wilkerson (2011)

Describe how students were selected to participate in the study:

Magnolia Consulting recruited eleven schools to participate in the study. The schools responded to a posting on the Pearson website and to an email blast from MDR. Once the schools agreed to participate in the study, students were assessed using the KeyMath3 assessment. Students who scored below the 30th percentile on the KeyMath3 assessment were invited to participate in the study.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The norm-referenced KeyMath3 assessment was administered at the beginning of the year to students at each study site. Magnolia Consulting selected only those students who scored below the 30th percentile on the KeyMath3 to participate in the study.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

The treatment is the Savvas Learning Company (formerly Pearson) focusMath intervention program.

Specify which condition is the control condition:

The control is a combination of “business as usual” methods (9 schools) and “homegrown math intervention” (2 schools).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

This study used a randomized control trial (RCT) design, wherein evaluators randomly assigned students who qualified for math intervention to treatment and control groups. Therefore, within the same school, some students participated in the focusMATH program (treatment group), while other students did not receive a published math intervention program (control group). This design allowed evaluators to estimate the difference in math achievement between treatment and control groups and to determine if the difference was significant.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The majority of facilitators held a master’s degree (59%), a bachelor’s degree (18%), or a high school degree (18%). Facilitators had been teaching for an average of 12 years and had taught at their current school for an average of seven years. Each facilitator received 1.5 days of training. The first day occurred prior to the start of implementation, and was 7 hours. The second day occurred after 4 – 6 weeks of implementation, and was 3 – 4 hours. During the seven-hour program training, curriculum specialists reviewed the theory behind focusMATH, completed a program walk-through, discussed successful environments for implementing the program with fidelity, completed a model lesson, reviewed focusMATH lesson planning, and had a Questions and Answer session. The follow-up half-day training focused on specific needs and requests by each school. Trainings allowed curriculum specialists to ascertain the level of treatment fidelity and to work with teachers in reaching higher levels of program implementation. These procedures, with the exception of the research implementation guidelines, reflect Savvas Learning’s standard training protocol for the focusMATH program. Curriculum specialists shared their contact information with facilitators, and facilitators were encouraged to contact them if they had questions or required assistance after the training was complete.

Describe when and how fidelity of treatment information was obtained.

Evaluators assessed program fidelity through two observations over the course of the school year and weekly self-report logs. Implementation fidelity scores were equally weighted to include ratings from observations and weekly log data. Throughout the school year, focusMATH facilitators completed 10 to 15-minute weekly implementation logs gauging the breadth and depth of their use of the focusMATH program. On the logs, facilitators indicated: 1. the frequency and extent to which they implement specific focusMath components and materials, 2. how often they use the program’s additional resources, including assessments, and 3. their perceptions about the focusMath program. Facilitators also reported interruptions in their intervention instruction periods (e.g., fire drill, testing, field trips, etc.), as well as student attrition. The logs allowed evaluators to assess implementation fidelity through adherence to required program components and weekly exposure to the program. Evaluators aggregated weekly log data at the end of the study period to aid in determining each teacher’s level of program implementation. Evaluators used this score to compare teachers’ implementation relative to the implementation guidelines established at the beginning of the study. The benchmarks reflect student exposure to the focusMATH program and teacher adherence to the required daily program components (e.g., Concept Development, Guided Practice, Independent Practice). Evaluators averaged the teachers’ overall log implementation ratings with overall observation fidelity ratings in order to create a fidelity measure that included teacher and observer ratings. Evaluators used the overall implementation fidelity score in analyses examining treatment student performance on outcomes and to describe teachers’ fidelity. To measure lesson implementation, evaluators observed student and teacher actions during 40- minute intervention periods in the fall and spring. During the observation, evaluators completed a checklist of materials facilitators and students used during the observation (e.g., student workbooks, practice pages, manipulatives, etc.) and rated quality, adherence, and exposure across 22 indicators (e.g., teacher–student interactions, use of instructional strategies, lesson implementation, and student engagement). The protocol allowed evaluators to indicate the extent of observed occurrences and to record notes for validity and reliability purposes. Evaluators qualitatively and quantitatively used the data to triangulate other data sources and to provide observer ratings of implementation fidelity. Evaluators established inter-rater reliability on the observation protocol through a facilitator video of a focusMATH lesson with students. Following observation of the video and individual determination of ratings on the 22 indicators, evaluators conducted an in-depth debriefing of the video and ratings, coming to a consensus on the meaning of high versus low ratings on each of the indicators. The five evaluators and researchers involved in observations and ratings established a high degree of inter-rater reliability (average measures intraclass correlation = 0.91, Cronbach’s Alpha = 0.91). Evaluators observed all small groups instructed by treatment teachers and control teachers (if applicable) twice at all sites and twice in 10 out of 11 schools. One exception was at School A in Site 2. In the spring of 2012, the school had a last-minute weather-related closing and evaluators were not able to observe focusMATH and control groups (if applicable). Also in School A, one control teacher was not available for observation in the fall. Another exception was in Site 6, due to time limitations and scheduling constraints, evaluators observed 8 out of 11 groups instructed by the same study treatment teacher across two schools (School I and J) in the fall and spring (total of 16 observations). In the fall, evaluators observed a total of 21 out of 22 study teachers, 143 treatment and 12 control students. In the spring, evaluators observed a total of 18 out of 22 study teachers, 152 treatment and 11 control students. The number of control students observed was less than the number of observed treatment students because the majority of control students in the study (92%) did not receive pull-out math intervention.

What were the results on the fidelity-of-treatment implementation measure?

The implementation fidelity grand mean for 14 third grade and 16 fifth grade focusMATH teachers was 87%, indicating that the facilitators implemented the program with high fidelity. Overall, 93% (n = 28) of facilitators implemented the program with high fidelity. One facilitator with a third and fifth grade classroom implemented the program with moderate fidelity.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

The KeyMath3 assessment also includes subtests of Algebra and Data Analysis & Probability (under Basic Concepts) and Applied Problem Solving (under Applications). These 3 subtests were not included in the study because focusMath does not address them. The test publisher provided an algorithm to derive an adjusted Total GV score with the exclusion of these three subtests.

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Evaluators conducted a multilevel model with students nested in teachers/classrooms in order to examine the impact of focusMATH on posttest achievement scores. Because random assignment occurred at the student level and the intervention was received at the student level, the student level of the model included an indicator variable for study condition, control or focusMATH. The model included the following student covariates: pretest KeyMATH3 Total GSV score, pretest self-efficacy score, gender, ethnicity, free or reduced lunch status, special education status, and grade level. The model also contained several teacher/classroom covariates, including, core math teacher’s years of teaching experience, core math classroom size, time spent in core math instruction and school. For full results, see Appendix F, Table F4.