System 44 Next Generation
Study: Beam et al. (2011)

Describe how students were selected to participate in the study:

A 2-step process was used to establish student eligibility for System 44. The Reading Inventory (RI) was used to screen students in Grades 4–8 who performed below the 50th percentile on the spring 2010 California Standards Test (CST) for System 44 eligibility. Those students who scored below 600 Lexiles on the RI were administered the Phonics Inventory (PI), a computer-based test used to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the PI were randomly assigned (stratified by school and grade level) to either the System 44 treatment group or the control group.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

A 2-step process was used to establish student eligibility for System 44. The Reading Inventory (RI) was used to screen students in Grades 4–8 who performed below the 50th percentile on the spring 2010 California Standards Test (CST) for System 44 eligibility. Those students who scored below 600 Lexiles on the RI were administered the Phonics Inventory (PI), a computer-based test used to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the PI were randomly assigned (stratified by school and grade level) to either the System 44 treatment group or the control group. Students who score in the Beginning and Developing reader categories of the Phonics Inventory are considered to be performing at or below the 25% percentile for their grade and in need of Tier III intervention.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

100.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

The program condition received the System 44 intervention.

Specify which condition is the control condition:

The control condition received "service as usual" . The interventions provided in the control condition varied by school. Elementary schools used a pull-out model, and the middle schools used a replacement model. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Houghton Mifflin Harcourt (formerly Scholastic Educational Technology and Services division) implemented System 44 in 11 elementary and middle schools in a large suburban school district in southern California during the 2010–2011 school year. The evaluation used a randomized trial design whereby RMC Research randomly assigned eligible students to a treatment (System 44) group or a control (―services as usual‖) group. A 2-step process was used to establish student eligibility for System 44. The Reading Inventory (RI) was used to screen students in Grades 4–8 who performed below the 50th percentile on the spring 2010 California Standards Test (CST) for System 44 eligibility. Those students who scored below 600 Lexiles on the RI were administered the Phonics Inventory (PI), a computer-based test used to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the PI were randomly assigned (stratified by school and grade level) to either the System 44 treatment group or the control group.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

Students who scored in the Beginning or Developing reader categories on the Phonics Inventory were randomly assigned (stratified by school and grade level) to either the System 44 treatment group or the control group.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

This evaluation used an intent-to-treat statistical model—a framework in which participants are analyzed within their initial random assignment group regardless of whether they actually received treatment. Because students were clustered within schools, a multilevel model was used to estimate the impact of the intervention on spring 2011 outcome scores while controlling for baseline score, ethnicity, English language proficiency, and school level. The model was run separately for each outcome measure. The school district provided individual student demographic data for grade level, sex, free or reduced-price meal eligibility, ethnicity, special education status, and English language proficiency. The following demographic covariates were included in the model: ethnicity (Caucasian), special education status, and sex. In addition, school level was added as a covariate. No data were missing for any of the demographic covariates. To assess the impact of System 44 on student outcomes, the evaluation team used hierarchical linear modeling, controlling for baseline scores and student demographic characteristics at Level 1 and school at Level 2.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Professional development for the System 44 teachers included a 1-day training in August 2010, 3 coaching visits from System 44 professional learning consultants during the 2010–2011 school year, and periodic district meetings that served as a System 44 professional learning community for participating teachers. The intervention teachers in the study had the following background: 92% female, with a mean of 12 years teachers experience; 33% had a Bachelor's degree, 42% had a Master's degree, and 25% had a specialist or Doctoral degree; in addition 8% had a reading specialist degree. Of the intervention teachers, 50% had previously taught System 44 and 83% had previously taught READ 180.

Describe when and how fidelity of treatment information was obtained.

The evaluation team assessed the fidelity of System 44 implementation in 2 ways: classroom observations (conducted in the fall, winter, and spring, classroom observations evaluated classroom setup, minutes of instruction, inclusion of program components, instructional management, and instructional delivery. Evaluation of the fidelity of implementation was also assessed based on teacher self-report in surveys and interviews. [Additional details are available in submitted report. Appendix A includes the fidelity measures (observation tools, surveys, interview questions.)]

What were the results on the fidelity-of-treatment implementation measure?

The evaluation team assessed fidelity of System 44 implementation through classroom observations and teacher self-report (surveys and interviews). Based on the classroom observation "classroom setup" was rated on 3 criteria: computers for at least one third of the class are accessible and functioning, auxiliary equipment (headsets, microphones, CD players) are accessible and functioning, and System 44 materials are easily accessible to students and teachers. The mean rating across these items was converted to a 4-point scale. In fall 2010, 58% of the 12 classrooms observed received the highest rating; this percentage increased to 75% in the winter and 100% of the classrooms in spring 2011. "Minutes of System 44 instruction" were consistent across all 3 observations. Although 9 of the 12 classrooms provided at least 55 minutes of instruction daily, 3 middle school classrooms provided between 45 and 49 minutes of instruction daily because the class period length did not allow for the allocation of 60 minutes for System 44 instruction. In regard to use of program components, the mean number of students using each type of material declined over time due to decreasing class size as students completed the program. Level of student engagement appeared to peak at the winter observation point. [See submitted report for additional detail] With regard to Fidelity of Implementation based on teacher self-report, the spring 2011 teacher survey respondents reported that students used the System 44 software between 20 and 30 minutes daily (average across classrooms: 25.3 minutes). The number of computers available in each classroom ranged from 5 to 20 (average 8.8). At the conclusion of the System 44 training in August 2010 and at the end of the 2010–2011 school year, the participating teachers were asked to respond to 10 questions designed to test their understanding of the System 44 program components. On average, the teachers answered 80% of the items correctly in fall 2010 and 81% of the items correctly in spring 2011. The teachers also rated their understanding of 5 key System 44 program components in the fall and spring (see Exhibit 9). Overall, the teachers reported a greater understanding of how to implement the program in spring 2011 than they had in fall 2010, but the differences were not statistically significant. In addition, the teachers’ responses varied less in the spring than in the fall, a finding that suggests most teachers believed they had a high level of understanding of how to use the System 44 program components.

Was the fidelity measure also used in control classrooms?

No

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Dosage impact was also examined but not reported above, as the results only apply to the program students. Because student use of the System 44 software is an essential component of the program, the evaluation team examined how students’ System 44 software usage was related to improved individual test scores, CST scores, and RI and PI scores by analyzing the outcome gains in relation to the following factors: program exit date and total number of series topics completed. Although the System 44 program was intended to be implemented for a full year, 26% of the treatment group students exited the program in winter (i.e., prior to March 2011), an additional 33% exited the program in spring (i.e., March or April 2011), and the remaining 41% stayed in the System 44 program through the end of the school year (i.e., May or June 2011). The System 44 students were expected to complete all 160 topics covered in the 25 series that compose the software component of the program. The analysis conducted by the evaluation team revealed that approximately half of the System 44 students completed all 160 topics (average number of topics completed: 133). The students were categorized by the number of topics completed: fewer than 100 topics (n = 43), between 100 and 159 topics (n = 37), and all 160 topics (n = 92). Repeated measures ANOVA models were used to assess differences between the 3 topic completion groups and change in outcomes. The ANOVA results showed significant differential gains between the 3 groups on the following tests: Woodcock-Johnson III Word Identification (p < .05), PI Sight Word Fluency (p < .001), PI Nonsense Word Fluency (p < .001), and PI Overall Fluency (p < .001). Specifically, those students who completed fewer than 100 topics showed significantly less gain on these 4 tests than the students who completed between 100 and 159 topics and the students who completed all 160 topics (see Exhibits 21–24). In addition, on 2 of the 4 tests for which differential gains were observed (PI Sight Word Fluency and PI Overall Fluency), the students who completed all 160 topics showed significantly greater gains than the students who completed between 100 and 159 topics. Additional analyses examined differences between these 3 groups on baseline characteristics (see Exhibit H2). Those students who completed fewer than 100 topics had significantly lower pretest scores than the students who completed more than 100 topics on all tests with the exception of PI Letter Name Accuracy.

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

This evaluation used an intent-to-treat statistical model—a framework in which participants are analyzed within their initial random assignment group regardless of whether they actually received treatment. Because students were clustered within schools, a multilevel model was used to estimate the impact of the intervention on spring 2011 outcome scores while controlling for baseline score, ethnicity, English language proficiency, and school level. The model was run separately for each outcome measure. The following demographic covariates were included in the model: ethnicity (Caucasian), special education status, and sex. In addition, school level was added as a covariate. RMC Research used multilevel models to estimate the impact of System 44 on spring 2011 student outcome scores while controlling for fall 2010 scores, minority status (Caucasian/non-Caucasian), and special education status (special education/not special education). The same analytic model was run for each of 6 individually administered standardized tests: TOSREC, CTOPP Elision, Woodcock-Johnson III Word Identification, Woodcock-Johnson III Word Attack, TOWRE Sight Word Efficiency, and TOWRE Phonetic Decoding Efficiency. Subsequent analyses examined program impacts on CST English- Language Arts scaled scores and on RI and PI scores.