System 44 Next Generation
Study: Beam et al. (2012)

Describe how students were selected to participate in the study:

The target sample was selected based on performance on the fall 2011 Michigan Educational Assessment Program (MEAP) or spring 2011 AIMSweb assessment. The Scholastic Reading Inventory (SRI) was used to screen students who performed below the 25th percentile on either the MEAP or AIMSweb assessments. Students who scored below 600 Lexiles on the SRI were subsequently administered the Scholastic Phonics Inventory (SPI) to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the SPI were randomly assigned to the System 44 treatment group or control group.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The target sample was selected based on performance on the fall 2011 Michigan Educational Assessment Program (MEAP) or spring 2011 AIMSweb assessment. The Scholastic Reading Inventory (SRI) was used to screen students who performed below the 25th percentile on either the MEAP or AIMSweb assessments. Students who scored below 600 Lexiles on the SRI were subsequently administered the Scholastic Phonics Inventory (SPI) to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the SPI were randomly assigned to the System 44 treatment group or control group.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

System 44

Specify which condition is the control condition:

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. Please see The Counterfactual in the Program Implementation Findings section of the attached report for more detail (p. 11).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

N/A

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

A total of 368 students met the eligibility criteria for System 44 and were randomly assigned to the treatment group (n = 187) or the control group (n = 181). A total of 195 students (53% of the students who were randomly assigned) had a designated learning disability. Exhibit 2 summarizes the characteristics of all treatment and control group students in the randomized sample and those students who were included in the analytic sample. The evaluation team conducted equivalence tests on key factors to determine whether differences between the treatment and control groups existed at baseline. Appendix A provides baseline equivalence test results for the analytic sample. Overall, the treatment and control groups in both the randomized and analytic samples were equivalent in terms of sex, eligibility for free or reduced-price meals, English language proficiency, special education status, ethnicity, and baseline MEAP scores.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The teacher survey administered to both the treatment group and control group solicited information about respondents’ teaching experience, education, certifications, and prior experience with Scholastic’s System 44 and READ 180 programs. Most of the System 44 teachers (96%) were female, and 89% of the control group teachers were female. System 44 teachers had more teaching experience than the control group teachers and were more likely to have a degree beyond a Bachelor’s degree. The System 44 teachers were also more likely to have a full teacher certification (Professional Education Certificate). Both groups had a few teachers with experience teaching READ 180 and 1 control group teacher had previous experience teaching System 44. See Exhibit 3 (p. 12) of the attached report for more detail. Professional development for the System 44 teachers included a 1-day training in September, 2011, follow-up training in November, 2011 and monthly coaching visits from Scholastic staff from December 2011 through May 2012. Please see Professional Development in the Program Implementation Findings section of the attached report (p. 12).

Describe when and how fidelity of treatment information was obtained.

Classroom observations were conducted by evaluators 3 times during the school year—in fall 2011, winter 2012, and spring 2012—in order to collect data on classroom‑level implementation. Implementation fidelity components included (a) classroom setup, (b) minutes of instruction on software and in small groups, (c) inclusion of all program components (e.g., the 44Book, System 44 paperback books, and the Decodable Digest), and (d) instructional management and delivery.

What were the results on the fidelity-of-treatment implementation measure?

Classroom implementation fidelity included 4 components: classroom setup, minutes of instruction, inclusion of program components, and instructional management and delivery. For (a) classroom setup, in fall 2011, 36% of the 20 classrooms observed received the highest rating; this percentage increased to 67% in the winter and 78% of classrooms in spring 2012. For (b) minutes of instruction, the Scholastic staff intended teachers to use the System 44 model in a 60‑minute class period. Across all 3 observation time points, 72% to 82% of the 20 classrooms provided at least 55 minutes of instruction daily. For (c) inclusion of program components, observations of instructional materials used during teacher‑led small group activities indicated that in the fall, average level of use was 1.8, in the winter average use was 1.7, and in the spring average use was also 1.7. For (d) instructional management and instructional delivery, 6 aspects of instructional management were rated, such as teacher’s ability to maintain a positive learning environment, monitor students, keep students on task, provide smooth transitions between rotations, use flexible student groups, and differentiate support based on student needs. Five aspects of instructional delivery were rated, such as pacing, appropriateness of the delivery for student skill level, teacher preparation, active engagement of students, and overall lesson execution. For instructional management, average score in the fall was 3.1, winter was 2.8, and spring was 2.8. For instructional delivery, average score in the fall was 3.0, winter was 3.4, and spring was 2.7.

Was the fidelity measure also used in control classrooms?

No, the control classrooms were not observed.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

This evaluation used an intent-to-treat statistical model. Because students were clustered within schools, a multilevel model was used to estimate the impact of the intervention on spring 2012 outcome scores while controlling for baseline score and demographic characteristics. In addition, school level was added as a covariate. To assess the impact of System 44 on student outcomes, the evaluation team used hierarchical linear modeling, controlling for baseline score, ethnicity (African American), English language learner status, and special education status at Level 1 and school at Level 2. The 2-level model for estimating the impact of System 44 on change in outcomes follows: Level 1 Model: Yij = β0j + β1j(Trtij) + β2j(Pretestij) + β3j(LDij) + β4j(Ethnicityij) + β5j(ELLij) + β6j(Levelij) + ij Level 2 Model: β0j = 00 + 0j β1j = 10 β2j = 20 β3j = 30 β4j = 40 β5j = 50 β6j = 60 where: ij = the random error term for student i in school j 0j = the random intercept term for school j and: Yij = the posttest score for student i in school j Trtij = the treatment indicator for student i in school j (0 = control, 1 = treatment) Pretestij = the posttest score for student i in school j LDij = the special education indicator for student i in school j (0 = no special education classification, 1 = learning disabled classification) Ethnicity ij = the ethnicity indicator for student i in school j (0 = non-African American, 1 = African American) ELLij = the English language learner indicator for student i in school j (0 = non-English language learner, 1 = English language learner) Levelij = the school level indicator for student i in school j (0 = middle, 1 = elementary) A similar 2-level model was used to assess the impact of System 44 on change in outcomes in the learning disabled sample; the model was the same as that specified above except for the removal of the special education indicator.