System 44 Next Generation

Study: Beam, Faddis, & Hahn (2011)

Beam, M., Faddis, B. J. & Hahn, K. (2011). Evaluation of System 44 in Murrieta Unified School District, CA. Unpublished Technical Report.

Descriptive Information

Usage

Acquisition and Cost

Program Specifications and Requirements

Training

System 44 Next Generation is the proven foundational reading program designed for the most challenged readers in Grades 3–12+. System 44 Next Generation is proven to help students master the foundational reading skills required for success with the rigorous state standards through explicit instruction in phonics, comprehension, and writing. System 44 integrates principles of cognition and learning with practices for instructional effectiveness for older struggling readers.

System 44 uses state-of-the art adaptive Software to deliver a personalized learning progression with explicit, research-based phonics instruction. Explicit, teacher-led instruction in close reading, comprehension, academic vocabulary and writing provides students with the skills needed to succeed with the rigorous state standards, college, and career.

System 44 Next Generation is intended for use in grades 3-12. The program is designed for use with students with learning disabilities, emotional or behavioral disabilities, English language learners, and any student at risk of academic failure.

 

The academic area of focus is reading (including phonological awareness, phonics/word study, comprehension, fluency, vocabulary, spelling, oral language development, and word analysis) and writing (including spelling, sentence construction, planning and revising, summary, informative and argument writing).

Where to obtain: 
125 High Street, Suite 900

Boston, MA 02110-2777

Website: hmhco.com/system44

System 44 Next Generation Upper Elementary and Secondary is a blended subscription model. As a blended instructional subscription model, System 44 Next Generation includes a teacher subscription, student subscription, classroom materials and implementation best practice services.

The System 44 Next Generation student subscription includes the student application, a student book (S44 student book) and our HMH hosting services. The teacher subscription includes access to the System 44 Next Generation Teacher Central Application. The classroom materials consist of paperbacks, audiobooks, ebooks and teacher resources for differentiation of instruction. The System 44 Next Generation purchases is supported with wrap round implementation best practices consisting of getting started for new teachers, in-person and virtual coaching, reporting and data analytics. Cost varies based on purchase and depends upon number of intervention students, classrooms, and intervention teachers.

System 44 Next Generation is designed for use with small groups of five to seven students.

The program takes 60 minutes per session with a recommended five sessions per week.

The program includes highly specified teacher manuals.

System 44 is a blended instructional model, where a half of the class is on the computer during the small-group rotations. Therefore, the number of computers depends on the number of students in the System 44 classroom. Internet connection is also required.

Houghton Mifflin Harcourt offers comprehensive professional learning and support for participating System 44 teachers and leaders. Houghton Mifflin Harcourt provides a day of upfront, in-person teacher training with the purchase of a complete stage of System 44. Houghton Mifflin Harcourt coaches also provide additional training, as well as in person and/or online coaching. System 44 also includes program-embedded professional learning resources, including a comprehensive suite of resources and tools for assessing students, differentiated instruction with both print and online materials, and classroom video models for key lesson types.

 

Participants: Convincing Evidence

Sample size: 344 students (173 program students and 171 control students)

Risk Status: A 2-step process was used to establish student eligibility for System 44. The Reading Inventory (RI) was used to screen students in Grades 4–8 who performed below the 50th percentile on the spring 2010 California Standards Test (CST) for System 44 eligibility. Those students who scored below 600 Lexiles on the RI were administered the Phonics Inventory (PI), a computer-based test used to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the PI were randomly assigned (stratified by school and grade level) to either the System 44 treatment group or the control group. Students who score in the Beginning and Developing reader categories of the Phonics Inventory are considered to be performing at or below the 25% percentile for their grade and in need of Tier III intervention.

Demographics:

 

Program Number

Program Percentage

Control Number

Control Percentage

Cox Index

Grade level

 

 

 

 

 

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

73

 

72

 

0.00

  Grade 5

25

 

27

 

0.10

  Grade 6

43

 

39

 

0.07

  Grade 7

20

 

21

 

0.00

  Grade 8

12

 

12

 

0.00

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

15

 

18

 

0.14

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

10

 

9

 

0.12

  Hispanic

29

 

26

 

0.09

  White

107

 

109

 

0.05

  Other

11

 

7

 

0.26

Socioeconomic status

  Subsidized lunch

58

 

56

 

0.03

  No subsidized lunch

97

 

103

 

0.10

Disability status

  Speech-language impairments

 17

 

21 

 

 0.12

  Learning disabilities

12

 

13

 

 0.09

  Behavior disorders

 

 

 

 

 

  Mental retardation

 

 

 

 

 

  Other

 24

 

 16

 

0.03 

  Not identified with a disability

120

 

121

 

0.06

ELL status

  English language learner

59

 

46

 

0.20

  Not English language learner

114

 

124

 

0.20

Gender

Female

67

 

75

 

0.12

Male

106

 

95

 

0.12

 

Training of Instructors: Professional development for the System 44 teachers included a 1-day training in August 2010, 3 coaching visits from System 44 professional learning consultants during the 2010–2011 school year, and periodic district meetings that served as a System 44 professional learning community for participating teachers. The intervention teachers in the study had the following background: 92% female, with a mean of 12 years teachers experience; 33% had a Bachelor's degree, 42% had a Master's degree, and 25% had a specialist or Doctoral degree; in addition 8% had a reading specialist degree. Of the intervention teachers, 50% had previously taught System 44 and 83% had previously taught READ 180.

Design: Partially Convincing Evidence

Did the study use random assignment? Yes

If not, was it a tenable quasi-experiment? NA

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes? Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? NA

Were the program and control groups demographically comparable at pretreatment? Yes

Was there attrition bias1No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)? Yes

 

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Fidelity of Implementation: Partially Convincing Evidence

Describe when and how fidelity of treatment information was obtained: The evaluation team assessed the fidelity of System 44 implementation in 2 ways: classroom observations (conducted in the fall, winter, and spring, classroom observations evaluated classroom setup, minutes of instruction, inclusion of program components, instructional management, and instructional delivery. Evaluation of the fidelity of implementation was also assessed based on teacher self-report in surveys and interviews.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: The evaluation team assessed fidelity of System 44 implementation through classroom observations and teacher self-report (surveys and interviews). Based on the classroom observation "classroom setup" was rated on 3 criteria: computers for at least one third of the class are accessible and functioning, auxiliary equipment (headsets, microphones, CD players) are accessible and functioning, and System 44 materials are easily accessible to students and teachers. The mean rating across these items was converted to a 4-point scale. In fall 2010, 58% of the 12 classrooms observed received the highest rating; this percentage increased to 75% in the winter and 100% of the classrooms in spring 2011. "Minutes of System 44 instruction" were consistent across all 3 observations. Although 9 of the 12 classrooms provided at least 55 minutes of instruction daily, 3 middle school classrooms provided between 45 and 49 minutes of instruction daily because the class period length did not allow for the allocation of 60 minutes for System 44 instruction. In regard to use of program components, the mean number of students using each type of material declined over time due to decreasing class size as students completed the program. Level of student engagement appeared to peak at the winter observation point. [See submitted report for additional detail] With regard to Fidelity of Implementation based on teacher self-report, the spring 2011 teacher survey respondents reported that students used the System 44 software between 20 and 30 minutes daily (average across classrooms: 25.3 minutes). The number of computers available in each classroom ranged from 5 to 20 (average 8.8). At the conclusion of the System 44 training in August 2010 and at the end of the 2010–2011 school year, the participating teachers were asked to respond to 10 questions designed to test their understanding of the System 44 program components. On average, the teachers answered 80% of the items correctly in fall 2010 and 81% of the items correctly in spring 2011. The teachers also rated their understanding of 5 key System 44 program components in the fall and spring (see Exhibit 9). Overall, the teachers reported a greater understanding of how to implement the program in spring 2011 than they had in fall 2010, but the differences were not statistically significant. In addition, the teachers’ responses varied less in the spring than in the fall, a finding that suggests most teachers believed they had a high level of understanding of how to use the System 44 program components.

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Elision, Comprehensive Test of Phonological Processing (CTOPP)

Internal consistency .89 (across all ages); test-retest reliability .82; interrater reliability .96–.99

Phonological awareness (student is instructed to say what is left of the word after being instructed to drop out designated sounds); assesses student’s ability to hear the distinct sounds that make up a word.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Letter-Word Identification, Woodcock-Johnson III

Subtest median test reliability statistic .94. WJ III 1-year test-retest study reliability .80-.90. WJ III reliability across forms .80-.96

Assesses a student’s ability to accurately read aloud printed words.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Overall Accuracy, Phonics Inventory (PI)

Internal consistency .71–.77; immediate test-retest reliability with alternate forms .70–1.00

Phonics accuracy; phonological decoding of nonsense words and sight word reading

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Overall Fluency, Phonics Inventory (PI)

Internal consistency .85–.91; immediate test-retest reliability with alternate forms .78–1.00.

Phonics speed and accuracy; phonological decoding of nonsense words and sight word reading

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Phonetic Decoding Efficiency, Test of Word Reading Efficiency (TOWRE)

Test-retest reliability .90; interrater reliability .99

Phonics; assesses student’s ability to apply grapho-phonemic knowledge by reading pronounceable non-words.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Sight Word Efficiency, Test of Word Reading Efficiency (TOWRE)

Test-retest reliability .91; interrater reliability .99

Reading fluency; measures the number of printed words a student can accurately read aloud in 45 seconds

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Word Attack, Woodcock-Johnson III

Subtest median test reliability statistic .87. WJ III 1 year test-retest study reliability .80-.90. WJ III reliability across forms .80-.96

Assesses a student’s ability to apply phonic and structural analysis skills by measuring the number of nonwords that a student can accurately pronounce aloud.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

 

Broader  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

California Standards Test, English Language Arts

 

Standardized state test measuring students reading ability.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Reading Inventory (RI)

Internal consistency (standard error of measurement matching text to Lexile) .92; test-retest reliability .89

Reading comprehension

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

Test of Silent Reading Efficiency and Comprehension (TOSREC), Forms A & C

Test-retest reliability .79 across ages, alternate form delayed administration; interrater reliability .99

Reading comprehension; assesses silent reading speed, accuracy, and comprehension.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention. The elementary school reading interventions included Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta Stone, Explode the Code, and various worksheets.

 

Number of Outcome Measures: 4 Prereading, 6 Reading

Mean ES - Targeted: 0.14*

Mean ES - Broader: 0.02

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Prereading

Elision, Comprehensive Test of Phonological Processing (CTOPP)

0.10

Prereading

Overall Accuracy, Phonics Inventory (PI)

0.30*

Prereading

Overall Fluency, Phonics Inventory (PI)

0.54***

Prereading

Phonetic Decoding Efficiency, Test of Word Reading Efficiency (TOWRE)

-0.02

Reading

Letter-Word Identification, Woodcock-Johnson III

0.02

Reading

Sight Word Efficiency, Test of Word Reading Efficiency (TOWRE)

0.01

Reading

Word Attack, Woodcock-Johnson III

0.06

 

Broader Measures

Construct

Measure

Effect Size

Reading

California Standards Test, English Language Arts

0.03

Reading

Reading Inventory (RI)

0.19

Reading

Test of Silent Reading Efficiency and Comprehension (TOSREC), Forms A & C

-0.16

 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

u      Effect size is based on unadjusted means

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small Groups, (n=5-7)

Duration of Intervention: 60 minutes, 5 times a week

Minimum Interventionist Requirements: Paraprofessional, 4-8 hours of training

Reviewed by WWC or E-ESSA: E-ESSA

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

Qualifying studies found no significant positive outcomes.

Other Research: Potentially Eligible for NCII Review: 2 studies

Scholastic Research (2012). Murrieta Valley Unified School District Research Update. New York, NY: Scholastic Inc.

Scholastic Research (2013). Saginaw Public Schools Research Update. New York, NY: Scholastic Inc.