System 44 Next Generation

Study: Beam, Faddis, & Hahn (2012)

Beam, M. Faddis, B. & Hahn, K. (2012). Evaluation of System 44. Grantee: Saginaw Public Schools in Saginaw, MI. Portland, OR: RMC Research Corporation.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

System 44 Next Generation is the proven foundational reading program designed for the most challenged readers in Grades 3–12+. System 44 Next Generation is proven to help students master the foundational reading skills required for success with the Common Core through explicit instruction in phonics, comprehension, and writing. System 44 integrates principles of cognition and learning with practices for instructional effectiveness for older struggling readers. System 44 uses state-of-the art adaptive software to deliver a personalized learning progression with explicit, research-based phonics instruction. Explicit, teacher-led instruction in close reading, comprehension, academic vocabulary, and writing provides students with the skills needed to succeed with the Common Core, college, and career.

System 44 Next Generation is intended for use in grades 3-12. The program is designed for use with students with learning disabilities, behavioral disabilities, English language learners, and any student at risk of academic failure. The academic area of focus is reading (including phonological awareness, phonics/word study, comprehension, fluency, vocabulary, spelling, oral language development, and word analysis) and writing (including spelling, sentence construction, planning and revising, summary, informative and argument writing).

System 44 is currently used on all 50 states, and in over 10,000 classrooms. There are over one hundred thousand active student licenses currently being used every day. Scholastic is able to track this data through their sales systems.

Where to obtain: 
Scholastic Inc.
557 Broadway New York, NY 10012
Phone: 1-877-234-READ
Website: scholastic.com/system44

Cost: Initial cost per student for implementing program: $1,100

Replacement cost per student for subsequent use: $24.95

System 44 pricing is determined by the number of students being served and number of classrooms set up. At the basic level, the cost is $22,000 for a Stage of System 44 to service 20 student licenses (This is where the $1,100 per student cost is derived). Please note: Licenses are sold on a perpetual basis. Volume discounts available.

System 44 includes per stage –

Teacher Materials: Teacher’s Edition, Library Teaching Resources, Resources for Differentiated Instruction, a Screening, Assessment, and Placement Guide, Anchor Media, Online access to all teacher materials, included a Dashboard with point of use access to data and instruction.

Student Materials – 20 student editions, 20 Decodable Digest, 280 paperbacks (56 titles, 5 copies each), 20 student licenses to the instructional software, and 20 licenses to the Scholastic Phonics Inventory, and the Scholastic Reading Counts quizzes.

Leadership materials – Access to a robust dashboard for data and implementation metrics. Also included in the price is professional development. Scholastic recommends ongoing professional development at an additional cost.

System 44 Next Generation is designed for use with small groups of five to seven students.

The program takes 50-60 minutes per session with a recommended five sessions per week.

The program includes highly specified teacher manuals.

System 44 is a blended instructional model, where a half of the class is on the computer during the small-group rotations. Therefore, the number of computers depends on the number of students in the System 44 classroom.

Training is required.

Scholastic offers comprehensive training and support for participating System 44 teachers and leaders. Scholastic provides a day of upfront teacher training with the purchase of a complete stage of System 44. Scholastic coaches also provide additional training, as well as in person coaching. System 44 also includes embedded professional development resources—a comprehensive suite of professional development resources and tools for assessing students and differentiated instruction with both print and online materials.

Instructors must be professionals and it is not assumed that the instructor has expertise in a given area.

System 44 training materials and teacher implementation guides are reviewed by expert Scholastic consultants and field-tested in dozens of pilot sites to ensure that the content and strategies are practical and best support teachers and students with successful implementation. The consultants who review and field test the materials bring knowledge and expertise from working side-by-side with System 44 teachers and leaders in classrooms and schools across the country.

Each System 44 teacher receives training materials and teacher implementation guides reviewed by Scholastic consultants and field tested by consultants who work with teachers using System 44 across the country. Practitioners may obtain ongoing professional/-technical support. Additional training available includes coordinator training, in classroom support, Scholastic Training Zone, and Scholastic U.

 

Participants: Convincing Evidence

Sample size: 317 students (155 program, 162 control)

Risk Status: The target sample was selected based on performance on the fall 2011 Michigan Educational Assessment Program (MEAP) or spring 2011 AIMSweb assessment. The Scholastic Reading Inventory (SRI) was used to screen students who performed below the 25th percentile on either the MEAP or AIMSweb assessments. Students who scored below 600 Lexiles on the SRI were subsequently administered the Scholastic Phonics Inventory (SPI) to identify students in need of additional phonics instruction. Students who scored in the Beginning or Developing reader categories on the SPI were randomly assigned to the System 44 treatment group or control group.

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

 31

20%

28

17%

 

  Grade 5

 54

35%

59

36%

 

  Grade 6

 33

31%

35

22%

 

  Grade 7

 19

12%

17

10%

 

  Grade 8

 18

12%

23

14%

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

124

80%

124

77%

 

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

 

 

 

 

 

  Hispanic

15

10%

21

13%

 

  White

 

 

 

 

 

  Other

 

 

 

 

 

  Minority

 139

 90%

 145

 87%

 

  Not minority

 16

 10%

 17

 13%

 

Socioeconomic status

  Subsidized lunch

148

96%

156

96%

 

  No subsidized lunch

7

4%

6

4%

 

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

84

54%

81

50%

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

71

46%

81

50%

 

ELL status

  English language learner

6

4%

9

6%

 

  Not English language learner

149

96%

153

94%

 

Gender

Female

61

39%

65

40%

 

Male

94

61%

97

60%

 

Training of Instructors: Professional development for the System 44 teachers included a 1-day training in September, 2011, follow-up training in November, 2011, and monthly coaching visits from Scholastic staff from December 2011 through May 2012.

Design: Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: N/A

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: N/A

Were the program and control groups demographically comparable at pretreatment?: Yes

Was there attrition bias1?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Unconvincing Evidence

Describe when and how fidelity of treatment information was obtained: Classroom observations were conducted by evaluators 3 times during the school year—in fall 2011, winter 2012, and spring 2012—in order to collect data on classroom‑level implementation. Implementation fidelity components included (a) classroom setup, (b) minutes of instruction on software and in small groups, (c) inclusion of all program components (e.g., the 44Book, System 44 paperback books, and the Decodable Digest), and (d) instructional management and delivery.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: Classroom implementation fidelity included 4 components: classroom setup, minutes of instruction, inclusion of program components, and instructional management and delivery. For (a) classroom setup, in fall 2011, 36% of the 20 classrooms observed received the highest rating; this percentage increased to 67% in the winter and 78% of classrooms in spring 2012. For (b) minutes of instruction, the Scholastic staff intended teachers to use the System 44 model in a 60‑minute class period. Across all 3 observation time points, 72% to 82% of the 20 classrooms provided at least 55 minutes of instruction daily. For (c) inclusion of program components, observations of instructional materials used during teacher‑led small group activities indicated that in the fall, average level of use was 1.8, in the winter average use was 1.7, and in the spring average use was also 1.7.[1] For (d) instructional management and instructional delivery, 6 aspects of instructional management were rated, such as teacher’s ability to maintain a positive learning environment, monitor students, keep students on task, provide smooth transitions between rotations, use flexible student groups, and differentiate support based on student needs. Five aspects of instructional delivery were rated, such as pacing, appropriateness of the delivery for student skill level, teacher preparation, active engagement of students, and overall lesson execution. For instructional management, average score in the fall was 3.1, winter was 2.8, and spring was 2.8.[2] For instructional delivery, average score in the fall was 3.0, winter was 3.4, and spring was 2.7.



[1]On a 3-point scale: 0 = none, 1 = partially, 2 = fully.

[2]On a 4-point scale: 0 = not at all, 1 = to a small extent, 2 = to a moderate extent, 3 = definitely. 

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure Reliability Statistics Relevance to Program Instructional Content Exposure to Related Content Among Control Group

Elision, Comprehensive Test Of Phonological Processing (CTOPP)

Internal consistency 0.89 (across all ages); test‑retest reliability 0.82; interrater reliability 0.96–0.99

Phonological awareness (student is instructed to say what is left of the word after being instructed to drop out designated sounds); assesses student’s ability to hear the distinct sounds that make up a word.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Phonemic Decoding Efficiency, Test of Word Reading Efficiency (TOWRE)

Test‑retest reliability 0.90; interrater reliability 0.99

Phonics; assesses student’s ability to apply grapho‑phonemic knowledge by reading pronounceable non‑words.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Overall Accuracy, Scholastic Phonics Inventory (SPI)

Internal consistency 0.71–0.77; immediate test‑retest reliability with alternate forms 0.70–1.00

Phonics accuracy; phonological decoding of nonsense words and sight word reading

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Overall Fluency, Scholastic Phonics Inventory (SPI)

Internal consistency 0.85–0.91; immediate test‑retest reliability with alternate forms 0.78–1.00 

Phonics speed and accuracy; phonological decoding of nonsense words and sight word reading

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Sight Word Efficiency, Test of Word Reading Efficiency (TOWRE)

Test‑retest reliability 0.91; interrater reliability 0.99 Reading fluency; measures the number of printed words a student can accurately read aloud in 45 seconds The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Word Identification, Woodcock‑Johnson III

Internal consistency 0.91; test‑retest reliability 0.95 Reading fluency; assesses a student’s ability to accurately read aloud printed words The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

 

Broader Measure Reliability Statistics Relevance to Program Instructional Content Exposure to Related Content Among Control Group

Form A and C, Test of Silent Reading Efficiency and Comprehension (TOSREC)

Form A and C, Test of Silent Reading Efficiency and Comprehension (TOSREC)

Reading comprehension; assesses silent reading speed, accuracy, and comprehension.

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

Scholastic Reading Inventory (SRI)

Internal consistency (standard error of measurement matching text to Lexile) 0.92; test‑retest reliability 0.89 

Reading comprehension

The counterfactual varied across schools because students in the control group did not receive a uniform alternative to the System 44 intervention.  

 

Number of Outcome Measures: 4 Prereading, 4 Reading

Mean ES - Targeted: Data Unavailableu

Mean ES - Broader: 0.06u

Effect Size:

Targeted Measures

Construct Measure Effect Size
Prereading CTOPP Elision 0.15u
Prereading TOWRE Phonetic Decoding Efficiency -0.02 u
Prereading SPI Overall Accuracy 0.03 u
Prereading SPI Overall Fluency 0.04 u
Reading TOWRE Sight Word Efficiency -0.02 u
Reading TOWRE Total

 Broader Measures

Construct Measure Effect Size
Reading TOSREC -0.06 u
Reading SRI 0.18 u

 

Key
*       p ≤ 0.05
**     p ≤ 0.01
***   p ≤ 0.001
–      Developer was unable to provide necessary data for NCRTI to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small groups, (n=5-7)

Duration of Intervention: 50-60 minutes, 5 times a week

Minimum Interventionist Requirements: Paraprofessional, 4-8 hours of training

Reviewed by WWC or E-ESSA: E-ESSA

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

Qualifying studies found no significant positive outcomes.

Other Research: Potentially Eligible for NCII Review: 2 studies

Scholastic Research (2012). Murrieta Valley Unified School District Research Update. New York, NY: Scholastic Inc.

Scholastic Research (2013). Saginaw Public Schools Research Update. New York, NY: Scholastic Inc.