iRead
Study: Hamilton et al. (2016)

Describe how students were selected to participate in the study:

Two school districts volunteered to participate in the iRead Early Outcomes QED study. School leaders, recruited the elementary schools, and the independent evaluator (Westat) matched the schools. School principals at the participating treatment and control schools selected one classroom per grade level to participate as data collection classrooms. All students within the selected subsample classrooms were assessed.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Grade 1 and Grade 2 students performing below the 30th percentile on the Group Reading Assessment and Diagnostic Evaluation (GRADE) pretest were identified as being at risk for academic failure.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

iRead

Specify which condition is the control condition:

Business-as-usual ELA instruction

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

This evaluation utilized a quasi-experimental design to estimate the impact of the iRead supplemental curriculum on reading achievement in grades K-2. The iRead intervention was implemented in school year 2013-2014 in District 1 and from 2014-2015 in District 2. Our combined study sample included a total of 804 students from 48 classrooms in 16 schools. During the one year program implementation, students in both the iRead and comparison classrooms were independently assessed at pre and post with five subtests from the Group Reading Assessment and Diagnostic Evaluation (GRADE) assessment. This evaluation is designed to answer the following confirmatory research question: To what extent is growth in literacy skills achieved by iRead students different from students in a matched sample who do not use iRead? Matching: Schools were matched within district on the previous year’s 3rd grade state standardized reading assessment scores. To the extent possible, we also matched on the percentage of English Language Learners (ELL), students with disabilities and students eligible for Free and Recued Priced Lunch (FRPL) to ensure that groups were comparable on these demographic variables. Baseline Equivalence: Consistent with the What Works Clearinghouse (2014) standards, we conducted baseline equivalence testing on the analytic sample (post attrition) at both the school and student levels. At the school level, 3rd grade state reading scores were used to determine the extent to which schools were equivalent. The baseline difference between the groups was not statistically significant (p = 0.55; effect size = 0.14), demonstrating that baseline equivalence was met. Nevertheless, the school-level 3rd grade reading scores were included in the regression analysis model at the school level. Student GRADE pre-test data were used to determine student level baseline equivalence. We conducted separate analyses for each of the five GRADE subtests to ensure that students in the iRead and comparison groups were equivalent. Table A1 shows that the pre-test differences between the students in the two conditions were not statistically significant, demonstrating that baseline equivalence was also met at the student level The effect sizes for the group differences were all less than 0.25 with p values greater than 0.05. Student-level GRADE pre-test scores were included in the regression analysis model at the student level.

What was the unit of assignment?

Schools

If other, please specify:

Please describe the unit of assignment:

No random assignment - schools were matched in a QED

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

A multilevel HLM analysis was conducted using student and school data simultaneously

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Sample Size: iRead=75; Comparison=63 Gender (male): 6%; 6% Age <45: 60%; 79% White: 86%; 82% Black: 4%; 5% Asian: 8%; 8% Hispanic: 6%; 3% Other Race: 1%; 6% Teaching experience <3 years: 11%; 30% Master's Degree: 57%; 70%

Describe when and how fidelity of treatment information was obtained.

The research firm created a custom fidelity tool for the study, which was comprised of five key components, each measured by a series of indicators. These components are: equipment, classroom structure and environment, software usage, instruction, and teacher training. Classroom fidelity was measured at the individual classroom level and aggregated to the district level, and combined sample level. Components of fidelity were gathered across the school year, with the exception of software usage, which was collected at the end of the school year.

What were the results on the fidelity-of-treatment implementation measure?

Both districts achieved the minimum recommended level of fidelity, for four of the five components: including fidelity for equipment, classroom structure and environment, software usage, and instruction. Only one of the two districts achieved the minimum fidelity (set for this study) for teacher training. For additional detail please see iRead QED Final Report August 2016 NCII, Figures B1- B5, pages B-1 through B-3.

Was the fidelity measure also used in control classrooms?

No. The fidelity tool was only used in treatment classrooms, as it measures fidelity to the implementation of iRead. The Classroom Assessment Scoring System (CLASS, K-3) was used to measure the classroom instructional climate in both the treatment and comparison classrooms.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

The CLASS data was used descriptively, and as such is not included in this submission.

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

District data were examined both separately (preliminary/exploratory) and then combined for the confirmatory impact analyses. Analyses consisted of a three-level multiple regression model with students nested within classrooms that were nested within schools. At the student level, we included the GRADE pre-test scores, gender, ELL and special education status and student ethnicity (student level Free and Reduced-Priced Lunch data were not available from both districts). At the classroom level, years of teaching experience, teachers’ educational background, and teacher instructional practice (CLASS scores) at baseline were included. At the school level, in addition to the treatment indicator, the 3rd grade state reading test scores were included in the analysis model. The three GRADE subtests were categorized into three early reading domains: early reading, comprehension and vocabulary, as shown in Table A2 of the attached report (iRead QED Final Report August 2016 NCII).