i-Ready Personalized Instruction for Mathematics
Study: Randal et al. (2020)

Describe how students were selected to participate in the study:

Students were eligible or inclusion in this study if they attended a public school using i-Ready Personalized Instruction for mathematics and/or i-Ready Assessment during the 2018-19 school year and had both fall and spring scores from the i-Ready Diagnostic. For a student within a treatment school to be eligible for inclusion, they must have met the minimum thresholds for appropriate use set by Curriculum Associates, meaning the students used i-Ready Personalized Instruction for mathematics for a minimum of 18 distinct weeks between the fall and spring assessment period AND for an average of at least 30 minutes per week of instruction. Treatment schools were included only if they began using i-Ready Personalized Instruction and i-Ready Assessment to some extent prior to the 2018–19 school year and over 90 percent of the students in a school had used i-Ready Personalized Instruction and i-Ready Assessment. This requirement is based on the understanding that i-Ready implementation, like the implementation of most new programs, requires start-up time to learn the technology and adjust to the schedule before i-Ready is fully implemented. Comparison schools were included only if 90 percent of the students in that school used i-Ready Assessment but not i-Ready Personalized Instruction. To be eligible for inclusion as a student in a comparison school, students must not have used any i-Ready Personalized Instruction lessons for mathematics during the 2018–19 school year.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The study looked at students who were identified as being high risk for academic failure: students in the lowest 20th percentile of mathematics achievement at baseline. Note that the published study also included students who were classified as being two or more grades below their chronological grades, all of whom meet the criteria for being “at-risk for academic failure” and are below the 35th percentile of mathematics achievement at baseline.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

100.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

In this study, schools with students who used i-Ready Personalized Instruction for a minimum of 18 distinct weeks between the fall and spring assessment for an average of at least 30 minutes per week are considered the intervention. Students had to have valid, non-rushed scores on the i-Ready Assessment in both the fall and spring assessment windows.

Specify which condition is the control condition:

In this study, schools with students who did not have any exposure to i-Ready Personalized Instruction were the control group. Students had to have valid, non-rushed scores on the i-Ready Assessment in both the fall and spring assessment windows.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

N/A

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

This study used a quasi-experimental design (QED) to evaluate the impact of i-Ready mathematics instruction on student mathematics achievement for striving learners in grades 2–5 on a nationally normed cognitive assessment, the i-Ready Diagnostic. The researchers examined students who fell at the bottom 20th percentile of mathematics achievement. The percentiles were based on mathematics achievement measured by the i-Ready Diagnostic at baseline. For a student within a treatment school to be eligible for inclusion, they must have met the minimum thresholds for appropriate use set by Curriculum Associates, meaning the students used i-Ready Personalized Instruction for mathematics for a minimum of 18 distinct weeks between the fall and spring assessment period AND for an average of at least 30 minutes per week of instruction. Treatment schools were included only if they began using i-Ready Personalized Instruction and i-Ready Assessment to some extent prior to the 2018–19 school year and over 90 percent of the students in a school had used i-Ready Personalized Instruction and i-Ready Assessment. This requirement is based on the understanding that i-Ready implementation, like the implementation of most new programs, requires start-up time to learn the technology and adjust to the schedule before i-Ready is fully implemented. Comparison schools were included only if 90 percent of the students in that school used i-Ready Assessment but not i-Ready Personalized Instruction. To be eligible for inclusion as a student in a comparison school, students must not have used any i-Ready Personalized Instruction lessons for mathematics during the 2018–19 school year. Students in both groups had to have valid, non-rushed scores on the i-Ready Assessment for both the fall and spring testing windows. Matching: The researchers used a multi-step process to identify analytic samples separately at each grade to address the confirmatory research question. First, school-level matching was conducted to identify a set of i-Ready (treatment) and a set of comparison schools from which to match students. Next, student-level matching was conducted with students within the selected matched schools to identify a set of i-Ready students and comparison students equivalent on mathematics achievement, as measured by the Diagnostic. Effect sizes were computed for all school- and student-level matching variables following matching and found baseline equivalence was achieved according to What Works Clearinghouse standards.

What was the unit of assignment?

Schools

If other, please specify:

Please describe the unit of assignment:

Treatment schools were included only if they began using i-Ready Personalized Instruction and i-Ready Assessment to some extent prior to the 2018–19 school year and over 90 percent of the students in a school had used i-Ready Personalized Instruction and i-Ready Assessment. Comparison schools were included only if 90 percent of the students in that school used i-Ready Assessment but not i-Ready Personalized Instruction. Across grades, 94.5 percent of students attended schools with students classified as only treatment or comparison. Separately by grade, the researchers excluded the small percentage of schools with some students classified as treatment and other students classified as comparison from our school-level assignment study.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

The outcome variable is the adjusted mean difference in the outcome between school study groups.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Educators who use i-Ready with their students tend to be K-8 classroom teachers, intervention specialists, and other types of classroom professionals. Schools that use i-Ready Personalized Instruction have teachers and administrators attend a standard set of professional development courses that include sessions on Data-Driven Leadership, Getting Good Data with i-Ready, Using Data to Plan Instruction, and Delivering Differentiated Instruction, as well as the opportunity for tailored support. The professional development series is structured to meet the needs of those new to i-Ready and those who are familiar with it, to provide a more targeted professional development approach.

Describe when and how fidelity of treatment information was obtained.

Treatment students had to have valid, non-rushed scores on the i-Ready Assessment in both the fall and spring testing windows. Study participants were included in this study only if they met certain fidelity of treatment characteristics related to their i-Ready Personalized Instruction usage. The i-Ready Personalized Instructional usage information was collected by the i-Ready system, which tracks i-Ready usage on a variety of data points, including time on task in minutes per week. The time on task data are collected as students interact with the content. Students who log in but do not make progress during a session are flagged. In addition, the use of distinct weeks of instruction is not simply calendar weeks, but time when a student is in school and progress is being made. Learning does not happen at the same rate each week, and using the distinct weeks of instruction allows for more precise understanding of when learning is happening. Teachers and administrators have access to these same data through the i-Ready user-interface, and students who are not meeting targets in relation to these metrics are flagged as well. Quarterly reviews of fidelity are conducted and shared in every school; these serve as a check on the accuracy of data being collected.

What were the results on the fidelity-of-treatment implementation measure?

The researchers specified that only those students who met certain fidelity-of-treatment characteristics, namely using i-Ready instruction for a minimum of 18 distinct weeks of instruction and for an average of at least 30 minutes per distinct week of instruction, were included in the impact analysis.

Was the fidelity measure also used in control classrooms?

Students were selected for the control group only if they had no i-Ready instruction usage recorded in the i-Ready system and had valid, non-rushed i-Ready Assessment scores in both the fall and spring testing windows. The control condition is interpreted as “business as usual” as students are getting some core and supplemental materials that are not i-Ready Personalized Instruction.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

No substantial or statistically significant differences in baseline equivalence in school level covariates or pretest scores were found.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

Students had both valid, non-rushed fall and spring assessment scores and schools had data related to each included covariate.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

N/A

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Hierarchical linear modeling (HLM) was used to estimate impacts of i-Ready on student achievement for the samples of striving learners both the confirmatory and exploratory research questions. For each analysis, a two-level model was used with level 1 as the student and level 2 as the school. For level 1 of the model, baseline i-Ready Diagnostic performance was included. For level 2 of the model, an indicator variable of group membership and school-level variables that were publicly available and known to be related to achievement were included (i.e., group membership (0 = comparison; 1 = i-Ready), urbanicity, percent of students eligible for free and reduced-price lunch, percent of students of historically marginalized races, grade-level enrollment). Level 1 of the benchmark model was specified as: Yij = β0j + β1j(BASELINEij) + eij where Yij is the spring Diagnostic score for student i in school j. β0j is the adjusted mean outcome for students in school j. β1j is the regression slope of the student’s baseline (fall) Diagnostic score for school j. eij is the random error in the outcome associated with student i in school j not accounted for in the model. Level 2 of the model was specified as: β0j = γ00 + γ01(TREATj) + Σγq(URBANICITYj) + γ02(%FRL) + γ03(%HMR) + γ05(ENROLL) + u0j β1j = γ10 where γ00 is the adjusted comparison group grand mean of the outcome, γ01 is the adjusted mean difference in the outcome between school study groups, and TREAT is an indicator variable coded as 1 for schools in the i-Ready treatment group and 0 for schools in the i-Ready comparison group. Σγq is a vector of indicator variables for school urbanicity (city, suburb, town, rural). γ02 - γ04 are regression slopes of the school-level covariates. ENROLL is number of students enrolled at the grade level in the analysis. u0j is the random error in the achievement outcome associated with school j. We conducted two sensitivity analyses for each grade to examine the robustness of the findings of the benchmark impact model for the samples of striving learners. The first sensitivity analyses included a school-level grand mean centered baseline covariate. The second included student-level Diagnostic domain level scores to account for the stratification and matching of students within fall Diagnostic placement profiles. Both analyses yielded results consistent with our benchmark model. We conducted two sensitivity analyses for each grade to examine the robustness of the findings of the benchmark impact model for the samples of striving learners. The first sensitivity analyses included a school-level grand mean centered baseline covariate. The second included student-level Diagnostic domain level scores to account for the stratification and matching of students within fall Diagnostic placement profiles. Both analyses yielded results consistent with our benchmark model. Sensitivity Analysis 1. The first sensitivity analysis examined the robustness of the findings to including a school-level grand mean centered baseline covariate. Level 1 of the model had the same specification as the benchmark model. Level 2 of the models was specified as follows: β0j = γ00 + γ01(TREATj) + γ02(BASELINE.j – BASELINE..)j + Σγq (URBANCITYj) + γ03(%FRL) + γ04(%HMR) + γ05(ENROLL) + u0j where γ00 is the adjusted comparison group grand mean of the outcome, γ01 is the adjusted mean difference in the outcome between school study groups, and TREAT is an indicator variable coded as 1 for schools in the i-Ready treatment group and 0 for schools in the comparison group. γ02 is the regression slope of the school-level baseline Diagnostic score (grand mean centered). Σγq and γ03 - γ05 are regression slopes of the school-level covariates specified as described in the benchmark model. ENROLL is number of students enrolled at the grade level in the analysis. u0j is the random error in the achievement outcome associated with school j. Results of this model were consistent with the benchmark model findings for all grades: i-Ready had statistically significant impacts on student Mathematics achievement, and impact estimates were similar to those from the benchmark model. Sensitivity Analysis 2. The second sensitivity analysis we conducted examined the robustness of the findings to including student level Diagnostic domain level scores to account for the stratification and matching of students within fall i-Ready domain placement profiles. Level 2 of this model as the same specification as Level 1 of the benchmark model. Level 1 of this sensitivity analysis model is specified as follows: Yij = β0j + β1j(BASELINEij) + β2j(DOMAIN1ij) + β3j(DOMAIN2ij) + β4j(DOMAIN3ij) + eij where Yij is the spring Diagnostic score for student i in school j. β0j is the adjusted mean outcome for students in school j. β1j is the regression slope of the student’s baseline (fall) Diagnostic score for school j. Β2j – β4j are regression slopes of the baseline (fall) Diagnostic domain scores for student i in school j. The mathematics domain scores at grades 2–5 include Algebra and Algebraic Thinking, Number and Operations, Geometry, and Measurement and Data. eij is the random error in the outcome associated with student i in school j not accounted for in the model. Results of this model were consistent with the benchmark model findings for all grades. i-Ready had statistically significant impacts on student Mathematics achievement, and impact estimates were similar to those from the benchmark model.