Lexia® PowerUp Literacy®
Study: Hurwitz (2020)

Describe how students were selected to participate in the study:

We selected as a partner a mid-sized school district enrolling nearly 5,000 students in a Midwestern US suburb. At the middle school level, approximately 70% of the students in the district were unable to read proficiently at the start of the study.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students were identified as at-risk for academic failure based on the fact that they were placed in supplemental literacy classes for students identified as needing reading support.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Treatment classes receiving Lexia PowerUp Literacy served as the intervention condition.

Specify which condition is the control condition:

Control classes receiving business-as-usual instruction served as the control condition.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

What was the unit of assignment?

Classes

If other, please specify:

Please describe the unit of assignment:

Seven supplemental literacy classes were randomly assigned to the PowerUp treatment group, and the remaining three classes were assigned to a business as usual control group that would use their existing supplemental literacy curriculum. Conditions were unbalanced at the request of the district. They historically had difficulty trying to improve students’ reading proficiency and were eager to use PowerUp with as many students as possible. Condition assignment was stratified by school, such that both middle schools contained treatment and control classes.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Eight of the 10 teachers provided personal demographic data for the study. The educators were fairly experienced. All but one (88%) had a Master’s degree, and nearly two-thirds (63%) had been teaching for at least 16 years. To support their implementation of PowerUp, the district participated in an Implementation Success Partnership (ISP), a coaching add-on intended to support successful program use. ISPs include check-ins with school administrators and leaders, and training events for staff that aim to provide a general program overview and deeper dive into interpreting program data and using the program’s offline materials.

Describe when and how fidelity of treatment information was obtained.

At the end of the school year we obtained records of students’ performance in each online strand (Word Study, Grammar, Comprehension), including start levels, weeks of use, and levels completed. Treatment teachers also completed a survey, reporting on how often they logged in to myLexia (PowerUp’s teacher-facing dashboard) using a scale that ranged from “never” to “more than once per day.” They also noted how frequently they administered Lexia Lessons and Skill Builders (PowerUp's offline materials) using a scale that ranged from “never” to “every day or nearly every day.”

What were the results on the fidelity-of-treatment implementation measure?

On average, treatment students used the online program for 16.60 weeks (SD = 3.33). System data reveal that students used the Word Study strand an average of 12.72 weeks (SD = 4.21), the Grammar strand 12.05 weeks (SD = 4.02), and the Comprehension strand 10.95 weeks (SD = 4.26). Although teachers asked students to focus on a different strand each class, PowerUp gives students the choice of which strand to work on. Students could have chosen strands they found easier and could make more progress. Despite the choice, on average students spent time working on all three strands in approximately 75% of their weeks of use. Thus, we can conclude that students received instruction across all three strands. On average students completed 3.02 Word Study levels (SD = 2.12), 1.21 Grammar levels (SD = 1.06), and 1.98 Comprehension levels (SD = 1.75). This amount of progress may seem modest. However, analyses show that during the same time period, PowerUp users nationally completed on average only 1.33 levels in Word Study, 0.45 in Grammar, and 0.98 in Comprehension (K.Vanacore, personal communication, April 13, 2021). PowerUp users in this study more than doubled the amount of progress students showed nationally. All seven treatment teachers indicated that they checked myLexia at least once per week, with four checking daily. Checking myLexia at least once per week corresponds to recommendations specified in the PowerUp Teacher’s Guide. Use of offline materials was somewhat lower than prescribed. Five of the seven teachers reported delivering Lexia Lessons to students identified as struggling in the online program. One teacher did so more than once a week, two did so once a week, and two did so less than once a week. Only two teachers reported using Skill Builders, but these teachers reported doing so weekly.

Was the fidelity measure also used in control classrooms?

No

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Six EL students were removed from the sample because they were receiving more English support than the rest of the sample and disproportionately in the control group: nControl = 4 and nTreatment = 2. Before analyses were conducted, an additional five students were removed from the dataset because their scores at one or both time points were outliers (more than +/- 3.0 standard deviations away from the full-sample mean).

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

To test the hypothesis which predicted that students using PowerUp would show better reading outcomes than control students, we fitted a multi-level model in HLM 8 (Model 1). This allowed us to account for the nested structure of our dataset (i.e., students nested within classes). The mixed model took the following form: 〖Spring〗_ij= γ_00+ γ_01*〖Treatment〗_j+ γ_10 * 〖Lunch Status〗_ij+γ_20 * 〖Special Ed Status〗_ij+〖γ_30 * 〖Fall〗_ij+γ_40 * 〖White〗_ij+γ_50 * 〖Other Race〗_ij+γ_60 * 〖School〗_(ij )+ 〗_ u_0j+〖u_1j * 〖Lunch Status〗_ij +r〗_ij, where 〖Spring〗_ij is the Spring STAR score for student i in class j, γ_oo is the expected Spring STAR score for the average student in a control class, γ_o1 is an estimate of the difference in expected Spring STAR scores between students in treatment and control classes (i.e., the treatment effect), 〖Treatment〗_j is an indicator of being in a treatment class (1 = treatment, 0 = control), ∑_1^6▒γ_X0 are the effects on Spring STAR scores of each of the control variables listed in the following two bullets: 〖Lunch Status〗_ij, 〖Special Ed Status〗_ij, 〖White〗_ij, 〖Other Race〗_ij, and 〖School〗_(ij )are dichotomous indicators of receiving free or reduced-price lunch, receiving special education services, being Anglo-American/White, being of “other” race (i.e., Hispanic-American or multi-ethnic) or attending the smaller of the two schools in this study, respectively. In this model, African-American/Black students served as the comparison group for race-related analyses. 〖Fall〗_ij is each student’s grand-mean centered Fall STAR score, u_0j is a teacher-level residual term, u_1j is a lunch status slope residual term, and r_ij is a residual term for each student within each class.