Classworks Individualized Learning Path Intervention for Language Arts
Study: McCollum (2022)

Describe how students were selected to participate in the study:

Participants in this study were 2,176 students in kindergarten through third grade, gathered from a convenience sample of active districts implementing Classworks during the 2021-2022 academic school year.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All participants were identified as academically-at-risk and in need of an academic intervention in Language Arts based on identification of performing below the 25th percentile at the fall baseline on the Classworks Language Arts Universal Screener.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

100.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Treatment group participants completed both the fall and spring Universal Screeners and participated in Classworks Language Arts Individualized Learning during the 2021-2022 school year. The submitted intervention is participation in Classworks Language Arts Individualized Learning.

Specify which condition is the control condition:

Comparison group participants completed both the fall and spring Universal Screeners and did not participate in Classworks Language Arts Individualized Learning during the 2021-2022 school year.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

Treatment group participants completed both the fall and spring Universal Screeners and participated in Classworks Language Arts Individualized Learning during the 2021-2022 school year. Comparison group participants completed both the fall and spring Universal Screeners and did not participate in Classworks Language Arts Individualized Learning during the 2021-2022 school year.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

Participants in this study were 2,176 students in kindergarten through third grade, gathered from a convenience sample of districts implementing Classworks during the 2021-2022 academic school year. All participants were identified as academically-at-risk and in need of an academic intervention in Language Arts based on identification of performing below the 25th percentile at the fall baseline on the Classworks Language Arts Universal Screener. Assignment of participants to the treatment and comparison groups was at the individual-level of students, on the basis of participation in Classworks Individualized Learning language arts instruction during the 2021-2022 school year. To determine participation in Individualized Learning language arts instruction, students were assigned to either the treatment or comparison group based on whether they had measures indicating a sum IL time on task, IL unit score averages, and a count of IL units completed. Students with these measures indicating ILP usage were assigned to the treatment group, and those who did not have these measures were assigned to the comparison group.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:
Individual student scores on universal screener assessment

Please describe the unit(s) used for primary data analysis:

This study utilized Classworks Universal Screener Assessments to identify participants’ percentile ranking below the 25th percentile as students academically-at-risk and in need of an academic intervention in Language Arts instruction. Additionally, the Classworks Universal Screener Scores were used as an outcome measure to determine the growth in student score performance, from the fall screening window to the spring screening window. The Classworks Universal Screeners have been found to be both psychometrically reliable and valid as instruments to measure grade level readiness, help identify baselines for instruction, identify students who may need additional intervention as part of the RTI/MTSS process, and measure student growth. Classworks Universal Screeners target strands of content vary by grade level and subject. Performance on the Universal Screeners is provided as a vertical score and feedback is also provided at the key strand level. In Language Arts, these strands include Grammar/Usage/Mechanics, Reading Comprehension, Study Skills, Word Analysis, Writing, and the Writing Process. This study compared student growth measured by student score performance from fall to spring on the Classworks Universal Screener Language Arts Assessment between ILP Users and Non-ILP Users. Participants included in the study completed at least both the fall and spring screener during the 2021-2022 school year.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Classworks recommends that all teachers who will use the program receive a one-hour kickstart training to get started and follow up with a one-hour data training within 6-8 weeks. Subsequently, teachers benefit from coaching and free virtual sessions throughout the year to deepen the implementation. Classworks professional learning is delivered via free online, on-demand courses, free live, virtual coaching sessions, and customized live training sessions which can be delivered virtually, in-person or a hybrid of the two for a cost. Instructors/interventionists are typically education professionals such as special education teachers, general education teachers, reading specialists, math specialists, interventionists, or student support services personnel.

Describe when and how fidelity of treatment information was obtained.

What were the results on the fidelity-of-treatment implementation measure?

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Our study calculated effect size of racial sub-demographic groups by grade level at pretreatment with Hedge's G. As shown in our study, all sub-demographic groups by grade level were within .25 MSD, with the exception of the first grade American Indian/Native Alaskan sub-demographic, which had a mean standardized difference beyond .25 SD (.33 SD). Participants in the second grade Hispanic/Latino sub-demographic had a mean standardized difference beyond .25 SD (.83 SD). Participants in the third grade American Indian/ Native Alaskan sub-demographic had a mean standardized difference beyond .25 SD (.42 SD). Further analysis of the sub-demographic groups that were beyond the mean standardized difference of .25 SD was conducted to determine if matching ILP Users (treatment/program group) and Non-ILP Users (comparison/control group) based on grade and racial sub-demographic group would further support these groups as being equivalently appropriate for comparison using Coarsened Exact Matching (CEM). Upon early application of CEM, the initial imbalances between the treatment and comparison groups based on racial sub-demographic groups (L1< 0.001, M< 0.001), grade level (L1< 0.001, M< 0.001) and baseline fall Universal Screener Score at pre-treatment (L1< 0.01, M= 0.63) were found to be balanced.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

All participants completed both the fall and spring universal screeners.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

N/A

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

This study compared student growth measured by student score performance from fall to spring on the Classworks Universal Screener Language Arts Assessment between ILP Users (treatment) and Non-ILP Users (comparison). Participants included in the study completed at least both the fall and spring screener during the 2021-2022 school year. In instances in which participants also participated in the winter screener, this data was also included in analysis. With repeated measures per participant over time, and data collected from participants at a single level (by grade), the best fit model employed in this analysis was a linear mixed-effect model. Models for the impact of ILP usage on grade and racial sub-demographic groups were estimated in R with the R-package lme4 . Each model was conducted on a grade-by-grade subset. For each model, the dependent variable was the universal screener score whereas time (1= fall, 2=winter, 3=spring), treatment (0=comparison, 1=treatment), and racial sub-demographic groups (1=American Indian/Native Alaskan, 2=Other (Asian, Native Hawaiian/Other Pacific Islander, two or more races, White), 3=Black/African American, and 4=Hispanic/Latino) were treated as fixed effects. The interaction between treatment and time was included to determine if there was a difference between the treatment and comparison groups over time.