Reading for All Learners
Study: Callow-Heusser & Sanborn (2016)

Describe how students were selected to participate in the study:

All children enrolled in 1st grade who were enrolled in school during beginning of year DIBELS benchmark testing and who scored below grade level (strategic or intensive needs in reading based on the DIBELS beginning of year benchmark assessment) were included in the study, per the school’s agreement with families about conducting research to best meet students’ needs. Students who enrolled after beginning of year assessments were assigned to appropriate small groups per placement testing results, with new students alternately assigned to treatment or comparison groups to ensure consistent sizing of groups and to minimize disruption from new students. However, students who enrolled after random assignment to groups were not included in the analysis reported here. Students with disabilities were also included in the study, with these students randomly assigned to treatment or comparison groups separately from students not identified with disabilities. Students with disabilities who were best taught in resource room settings were also randomly assigned to treatment or comparison groups, with those students taught either one-on-one or in small groups of two students.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students who were assigned to treatment or comparison groups based on DIBELS beginning of year benchmark composite scores included 30% Hispanic and 8% refugee or immigrant students from other countries. More than 25% spoke a home language other than English. According to the NCES, 99% of students qualified for free and reduced lunch in the 2013-2014 school year. Approximately 11% of the students who participated in the study had been identified for special services other than speech language services.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Reading for All Learners (RFAL) authored by Dr. Alan Hofmeister, distributed by Academic Success for All Learners

Specify which condition is the control condition:

Early Reading Intervention (ERI) authored by Dr. Edward J. Kame’enui and Dr. Deborah C. Simmons, distributed by Scott Foresman. ERI had been used as the primary intervention for several years prior to this study; therefore, it served as the “business-as-usual” intervention at the school.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

After beginning of year DIBELS benchmark assessments were completed, researchers met with literacy specialists at the school to randomly assign students to groups. All students who performed below grade level based on the DIBELS composite score were selected. Those students were grouped according to similar patterns of scores on the DIBELS subtests. For example, first grade students needing strategic support based on the composite score were grouped together based on levels of risk on the individual subtests: Phoneme Segmentation and Nonsense Words Fluency. Those who needed strategic support on both subtests were placed in one group, while those who needed intensive support on Nonsense Words Fluency but strategic support on Phoneme Segmentation Fluency were placed in another group. Those with intensive needs on both subtests were placed in yet another group. Students were ordered within groups according to DIBELS composite scores. Then, the first two students from a group were selected. A coin was flipped to determine if the first student was assigned to the treatment group (heads) or the comparison group (tails). The other student was assigned to the other group. Then the next two students were selected and again, the coin was tossed. If there was an odd number of students in the original group, the coin was tossed to determine the assignment of the last unpaired student. In this way, the odds for each student of being assigned to the treatment group was 50%.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Licensed teachers and paraprofessionals who had been teaching reading for a minimum of 3 years conducted the small group instruction. Academic Success for All Learners staff provided a 4-hour RFAL training for all instructors assigned to the program/ treatment groups at the outset of the study. A trainer observed small group instruction and provided feedback on a monthly basis.

Describe when and how fidelity of treatment information was obtained.

Approximately once per month, a RFAL trainer visited the school to answer questions specific to RFAL, conduct classroom observations, and offer suggestions to improve quality of instruction. Most of these suggestions applied equally to both RFAL and ERI reading groups, as they involved pacing, error correction, academic feedback, frequency of group choral versus individual responses, and other effective teaching practices. Additionally, observers conducted classroom observations of all treatment and control groups on a monthly basis utilizing a standardized observation form designed to capture fidelity of implementation data. Results indicated that fidelity of implementation was consistent across groups and programs.

What were the results on the fidelity-of-treatment implementation measure?

Because all instructors taught both treatment and comparison groups, differences due to instructors were minimized. Additionally, observation of instructors under both conditions showed comparable fidelity of implementation of both programs.

Was the fidelity measure also used in control classrooms?

Yes. See above.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

This study included random assignment to treatment and comparison groups of those students who fell below grade level on the beginning of year DIBELS benchmark assessment in one elementary school. As such, the sample size for the number of students participating in the study is adequate to achieve sufficient statistical power when analyzed using student as the unit of analysis. Using cluster sizes of small group or classroom would not be feasible for power requirements with this sample. To help account for teacher effects, teachers and paraprofessionals taught small groups of students using both the treatment (RFAL) and the comparison (ERI) programs, having been trained to use both. Several analyses were conducted. Based on the assumption that random assignment to groups created “equal” groups, analysis of variance statistical tests were conducted for each outcome measure. Additionally, to account for baseline group differences in beginning of year assessments despite random assignment to groups, linear regression on the outcome variable with beginning of year composite scores and group (treatment or comparison) as independent variables was conducted to determine the proportion of variability in the outcome that could be accounted for by beginning of year reading skills and group membership. Sample sizes, means and standard deviations for study data are included. Because groups were equivalent at baseline (after random assignment to groups), no statistical adjustments were made. Additionally, no adjustments were calculated to account for clustering. A Bonferroni correction could be used to account for multiple comparisons within the same domain. A Bonferroni correction adjusts the p-value (or confidence interval) by dividing by the number of comparisons. However, actual p-values are reported below, and because all are larger than .05 and, hence, not statistically significant when uncorrected, a Bonferroni correction was not necessary.