Reading Plus
Study: Reutzel et al. (2012)

Describe how students were selected to participate in the study:

Students who met the following criteria were selected for the control and treatment group samples: 1) students were age 9 or older; 2) students who were identified as being at risk by scores from their end-of-year high-stakes tests; and 3) students retained to repeat 3rd grade based on FCAT and SAT10 test scores

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students were identified as at risk for academic failure based on scores from 3rd grade end-of-year high-stakes assessments.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Reading Plus® is the submitted program

Specify which condition is the control condition:

The control group used any of the following curriculum-approved interventions in a business as usual condition: Soar to Success, Essential Elements of Reading: Vocabulary, Voyager Passport, and Earobics

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

This study used a matched, quasi-experimental research design. The study’s quasi-experimental control and treatment groups were constructed by the use of a propensity score sampling and matching process.

What was the unit of assignment?

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Teachers had diverse backgrounds and varying levels of teaching experience, but all teachers using the treatment program received an initial hands-on 4-hour professional development course, which covered the pedagogy of the program, and basic functionality of the program and teacher management system. The initial professional development also allowed for teachers and administrators to design implementation and motivation plans specific for their school. A follow up 3-hour professional development course was provided approximately 4 weeks later. This time period allowed the teachers to gain experience and allowed the students to accumulate usage data. The follow up professional development was then able to answer specific teacher usage questions and delve deeper into the progress monitoring and reporting features of the program. All teachers and administrators using the program received on-going support via site visits, email, and phone communications to address any questions or concerns that arose during the implementation period.

Describe when and how fidelity of treatment information was obtained.

Reading Plus is a web-based intervention system delivered via the internet. The imbedded teacher management system provides a scheduling tool which allows the teacher or administrator to set schedule goals for students within the program in terms of number of sessions per week, length of sessions and length of intervention period. Teachers and administrators are given real-time feedback that assists them in creating an acceptable schedule that fall within the parameters given by the publisher. Detailed logs maintained within the system recorded each students’ time on task and performance on the various program components. We compared the students’ actual usage with the schedule set by the teacher and/or administrator and with this information we were able to explicitly assess each students’ level of fidelity

What were the results on the fidelity-of-treatment implementation measure?

100% of the students in the treatment group completed at least the minimum number of lessons suggested by the publisher. The students in the treatment group (n=40), completed an average of 71 lessons with a range of 40 lessons to 119 lessons. All work, including lesson completion dates, lesson format, reading rates, comprehension performance, and time on task are tracked within the Reading Plus management system. The Reading Plus® intervention was scheduled for 20 weeks, 3 sessions per week, and 30 minutes per session. Prior to students beginning the implementation key administrators, lead teachers/reading coaches, and classroom teachers were trained on how to use the Reading Plus® management system. The management system is a key component of a successful intervention as it provides built-in school, class, and student reports that enforce effective schedule guidelines, support monitoring of progress, and help ensure program fidelity. The second step to guarantee program fidelity was to ensure that all 40 students completed the required Reading Placement Appraisal™ (RPA™). This 20-minute placement test assesses independent reading level, rate, comprehension, and vocabulary level and establishes each student’s initial placement level within the program. Teachers physically monitored the administration of this assessment and reviewed the results using the Reading Plus® management system to be certain students had been accurately placed. After the RPA™ was completed Reading Plus® sessions began according to the aforementioned schedule. Each session began with the perceptual accuracy and visual efficiency (PAVE™) warm-up activity. Each student was required to complete the PAVE™ warm-up before the program would allow them to continue to the next instructional component. Teachers received a detailed student summary report via the Reading Plus® management system, which provided monitoring feedback on PAVE™ that included total time spent in the activity, scan rate improvement, and accuracy with flash characters. After completing PAVE™ students continued to the Structured Silent Reading (Guided Reading™ – GR™) lesson – the heart of the intervention. Students build fluency in GR™ by reading narrative, expository, and informational texts that are presented in both a guided silent reading format (a moving window guides students’ eyes across lines of print from left to right) and an independent reading format that dynamically changes the length of text segments and frequency of comprehension questions. Students must be able to read appropriately leveled passages at the current content difficulty levels with research-based grade-appropriate rates and good comprehension before they advance to subsequent levels (Taylor, Frackenpohl, & Pettee, 1960). The Reading Plus® system requires 70% correct responses on comprehension questions for a student to be considered successful. Usual Aural/Oral Range: The Reading Plus® system constantly reviews each student’s progress and evaluates reading, content level, comprehension, and lesson consistency with a complex algorithm of logic that assures students interact with the program as intended and do not simply “click their mouse”. As students engage in Structured Silent Reading lessons the system monitors progress and will notify teachers if students are struggling. In some cases the system may even “suspend” a student until the teacher had a chance to meet with him or her. The system will alert teachers if students are demonstrating excessive reading rates and/or prolonged, inconsistent performance. If students are suspended, the student cannot continue in the program until the teacher “unsuspends” the student. Typically, the teacher would conference with the student to help determine the issue or directly observe the student completing a lesson to gain insight into the student’s struggles. In a previous study, the publisher learned that students who engaged in at least 40 treatment lessons achieved significantly higher gains than students who did not (Rasinski, Samuels, Hiebert, Petscher, & Feller, 2009). Therefore, the number of GR™ lessons was the variable used in this study. However, the staff training, management system reports, and “suspension” logic all provide evidence to confirm that students were doing what Reading Plus® program developers intended during the time they were logged into the system on the computer. Prior to students beginning the study, key administrators, lead teachers/reading coaches, and classroom teachers were trained by Reading Plus® Implementation Specialists. The training focused on effective Reading Plus® implementation, which included a detailed overview of the instructional components students were going to engage in for the duration of the study, scheduling and weekly assignments, as well as an in-depth overview of the teacher management system. The management system plays a key role in a successful Reading Plus® implementation as it provides built-in school, class, and student reports. These reports enforce effective schedule guidelines, support monitoring of progress, and ensure program fidelity by providing weekly benchmarks and milestone-related benchmarks for program use and performance status and improvements (e.g., content level increases, comprehension performance, and comprehension-based reading rate changes). Each teacher also had access to a management “dashboard” for their class that provided a quick overview as to whether their students were achieving weekly assignment and comprehension goals. Implementation Specialists continued to support school staff during the study. After a month of treatment each study site completed a follow-up training that continued to assist teachers in using the reports and progress reviews. For the duration of the entire study period Implementation Specialists were on call to provide on demand off-site and on-site progress monitoring support. So, in conclusion, the system logic, management system reports, notification system, staff training, and implementation team support, all provide evidence to confirm that students were doing what Reading Plus® program developers intended during the time they were logged into the system on their computers.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

A propensity score analysis was used to match the 40 students who did not receive the supplementary silent reading fluency intervention to a group of 40 students who were similar with regard to demographics, prior FCAT achievement, and performance on the Stanford Achievement Test (SAT-10) in this study. At the end of the intervention, the resulting student scores were analyzed using a one-way ANOVA with a linear step-up to control for the false-discovery rate. As the data indicate, the two groups were reasonably matched from the propensity analysis. Because students who participated in the program were from different classes and schools, the analysis was based on available, archival data, the ratio of students to classes was small, precluding a mixed effects modeling of the data to account for clustering at the classroom and school levels.