Read Naturally
Study: Christ & Davie (2009)

Describe how students were selected to participate in the study:

A total of 109 third grade students (60 male, 49 female) from six schools in four Midwestern suburban school districts participated in the study. The schools had not previously used the Read Naturally program. Of the students, 10% were receiving special education services, 23% were English Language Learners (ELL), and 60% received free or reduced lunch. A total of 42% of the students were White, 28% were African American, 23% were Hispanic, 6% were Asian American, and 1% were American Indian. All participants performed at or below the 40th percentile on a measure of oral reading fluency administered in the fall of third grade (DIBELS or AIMSweb), as well as at or below the 40th percentile on a measure of reading comprehension administered at the end of second grade (the Measures of Academic Progress test). Students who qualified for the study, and whose parents gave permission for them to participate, were randomly assigned within their classrooms to the control group (54 students) or the intervention group (55 students). Control group students continued to experience the reading instruction typical of their classroom with no supplemental fluency instruction. Estimates within group for both verbal ability (PPVT-4) and processing ability (CTOPP) are presented in Table 1.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students were eligible for participation if their performance was at or below the 40th percentile on both the DIBELS and Measures of Academic Progress (MAP) reading test.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Read Naturally Software Edition was used to improve reading fluency. We also monitored reading accuracy and comprehension; however, only fluency was the target for intervention. The daily procedure for the Read Naturally Software Edition (SE) is as follows: 1. Select a Story: Students click a picture to select a story to read at their appropriate level; 2. Read Along to the Key Words: Students click on each of the vocabulary words to hear them pronounced and to learn what they mean. Students should read along as the words are pronounced and read the student-friendly definitions along with the narrator; 3. Write a Prediction: Students use the story illustration, key words, and title to write one or more sentences about what they think the story will say about the topic; 4. Cold Timing: The students are timed reading the unpracticed story aloud for one minute. Students are instructed to click difficult words (words they stop in front of, skip, or stumble on) as they read. The SE subtracts the number of errors, determined by the number of words the student clicks on, from the total number of words read to calculate their cold timing or baseline score. A graph displays their cold timing score in blue; 5. Read Along to the Story: The student reads along quietly while the SE plays a recording of the story. A highlighted reading guide automatically moves along to each sentence of the story as the student reads along with the synchronized audio recording. Subvocalizing along with the narrator on the recording ensures that students actually read along, and it helps them learn proper pronunciation, expression, and phrasing. This component is designed to build word recognition and word reading accuracy in preparation for subsequent reading practice. The SE includes three recordings appropriate to the readability level of the selected story, each one slightly faster than the previous one. Teachers can customize the read-along step for each student by adjusting the number of read alongs and selecting the reading speeds; 6. Practice Reading the Story: Students practice reading the story independently without the recording multiple times until they can read at the predetermined goal rate. A highlighted reading guide helps students focus on one line at a time, and students may click on a difficult word to hear the word pronounced during practice. The SE times each practice and indicates when students are ready to pass the timed reading by allowing them to move on to the next step; 7. Answer the Questions: Students answer five to nine multiple choice and open-ended questions about the story. For most of the levels, the questions follow a specific pattern, allowing for the teacher to detect when a student has trouble with particular types of questions. The SE corrects the multiple-choice questions. The teacher corrects the open-ended questions during the Pass step (step 9); 8. Retell the Story: In the sequenced levels, students retell or write what they learned from the story. Teachers can specify how much time is allowed for the retell step. If time is limited or students have inadequate keyboarding skills, teachers may choose to have students retell the story orally or skip this step; 9. Pass Timing: Students read the story aloud to the teacher in order to demonstrate that they can read the story at the goal rate. To pass a story, students must reach their goal, make no more than three errors, read with good expression, and answer the questions correctly. The teacher and student view the results together. If the student passes, a graph shows their pass timing score as a red bar above their cold timing score (the blue bar). If they do not pass, the teacher can assign some remedial work (e.g., further practice) and then test them again; 10. Repeat: Once students pass a story, they repeat these steps with a new story.

Specify which condition is the control condition:

No treatment control (business as usual)

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Students who qualified for the study, and whose parents gave permission for them to participate, were randomly assigned within their classrooms to the control group (54 students) or the intervention group (55 students).

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The four teachers were recommended by employees of the school districts involved in the study to guide implementation across the six sites. Three out of the four teachers had recently retired from the school districts. These teachers received a stipend for their participation in this study. The fourth teacher was on school staff and used available time in her schedule to implement the intervention. This school received additional levels of the Read Naturally software as compensation for the teacher’s time. All four of the implementation teachers attended a six-hour training session in December 2008. This is the typical initial training provided by Read Naturally consultants when the program is implemented in elementary school settings. The training included lecture and hands-on practice with the software. All the teachers were given the Read Naturally SE Self-Study Course to take home at the end of the training. This allowed for additional practice with the SE materials. Teachers were offered compensation for up to two additional hours to review the self-study course on their own.

Describe when and how fidelity of treatment information was obtained.

Trained observers conducted observations of intervention fidelity every two weeks to assess the integrity with which the intervention was implemented. Researchers used two fidelity checklists that were developed by Read Naturally to evaluate teacher implementation and student implementation. In addition to the information gathered by the observers, the software documented the time spent on the program and the way students progressed through the program. The integrity checks were carried out for both teacher behaviors and student behaviors. Teacher behaviors included Planning and Set up, Implementation, Monitoring Progress, Teacher Management, and Communication. Student behaviors included a General checklist and Intervention Steps checklist. See tables below for details.

What were the results on the fidelity-of-treatment implementation measure?

The tables that follow document the Percent Collected and Percent Endorsed. Percent Collected indicates how much of the data was collected as compared with data that was planned for collection. Percent Endorsed indicates the proportion of observations where the step occurred. Student fidelity was above 90% for all behavior except for “engaged for 30 min and work independently through the steps” (68%) and “students can pass with 3 to 10 repeated reading passages” (60%). Teacher fidelity was typically 100% and consistently above 90% for all except “session length is 30 min.” The actual session length averaged 20 min per day for each of 5 days.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

The initial study included pretest measures of vocabulary and processing speed to evaluate some relevant differences between groups (prior to intervention). There were no pretest differences on those measures. They were not readministered post-test so they are not reported here (as they were not measures of intervention effects).

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Multivariate Analysis of Covariance (MANCOVA), using pre-test performance as a covariate, was used to test for statistically significant differences (Wilks Criterion, α = .05) on post-test performance for the multivariate combinations of the measures for reading fluency (3), reading accuracy (2), and reading comprehension (2). Planned Analyses of Covariance (ANCOVAs) were used to test for statistically significant differences across the control and experimental conditions for all univariate measures if statistically significant differences of multivariate combination were observed. Hedges’ g was computed as the “covariate adjusted mean difference divided by unadjusted pooled within-group SD” (emphasis original, p. 37) and was used to estimate univariate effect sizes per the recommended procedures within the What Works Clearinghouse Procedures and Standards Handbook V2 (WWC, 2008). Those estimates can be interpreted such that effects that approximate .20 are small, .50 are modest and .80 or greater are strong (Cooper & Hedges, 1994). In addition, CBM-R measures are often administered repeatedly to estimate the rate of growth for individuals and groups of students. The CBM-R was administered at three points in time to facilitate analysis of slope, which were calculated to estimate the rate of weekly gain in units of WRCM (Figure 1). The mean (SD) level of performances within the control and experimental groups were 1.10 (1.00) and 1.53 (1.10) WRCM gain per week, which was a statistically significant difference and a small effect size, F(1, 99) = 4.58, p < .05, Hedges’ g = .18.