Read Naturally

Study: Christ & Davie (2009)

Christ, T. J., Davie, J. (2009) Empirical Evaluation of Read Naturally Effects: A Randomized Control Trial (RCT).
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

Read Naturally programs develop fluency, support vocabulary development, and promote comprehension using the research-based strategies of teacher modeling, repeated reading, and progress monitoring. Three versions are available. Read Naturally Encore Edition uses printed stories with audio support on audio CDs. Read Naturally Software Edition (SE) uses computer software that guides students through the steps of the program. Read Live is our web-based intervention version and assessment program.

Read Naturally is intended for use in grades 1 through high school. It is designed for use with students with disabilities, (including learning disabilities, intellectual disabilities, and behavioral disabilities), English language learners, and any student at risk of academic failure. The academic area of focus is reading comprehension and fluency.

Read Naturally is used in all 50 states and several countries. Thousands of districts have implemented Read Naturally since 1991.

Where to obtain:
Read Naturally
2945 Lone Oak Dr. #190
St. Paul, MN 55121-4902
Phone: 800-788-4085
Website: www.readnaturally.com
Email:  info@readnaturally.com

Cost:
One time pricing:

Read Naturally Encore: $129 per level including license for all students. Encore audio CD level contains 12 audio CDs, 24 nonfiction high interest stories, and Teachers Manual and Glossary.

Software Edition: $125 stand alone per level, Server (school $200 or district $1000) license for all students, $399 school site license per level for all students in school.

Read Live subscription: 
1 seat $149
6 seats $299
30 seats $599
130 seats $1999

Read Naturally is designed for use with individual students or small groups of six to eight students.

Read Naturally takes 30 minutes per session with a recommended three to five sessions per week for 36 weeks.

The program includes a highly specified teacher’s manual.

The Encore Edition requires a timer, CD player, and headphones.

The Software Edition requires a computer.

Read Live requires a computer and high speed internet access.

Four to eight hours of training is required for the instructor.

Instruction training options include: full day seminars; SE self study course; school district training; virtual seminars; video workshop for teachers; and video workshop for assistants and Read Live
Hands-on Training Workbook.

Instructors must have expertise in reading instruction and may be professionals, paraprofessionals, or volunteers.

Customer service support, virtual seminars, and follow-up training are available.

 

Participants: Unconvincing Evidence

Sample size: 109 students across six schools in four districts in third grade. (55 students in the treatment group and 54 students in the control group). 106 students completed pre- and post-testing.

Risk Status: Students were eligible for participation if their performance was at or below the 40th percentile on both the DIBELS and Measures of Academic Progress (MAP) reading test.

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 

 

 

 

 

  Grade 3

16

14

15

14

p > 0.05

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

16

14

15

14

p > 0.05

  American Indian

0

0

1

1

p > 0.05

  Asian/Pacific Islander

3

3

4

3

p > 0.05

  Hispanic

13

12

12

11

p > 0.05

  White

23

21

23

21

p > 0.05

  Other

 

 

 

 

 

Socioeconomic status

  Subsidized lunch

33

30

32

30

p > 0.05

  No subsidized lunch

22

20

22

20

p > 0.05

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

2

.02

2

.02

p > 0.05

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

53

48

52

48

p > 0.05

ELL status

  English language learner

13

12

12

12

p > 0.05

  Not English language learner

 

 

 

 

 

Gender

Female

30

27

30

27

p > 0.05

Male

25

23

24

22

p > 0.05

Training of Instructors: The four teachers were recommended by employees of the school districts involved in the study to guide implementation across the six sites. Three out of the four teachers had recently retired from the school districts. These teachers received a stipend for their participation in this study. The fourth teacher was on school staff and used available time in her schedule to implement the intervention. This school received additional levels of the Read Naturally software as compensation for the teacher’s time.

All four of the implementation teachers attended a six-hour training session in December 2008. This is the typical initial training provided by Read Naturally consultants when the program is implemented in elementary school settings. The training included lecture and hands-on practice with the software. All the teachers were given the Read Naturally SE Self-Study Course to take home at the end of the training. This allowed for additional practice with the SE materials. Teachers were offered compensation for up to two additional hours to review the self-study course on their own.

Design: Partially Convincing Evidence

Did the study use random assignment?: Yes.

If not, was it a tenable quasi-experiment?: Not applicable.

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: No.

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? Yes.

Were the program and control groups demographically comparable at pretreatment?: No.

Was there attrition bias1? No.

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes.

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Trained observers conducted observations of intervention fidelity every two weeks to assess the integrity with which the intervention was implemented. Researchers used two fidelity checklists that were developed by Read Naturally to evaluate teacher implementation and student implementation. In addition to the information gathered by the observers, the software documented the time spent on the program and the way students progressed through the program. The integrity checks were carried out for both teacher behaviors and student behaviors. Teacher behaviors included Planning and Set up, Implementation, Monitoring Progress, Teacher Management, and Communication. Student behaviors included a General checklist and Intervention Steps checklist. See tables below for details.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: The tables that follow document the Percent Collected and Percent Endorsed. Percent Collected indicates how much of the data was collected as compared with data that was planned for collection. Percent Endorsed indicates the proportion of observations where the step occurred.

Student fidelity was above 90% for all behavior except for “engaged for 30 min and work independently through the steps” (68%) and “students can pass with 3 to 10 repeated reading passages” (60%).

Teacher fidelity was typically 100% and consistently above 90% for all except “session length is 30 min.” The actual session length averaged 20 min per day for each of 5 days.

Teacher Responsibilities Fidelity Checklist (Read Naturally)

Items Percent Collecteda Percent Endorsed

Planning & Setting Up

 

Y/N

  • Session length is 30 minutes.

88

67

  • Students attend 5 sessions per week.

92

100

  • Comfortable workstations with working computers/software, headphones, etc.

91

100

  • Setting promotes students’ engagement for entire session.

95

100

  • Ratio of adults to students is no greater than 1:6.

97

99

  • System for indicating when students need teacher.

87

99

Implementing the Steps

 

 

1.  Select a Story—Teacher reinforces making a story selection after pass step on previous story—before quitting.

90

100

2.  Key Words—Teacher reminds students to click on each key word and listen to the definition and sample sentence.

81

92

3.  Prediction—Teacher encourages students to quickly write a brief prediction based on story title, key words, and illustration.  If keyboarding skills are weak, teacher may listen to an oral prediction or “take dictation” and type prediction for student—to allow more time on task for reading.

85

100

4.  Cold Timing—Teacher encourages students to click on difficult words and do an “honest” cold timing.  Teacher may choose to monitor some students’ cold timings or set options to require teacher for cold timing.

86

100

5.  Read Along—Teacher monitors Read Along to ensure students are reading quietly aloud and following the highlighted sentences with the audio.

87

97

6.  Practice—Teacher checks to see that students know their fluency goals and monitors practice to see that students read quietly aloud throughout each practice, click on difficult words, and click on last word read when bell sounds. Teacher confirms that student usually needs 3-10 practices to pass a story.

76

97

7.  Quiz—Teacher monitors to be sure students read and thoughtfully answer all questions. Teacher checks to be sure students are not just clicking multiple choice answers until the right one is chosen.

81

100

8.  Retell- Teacher sets a reasonable time limit and encourages students to write a main idea and several supporting sentences—not to spend too much time on retell, because the emphasis is to have the student spend most of the time reading.

79

100

9.  Pass-Teacher:

  • reminds students to continue practicing until she/he is available to come for hot timing.
  • counts errors, rates expression, and directs students to click on final word. 
  • goes to Pass Requirements Results page and directs student to review work, additional practice, and/or retest as indicated.

82

100

Monitoring Progress

 

 

Teacher monitors student progress regularly using SE reports* at these times:

  • when a student seems to be struggling with accuracy or goal rate
  • after first six stories in Read Naturally
  • after first 12 stories in a level
  • at the end of a level

(*i.e. Needs At-A-Glance for Students Report)

42

100

Decisions are made based on data:

  • Continue at same level & goal
  • Adjust the reading rate goal
  • Move to more difficult (or easier) material
  • Other decisions:
    • Rate of read-alongs
    • Teacher required for cold timing
    • Number of read-alongs

40

100

Teacher Management

 

 

Teacher uses Teacher Management to:

  • Set up class/student information
  • Set up story options (goal, level, SE options)
  • Reset stories
  • Place students (level and goal)
  • Print awards and parent letters
  • Print stories in a level
  • Generate reports

55

100

Communicating with Teachers, Students & Parents

 

 

  • Teacher interactions with students are positive and encouraging.

73

100

  • Teacher monitors students and provides assistance as needed.

73

100

  • Teacher confers with student before making a change in the program.

55

100

  • Teacher communicates with regular classroom teacher and parents by using appropriate reports and calling to discuss progress as necessary.

4

100

Note. Data were collected approximately every two weeks within each of six sites for a total of 72 possible observations.
a percent collected is the proportion of possible data collection instances for which data were available
b percent endorsed is the proportion of instances that a positive response (yes) was recorded

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted Measure Score type and range of measure Reliability statistics Relevance to program instructional content

Fluency

CBM-R – Words Correctly per Min

Raw scores in units of Words Read Correctly per minute.
Range (within sample), 4 to 134

The possible range for CBM-R is 0 to 300+

Test-retest & alternate form approximate or exceed 0.85

Most direct measure of oral reading fluency in context.

Widely used index of early reading achievement and reading fluency. Criterion related validity coefficients with other reading measures typically range from 0.45 to 0.88.

Fluency

TOWRE – Word Reading Efficiency

M = , SD =

Standard Scores

Average Alternate-form > 0.90 (range, 0.84 to 0.97);

Secondary measure of oral reading fluency with words in isolation.

Criterion related validity coefficients are typically in the range of 0.89 (with WRMT-R word ID task) and 0.75 with similar measures of fluency (GORT)

Fluency

GORT-4 – Fluency

M = 10 , SD = 3

Standard Scores

Mean Alpha ~ 0.93 (range, 0.90 to 0.96)

Test-retest range, 0.84 to 0.98

Secondary measure of oral reading rate in context. Similar to CBM-R, but scores transformed to standard score units.

Criterion validity on a variety of reading measures range from 0.39 to 0.72

Accuracy

GORT-4 – Accuracy

M =10 , SD =3

Standard Scores

Mean Alpha ~ 0.92 (range, 0.87 to 0.94)

Test-retest range, 0.82 to 0.97
Less relevant than fluency; however, criterion validity on a variety of reading measures range from 0.39 to 0.87

Accuracy

WRMT-R – Word Identification

M = 100, SD = 15

Standard Scores
Mean internal consistency of .91 (range, 0.61 to 0.98); Mean split half 0.95 (range, 0.87 to 0.98) Less relevant than fluency; however, criterion validity coefficients modest to robust with measures of broad reading achievement

 

Broader Measure Score type and range of measure Reliability statistics Relevance to program instructional content

GORT-4 Reading Comp

M = 10, SD = 3

Internal consistency above 0.90; split half above 0.80

Less relevant than fluency; however, comprehension is the goal of reading instruction.

WRMT-R Passage Comp

M = 100, SD = 15

Internal consistency above 0.90; split half above 0.80

Less relevant than fluency; however, comprehension is the goal of reading instruction.

If you have excluded a variable on which report data in the attached document, explain rationale for exclusion:

The initial study included pretest measures of vocabulary and processing speed to evaluate some relevant differences between groups (prior to intervention). There were no pretest differences on those measures. They were not readministered post-test so they are not reported here (as they were not measures of intervention effects).

Number of Outcome Measures: 7 Reading

Mean ES - Targeted: 0.38*

Mean ES - Broader: -0.07

Effect Size:

Targeted Measures

Construct Measure Effect Size
Reading CBM-R 0.22
Reading TOWRE Word Reading Efficiency 0.19
Reading GORT-4 Fluency 0.66***
Reading GORT-4 Accuracy 0.66***
Reading WRMT-R Word Identification 0.16

 Broader Measures

Construct Measure Effect Size
Reading GORT-4 Reading Comprehension 0.00
Reading WRMT-R Passage Comprehension -0.14

 

Key
*      p ≤ 0.05
**    p ≤ 0.01
***  p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small Group, (n=3-5)

Duration of Intervention: 30 minutes, 5 times a week, 8 weeks

Minimum Interventionist Requirements: Professional, 6 hours of training

Reviewed by WWC or E-ESSA: WWC

What Works Clearinghouse Review

Adolescent Literacy Evidence Protocol

Effectiveness: Read Naturally® was found to have potentially positive effects on general literacy achievement for adolescent readers.

Studies Reviewed: 1 study meets standards out of 4 studies total

Full Report

Beginning Reading Protocol

Effectiveness: Read Naturally® was found to have no discernible effects on fluency and reading comprehension.

Studies Reviewed: 5 studies meet standards out of 11 studies total

Full Report

English Language Learners Protocol

Effectiveness: Read Naturally® was found to have no discernible effects on reading achievement and English language development of elementary school English language learners.

Studies Reviewed: 2 studies meet standards out of 3 studies total

Full Report

Students with Learning Disabilities Protocol

Effectiveness: Read Naturally® was found to have no discernible effects on reading fluency and potentially positive effects on writing for students with learning disabilities.

Studies Reviewed: 2 studies meet standards out of 3 studies total

Full Report

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

Other Research: Potentially Eligible for NCII Review: 6 studies

Arvans, R. (2010). Improving reading fluency and comprehension in elementary students using Read Naturally. Dissertation Abstracts International, 71(01B), 74-649.
 

Chenault, B., Thomson, J., Abbott, R. D., & Berninger, V. W. (2006). Effects of prior attention training on child dyslexics’ response to composition instruction. Developmental Neuropsychology, 29(1), 243–260.
 

Denton, C. A., Anthony, J. L., Parker, R., & Hasbrouck, J. E. (2004). Effects of two tutoring programs on the English reading development of Spanish-English bilingual students. The Elementary School Journal, 104(4), 289–305.
 

Hancock, C. M. (2002). Accelerating reading trajectories: The effects of dynamic research-based instruction. Dissertation Abstracts International, 63(6), 2139A. (UMI No. 3055690)
 

Kemp, S. C. (2006). Teaching to Read Naturally: Examination of a fluency training program for third grade students (Doctoral dissertation, University of California, Irvine and University of California, Los Angeles, 2006). Dissertation Abstracts International, 67(7A), 95-2447.
 

Mesa, C. L. (2004). Effect of Read Naturally software on reading fluency and comprehension. Unpublished master’s thesis, Piedmont College, Demorest, GA.