Leveled Literacy Intervention System
Study: Ransford-Kaldon et al. (2010)

Summary

The Fountas & Pinnell Leveled Literacy Intervention System (LLI) is a small-group, supplementary intervention designed for children who perform below grade-level expectations in reading and writing. LLI is designed and has been proven to bring children quickly to grade-level competency through 30-minute lessons delivered 5 days/week for 14 to 18 weeks on average. LLI serves those students who need intensive support to achieve grade-level competency. Studies have confirmed that LLI improves reading achievement in children from various socio-economic backgrounds, English language learners and children with special needs. Through explicit instruction in reading, writing and word work combined with opportunities for increased language modeling and oral language development, students are moved quickly toward grade level goals. Specific strategies for English language learners are included in the instructional plan. Three systems each support instruction at different levels on the Fountas & Pinnell A–Z Text Level Gradient™: • Orange System: Levels A through C - Kindergarten • Green System: Levels A through J – Grade One • Blue System: Levels C through N – Grade Two Leveled books are a key component in helping children become competent readers. Each LLI system includes a collection of carefully developed and expertly leveled books based on ten text characteristics to provide enough support and challenge for the reader so that he/she can be successful and make steps toward grade-level goals. Assessment is an ongoing process in LLI and is tied to the Continuum of Literacy Learning, the instructional framework for the systems. Teachers are provided with goals and objectives for each lesson, observational suggestions, and resources to conduct a reading record weekly with each child. Progress is managed and monitored through the Classroom Management System, a computer-based resource that collects student data and reports results while aiding teachers in making instructional decisions. The Fountas and Pinnell Benchmark Assessment System (BAS) is recommended, but not required, to be used with LLI to screen and place students at the appropriate level in LLI and to monitor their ongoing progress. BAS has been proven to be a reliable, effective tool to corroborate the results of the intervention while providing valuable data on each child’s reading levels and reading progress Professional Development is embedded throughout the system through clear, explicit instructional lessons, classroom videos that model best practices, the Prompting Guide that offers clear and precise language to support student interactions, and professional books that build teacher expertise. In addition, fee-based professional development is offered through Heinemann as well as Lesley and Ohio State Universities’ Literacy Collaborative.

Target Grades:
K, 1, 2, 3, 4
Target Populations:
  • Students with disabilities only
  • English language learners
  • Any student at risk for academic failure
Area(s) of Focus:
  • Phonological awareness
  • Phonics/word study
  • Comprehension
  • Fluency
  • Vocabulary
  • Spelling
  • Other: Oral Language Development
  • Spelling
  • Sentence construction
  • Other: writing in response to reading to increase comprehension
Where to Obtain:
Heinemann
PO Box 6926 Portsmouth, NH 03802-6926
800-225-5800
http://www.fountasandpinnell.com
Initial Cost:
$84.79 per student
Replacement Cost:
$29.24 per student per set of materials

Each grade level System is priced as a package and is inclusive of all materials necessary for instructional intervention of 3 students per group. The list price for 2010 is $1787.50 for Kindergarten (Orange); $2,750 for First Grade (Green); and $3,093.75 for Second Grade (Blue). Special pricing is available for online purchases. All resources are available for sale separately to accommodate replacement, etc. Each System contains 4 copies of each little book utilized for instruction along with 6-copies of the same book in black & white format for take-home/classroom use. All together there are over 300 different titles across the 3 Systems. In addition, the following resources support instruction: 10 Lap Books - Supports Getting Started Lessons at levels Orange and Green; Program Guide - complete overview of the program; Classroom Guide - 30-minute lessons and assessment resources for the daily intervention; Take-Home Storage Bags – carry take-home books and fold sheets; Lesson and Student Folders – teacher organization; My Writing books – support writing development; F & P Calculator/Stop Watch – facilitate reading records; Prompting Guide 1 – facilitates teacher language critical to targeted instruction; When Reader’s Struggle – Professional Book; Technology Package – includes Lesson Resources CD-Rom, Data Management System CD-ROM and Professional Development DVD & Assessment Tutorial. The average cost per student is $84.79, based on 4 groups of 3 students for two different 18 week sessions. The replacement cost per student for subsequent use is $29.24 (take home bags, little books, and writing books).

Staff Qualified to Administer Include:
  • Special Education Teacher
  • General Education Teacher
  • Reading Specialist
  • Math Specialist
  • EL Specialist
  • Interventionist
  • Student Support Services Personnel (e.g., counselor, social worker, school psychologist, etc.)
Training Requirements:
Training not required

In addition to the PD included with the system, fee-based, on- site and off-site training is available through Heinemann Professional Development. These sessions are typically 2-3 days in length conducted by Fountas & Pinnell trained consultants to support implementation. Ongoing training may be customized.


Fidelity of implementation was a goal of the CREP study, 2009-2010. 3 CREP researchers facilitated the selection of students and training of teachers based on the developers’ implementation guidelines. Pre- and Post- tests as well as surveys afforded researchers insight into the quality of materials and training.

Access to Technical Support:
Through Heinemann Professional Development, the Literacy Collaborative at Ohio State and Lesley Universities and by interactions with the Heinemann/Fountas and Pinnell website.
Recommended Administration Formats Include:
  • Small group of students
Minimum Number of Minutes Per Session:
30
Minimum Number of Sessions Per Week:
4
Minimum Number of Weeks:
14
Detailed Implementation Manual or Instructions Available:
Yes
Is Technology Required?
  • Computer or tablet

Program Information

Descriptive Information

Please provide a description of program, including intended use:

The Fountas & Pinnell Leveled Literacy Intervention System (LLI) is a small-group, supplementary intervention designed for children who perform below grade-level expectations in reading and writing. LLI is designed and has been proven to bring children quickly to grade-level competency through 30-minute lessons delivered 5 days/week for 14 to 18 weeks on average. LLI serves those students who need intensive support to achieve grade-level competency. Studies have confirmed that LLI improves reading achievement in children from various socio-economic backgrounds, English language learners and children with special needs. Through explicit instruction in reading, writing and word work combined with opportunities for increased language modeling and oral language development, students are moved quickly toward grade level goals. Specific strategies for English language learners are included in the instructional plan. Three systems each support instruction at different levels on the Fountas & Pinnell A–Z Text Level Gradient™: • Orange System: Levels A through C - Kindergarten • Green System: Levels A through J – Grade One • Blue System: Levels C through N – Grade Two Leveled books are a key component in helping children become competent readers. Each LLI system includes a collection of carefully developed and expertly leveled books based on ten text characteristics to provide enough support and challenge for the reader so that he/she can be successful and make steps toward grade-level goals. Assessment is an ongoing process in LLI and is tied to the Continuum of Literacy Learning, the instructional framework for the systems. Teachers are provided with goals and objectives for each lesson, observational suggestions, and resources to conduct a reading record weekly with each child. Progress is managed and monitored through the Classroom Management System, a computer-based resource that collects student data and reports results while aiding teachers in making instructional decisions. The Fountas and Pinnell Benchmark Assessment System (BAS) is recommended, but not required, to be used with LLI to screen and place students at the appropriate level in LLI and to monitor their ongoing progress. BAS has been proven to be a reliable, effective tool to corroborate the results of the intervention while providing valuable data on each child’s reading levels and reading progress Professional Development is embedded throughout the system through clear, explicit instructional lessons, classroom videos that model best practices, the Prompting Guide that offers clear and precise language to support student interactions, and professional books that build teacher expertise. In addition, fee-based professional development is offered through Heinemann as well as Lesley and Ohio State Universities’ Literacy Collaborative.

The program is intended for use in the following age(s) and/or grade(s).

not selected Age 0-3
not selected Age 3-5
selected Kindergarten
selected First grade
selected Second grade
selected Third grade
selected Fourth grade
not selected Fifth grade
not selected Sixth grade
not selected Seventh grade
not selected Eighth grade
not selected Ninth grade
not selected Tenth grade
not selected Eleventh grade
not selected Twelth grade


The program is intended for use with the following groups.

selected Students with disabilities only
not selected Students with learning disabilities
not selected Students with intellectual disabilities
not selected Students with emotional or behavioral disabilities
selected English language learners
selected Any student at risk for academic failure
not selected Any student at risk for emotional and/or behavioral difficulties
not selected Other
If other, please describe:

ACADEMIC INTERVENTION: Please indicate the academic area of focus.

Early Literacy

not selected Print knowledge/awareness
not selected Alphabet knowledge
not selected Phonological awareness
not selected Phonological awarenessEarly writing
not selected Early decoding abilities
not selected Other

If other, please describe:

Language

not selected Expressive and receptive vocabulary
not selected Grammar
not selected Syntax
not selected Listening comprehension
not selected Other
If other, please describe:

Reading

selected Phonological awareness
selected Phonics/word study
selected Comprehension
selected Fluency
selected Vocabulary
selected Spelling
selected Other
If other, please describe:
Oral Language Development

Mathematics

not selected Computation
not selected Concepts and/or word problems
not selected Whole number arithmetic
not selected Comprehensive: Includes computation/procedures, problem solving, and mathematical concepts
not selected Algebra
not selected Fractions, decimals (rational number)
not selected Geometry and measurement
not selected Other
If other, please describe:

Writing

not selected Handwriting
selected Spelling
selected Sentence construction
not selected Planning and revising
selected Other
If other, please describe:
writing in response to reading to increase comprehension

BEHAVIORAL INTERVENTION: Please indicate the behavior area of focus.

Externalizing Behavior

not selected Physical Aggression
not selected Verbal Threats
not selected Property Destruction
not selected Noncompliance
not selected High Levels of Disengagement
not selected Disruptive Behavior
not selected Social Behavior (e.g., Peer interactions, Adult interactions)
not selected Other
If other, please describe:

Internalizing Behavior

not selected Depression
not selected Anxiety
not selected Social Difficulties (e.g., withdrawal)
not selected School Phobia
not selected Other
If other, please describe:

Acquisition and cost information

Where to obtain:

Address
PO Box 6926 Portsmouth, NH 03802-6926
Phone Number
800-225-5800
Website
http://www.fountasandpinnell.com

Initial cost for implementing program:

Cost
$84.79
Unit of cost
student

Replacement cost per unit for subsequent use:

Cost
$29.24
Unit of cost
student
Duration of license
set of materials

Additional cost information:

Describe basic pricing plan and structure of the program. Also, provide information on what is included in the published program, as well as what is not included but required for implementation (e.g., computer and/or internet access)

Each grade level System is priced as a package and is inclusive of all materials necessary for instructional intervention of 3 students per group. The list price for 2010 is $1787.50 for Kindergarten (Orange); $2,750 for First Grade (Green); and $3,093.75 for Second Grade (Blue). Special pricing is available for online purchases. All resources are available for sale separately to accommodate replacement, etc. Each System contains 4 copies of each little book utilized for instruction along with 6-copies of the same book in black & white format for take-home/classroom use. All together there are over 300 different titles across the 3 Systems. In addition, the following resources support instruction: 10 Lap Books - Supports Getting Started Lessons at levels Orange and Green; Program Guide - complete overview of the program; Classroom Guide - 30-minute lessons and assessment resources for the daily intervention; Take-Home Storage Bags – carry take-home books and fold sheets; Lesson and Student Folders – teacher organization; My Writing books – support writing development; F & P Calculator/Stop Watch – facilitate reading records; Prompting Guide 1 – facilitates teacher language critical to targeted instruction; When Reader’s Struggle – Professional Book; Technology Package – includes Lesson Resources CD-Rom, Data Management System CD-ROM and Professional Development DVD & Assessment Tutorial. The average cost per student is $84.79, based on 4 groups of 3 students for two different 18 week sessions. The replacement cost per student for subsequent use is $29.24 (take home bags, little books, and writing books).

Program Specifications

Setting for which the program is designed.

not selected Individual students
selected Small group of students
not selected BI ONLY: A classroom of students

If group-delivered, how many students compose a small group?

   3

Program administration time

Minimum number of minutes per session
30
Minimum number of sessions per week
4
Minimum number of weeks
14
not selected N/A (implemented until effective)

If intervention program is intended to occur over less frequently than 60 minutes a week for approximately 8 weeks, justify the level of intensity:

Does the program include highly specified teacher manuals or step by step instructions for implementation?
Yes

BEHAVIORAL INTERVENTION: Is the program affiliated with a broad school- or class-wide management program?

If yes, please identify and describe the broader school- or class-wide management program:

Does the program require technology?
Yes

If yes, what technology is required to implement your program?
selected Computer or tablet
not selected Internet connection
not selected Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:
Lesson Resource CD provides specific resources that are to be printed out for LLI lessons as needed. This resource accompanies the program. Other technology included with the program is not required, but highly recommended for use (Data Management CD and Professional Development DVD & Tutorials.

Training

How many people are needed to implement the program ?

Is training for the instructor or interventionist required?
No
If yes, is the necessary training free or at-cost?

Describe the time required for instructor or interventionist training:
Training is not required; 2-3 days of training recommended

Describe the format and content of the instructor or interventionist training:
In addition to the PD included with the system, fee-based, on- site and off-site training is available through Heinemann Professional Development. These sessions are typically 2-3 days in length conducted by Fountas & Pinnell trained consultants to support implementation. Ongoing training may be customized.

What types or professionals are qualified to administer your program?

selected Special Education Teacher
selected General Education Teacher
selected Reading Specialist
selected Math Specialist
selected EL Specialist
selected Interventionist
selected Student Support Services Personnel (e.g., counselor, social worker, school psychologist, etc.)
not selected Applied Behavior Analysis (ABA) Therapist or Board Certified Behavior Analyst (BCBA)
not selected Paraprofessional
not selected Other

If other, please describe:

Does the program assume that the instructor or interventionist has expertise in a given area?
No   

If yes, please describe: 


Are training manuals and materials available?
Yes

Describe how the training manuals or materials were field-tested with the target population of instructors or interventionist and students:
Fidelity of implementation was a goal of the CREP study, 2009-2010. 3 CREP researchers facilitated the selection of students and training of teachers based on the developers’ implementation guidelines. Pre- and Post- tests as well as surveys afforded researchers insight into the quality of materials and training.

Do you provide fidelity of implementation guidance such as a checklist for implementation in your manual?

Can practitioners obtain ongoing professional and technical support?
Yes

If yes, please specify where/how practitioners can obtain support:

Through Heinemann Professional Development, the Literacy Collaborative at Ohio State and Lesley Universities and by interactions with the Heinemann/Fountas and Pinnell website.

Summary of Evidence Base

Please identify, to the best of your knowledge, all the research studies that have been conducted to date supporting the efficacy of your program, including studies currently or previously submitted to NCII for review. Please provide citations only (in APA format); do not include any descriptive information on these studies. NCII staff will also conduct a search to confirm that the list you provide is accurate.

Study Information

Study Citations

Ransford-Kaldon, C. R., Flynt, E. S., Ross, C. L., Franceschini, L. A., Zoblotsky, T. A., Huang, Y. & Gallagher, B. (2010). Implementation of Effective Intervention: An Empirical Study to Evaluate the Efficacy of Fountas and Pinnell’s Leveled Literacy Intervention (LLI). Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Participants Empty Bobble

Describe how students were selected to participate in the study:
The study employed a randomized controlled trial, mixed-methods design, which included both quantitative and qualitative data and allowed students to be randomly selected for the treatment (i.e., LLI in the first semester) or control (i.e., LLI in the second semester, if needed) condition. A matched-pair design was also utilized to ensure equivalency between treatment and control groups, and pre-post comparisons of student achievement in literacy were conducted. (see details in the study ie. Summary, pg 2 and Full pgs 16-24)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):
At the beginning of the 2009-2010 school year, each district provided the researchers with a list of first and second grade students that they had identified as eligible for LLI using their own selection criteria, and whose parents had provided consent to participate in the study. Pre-testing of these students with the LLI Benchmarks and DIBELS began during the first three weeks of school. Kindergarten children were selected in the late winter of 2010 in the same manner as noted above.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:
  • below the 30th percentile on local or national norm, or
  • identified disability related to the focus of the intervention?
%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:
  • emotional disability label,
  • placed in an alternative school/classroom,
  • non-responsive to Tiers 1 and 2, or
  • designation of severe problem behaviors on a validated scale or through observation?
%

Specify which condition is the submitted intervention:
Children identified for the treatment group received Leveled Literacy Intervention (LLI) as their intervention in addition to regular classroom instruction. No other pull-out intervention was provided during this period.

Specify which condition is the control condition:
Children identified in the control group received only regular classroom instruction. During the study, no additional pull-out intervention was provided. Control group students did not receive LLI until the first and second grade evaluation period ended (they are referred to as delayed-LLI/control).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Age less than 1
Age 1
Age 2
Age 3
Age 4
Age 5
Kindergarten 76 70 0.00
Grade 1 65 65 0.09
Grade 2 81 70 0.05
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Grade 12

Race–Ethnicity

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
African American 74 68 0.00
American Indian 0 0 0.00
Asian/Pacific Islander 1 1 0.00
Hispanic 84 74 0.05
White 61 59 0.06
Other 2 2 0.00

Socioeconomic Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Subsidized Lunch 201 177 0.30
No Subsidized Lunch 21 27 0.25

Disability Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Speech-Language Impairments
Learning Disabilities
Behavior Disorders
Emotional Disturbance
Intellectual Disabilities
Other 19 16 0.08
Not Identified With a Disability 203 188 0.08

ELL Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
English Language Learner 25 32 0.26
Not English Language Learner 197 172 0.26

Gender

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Female 104 89 0.10
Male 118 116 0.10

Mean Effect Size

0.10

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences between groups in the descriptions below, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not demographic characteristics, please describe the results of those analyses here.

Design Full Bobble

What method was used to determine students' placement in treatment/control groups?
Random
Please describe the assignment method or the process for defining treatment/comparison groups.
Subsequently, CREP conducted the randomization of the matched pairs of first and second graders based on demographic characteristics (i.e., gender, ethnicity, ELL status, special education status, and free/reduced lunch status) and pre-test LLI benchmark scores of instructional reading level. Students in the treatment group were then placed in LLI groups by LLI teachers, and the planned 90 days of LLI instruction for first and second graders began. Control group students did not receive LLI until the first and second grade evaluation period ended, and neither treatment nor control students received any additional pull-out literacy interventions during the study period.

What was the unit of assignment?
Students
If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?
not selected Schools
not selected Teachers
selected Students
not selected Classes
not selected Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

Fidelity of Implementation Half Bobble

How was the program delivered?
not selected Individually
selected Small Group
not selected Classroom

If small group, answer the following:

Average group size
3
Minimum group size
Maximum group size

What was the duration of the intervention (If duration differed across participants, settings, or behaviors, describe for each.)?

Weeks
16.00
Sessions per week
5.00
Duration of sessions in minutes
30.00
What were the background, experience, training, and ongoing support of the instructors or interventionists?
The LLIOT, developed by CREP researchers for the purposes of the study, involves a targeted, 30 minute observation of a randomly selected LLI lesson. The LLIOT is used to rate LLI teachers’ fidelity to the LLI program model as well as the quality of their literacy instructional strategies and learning environment of the lesson. Ratings are provided using a 4-point scale that ranges from 0 (Not Observed) to 3 (Excellent). Containing 20 items, the LLIOT is comprised of 3 subscales: Quality of LLI Implementation, which is designed to measure LLI teachers’ implementation of the 10 main LLI lesson components; Literacy Instructional Strategies, which is designed to assess LLI teachers’ use of general teaching strategies that should be present in a successful literacy intervention; and Learning Environment, which is designed to assess the quality of lesson factors such as organization, pacing, and the availability of materials. On-site researchers trained by CREP conducted observations of two intervention sessions with each participating LLI group, one near the beginning of the study period and one near the end, using the LLIOT. This observation data contributed to the evaluation of fidelity to the LLI model. To ensure the reliability of data, observers received a manual which provided definitions of terms, examples and explanations of target strategies, and a description of procedures for completing the instruments. Observers also received instruction on the instrument in a group session and participated in practice exercises. In addition, the LLI Data Management System and select focus groups supplied information on fidelity.

Describe when and how fidelity of treatment information was obtained.
The Leveled Literacy Intervention Observation Tool (LLIOT) involved a targeted, 30-minute observation of LLI program implementation and instructional strategies (n = 160 observations). See Table 30 in the 2010 publication by Ransford- Kaldon et. al to view the frequencies for each item on the LLIOT, as observed during the visits. The results from the LLIOT revealed that 7 of the 10 LLI lesson components were rated “Acceptable” or “Excellent” over 90% of the time, indicating a high level of program implementation fidelity across both districts. The highest rated lesson components (i.e., those demonstrating the highest degree of implementation fidelity) included writing about reading, phonics/word work, and reading a new book, which were rated “Acceptable” or “Excellent” 98.8%, 97.6%, and 95.7% of the time, respectively. The lowest rated lesson components (i.e., those demonstrating the lowest degree of implementation fidelity) included classroom and home connections, which were not observed 51.9% and 22.5% of the time, respectively. Teachers were also rated highly on their use of literacy instructional strategies, such as modeling and encouraging fluent oral reading (96.9% “Acceptable” or “Excellent”) and appropriate reading strategies (95.7%) and assisting students in problem-solving (95.6%). Further, in the majority of observed lessons, instructional materials were readily available; the lesson was well-organized; and students were engaged and attentive (100.0%, 99.4%, and 98.1% “Acceptable” or “Excellent,” respectively). Overall, observers perceived that the lesson was delivered as designed 96.3% of the time.

What were the results on the fidelity-of-treatment implementation measure?
The Leveled Literacy Intervention Observation Tool (LLIOT) involved a targeted, 30-minute observation of LLI program implementation and instructional strategies (n = 160 observations). Table 30 illustrates the frequencies for each item on the LLIOT, as observed during the visits. The results from the LLIOT revealed that 7 of the 10 LLI lesson components were rated “Acceptable” or “Excellent” over 90% of the time, indicating a high level of program implementation fidelity across both districts. The highest rated lesson components (i.e., those demonstrating the highest degree of implementation fidelity) included writing about reading, phonics/word work, and reading a new book, which were rated “Acceptable” or “Excellent” 98.8%5, 97.6%, and 95.7% of the time, respectively. The lowest rated lesson components (i.e., those demonstrating the lowest degree of implementation fidelity) included classroom and home connections, which were not observed 51.9% and 22.5% of the time, respectively. Teachers were also rated highly on their use of literacy instructional strategies, such as modeling and encouraging fluent oral reading (96.9% “Acceptable” or “Excellent”) and appropriate reading strategies (95.7%) and assisting students in problem-solving (95.6%). Further, in the majority of observed lessons, instructional materials were readily available; the lesson was well-organized; and students were engaged and attentive (100.0%, 99.4%, and 98.1% “Acceptable” or “Excellent,” respectively). Overall, observers perceived that the lesson was delivered as designed 96.3% of the time. All items can be found in Table 30, pages 38-39.

Was the fidelity measure also used in control classrooms?

Measures and Results

Measures Targeted : Full Bobble
Measures Broader : Full Bobble

Study measures are classified as targeted, broader, or administrative data according to the following definitions:

  • Targeted measures
    Assess outcomes, such as competencies or skills that the program was directly targeted to improve.
    • In the academic domain, targeted measures typically are not the very items taught but rather novel items structured similarly to the content addressed in the program. For example, if a program taught word-attack skills, a targeted measure would be decoding of pseudo words. If a program taught comprehension of cause-effect passages, a targeted measure would be answering questions about cause-effect passages structured similarly to those used during intervention, but not including the very passages used for intervention.
    • In the behavioral domain, targeted measures evaluate aspects of external or internal behavior the program was directly targeted to improve and are operationally defined.
  • Broader measures
    Assess outcomes that are related to the competencies or skills targeted by the program but not directly taught in the program.
    • In the academic domain, if a program taught word-level reading skill, a broader measure would be answering questions about passages the student reads. If a program taught calculation skill, a broader measure would be solving word problems that require the same kinds of calculation skill taught in the program.
    • In the behavioral domain, if a program taught a specific skill like on-task behavior in one classroom, a broader measure would be academic performance in that setting or on-task behavior in another setting.
  • Administrative data measures apply only to behavioral intervention tools and are measures such as office discipline referrals (ODRs) and graduation rates which do not have psychometric properties as do other, more traditional targeted or broader measures.

Click here for more information on effect size.


What populations are you submitting outcome data for?
selected Full sample
not selected Students at or below the 20th percentile
not selected English language learners
not selected Racial/ethnic subgroups
not selected Economically disadvantaged students (low socioeconomic status)
Targeted Measure Reverse Coded? Reliability Relevance Exposure
Broader Measure Reverse Coded? Reliability Relevance Exposure
Administrative Data Measure Reverse Coded? Relevance

Posttest Data

Targeted Measures (Full Sample)

Measure Sample Type Effect Size P

Broader Measures (Full Sample)

Measure Sample Type Effect Size P

Administrative Measures (Full Sample)

Measure Sample Type Effect Size P

Targeted Measures (Subgroups)

Measure Sample Type Effect Size P

Broader Measures (Subgroups)

Measure Sample Type Effect Size P

Administrative Measures (Subgroups)

Measure Sample Type Effect Size P
For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.
Please explain any missing data or instances of measures with incomplete pre- or post-test data.
If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:
Describe the analyses used to determine whether the intervention produced changes in student outcomes:
As the LLI benchmarks were scored in terms of alphabetic levels (i.e., pre-A, A, B, C, etc.), these outcomes first had to be recoded into numeric equivalents before analysis. Additionally, because some students were unable to reach the initial benchmark Level A as measured in the LLI benchmark system, we created a new category, pre-A benchmark level, in order to assign scores to those who were below Level A so those students could be included in the study. All benchmark outcomes were assigned numeric equivalents for each grade level before a series of mixed (i.e., “one between groups”/”one within groups”) analysis of variance (ANOVA) procedures was conducted on the transformed measures to determine whether larger gains were observed for one of the two conditions overall (i.e., LLI/treatment vs. delayed-LLI/control) and for several demographic subgroups nested within the two conditions (e.g., ethnicity, special education status, English Language Learner status). Also, variations in the sample sizes across each analysis were seen due to limited cases of missing data. In the total sample, any cases with missing data could not be included in the analysis. Missing data resulted from several situations: 1) only cases with both pre-test and post-test data were able to be included in the analyses; 2) both achievement measures had “frustration” level cut-offs, which meant some students may not have had a score if they could not meet the minimum frustration level; and 3) students were allowed to voluntarily participate in the testing. Tests for normality of data and statistical assumptions (i.e., normal distribution; independence of measures) as well as measures of central tendency (i.e., means, standard deviations) were conducted on all outcomes for each grade level prior to the series of mixed ANOVAs. A series of mixed ANOVAs was also conducted on the means of the 4 DIBELS measures in kindergarten (ISF, LNF, PSF, and NWF), the 4 DIBELS measures in 1st grade (LNF, PSF, NWF, and ORF), and the 2 DIBELS measures in 2nd grade (NWF and ORF), both overall and by demographic subgroup. Additionally, analyses were conducted on treatment and control group difference scores (i.e., pre-test to post-test difference) in order to determine if any significant gain, or rate of change over time, was found for either group. From the pre- and post-test outcomes on the benchmark tests and DIBELS measures, difference scores were computed and analyzed for treatment and control group students in the aggregate, as well as by demographic subgroup.

Additional Research

Is the program reviewed by WWC or E-ESSA?
WWC & E-ESSA
Summary of WWC / E-ESSA Findings :

What Works Clearinghouse Review

Beginning Reading

Effectiveness: Leveled Literacy Intervention had positive effects on general reading achievement, potentially positive effects on reading fluency, and no discernible effects on alphabetics for beginning readers.

Studies Reviewed: 2 studies met standards out of 10 studies total.

Full Report

Evidence for ESSA

Program Outcomes: LLI has been evaluated in two qualifying studies. In one, in rural and suburban Georgia and New York, students were randomly assigned to LLI or control conditions. Across 5 DIBELS scales, the average effect size was +0.17, with significant differences on Non-Word Fluency and Oral Reading Fluency. In a second study in Denver, there were very positive outcomes on the DRA2 in kindergarten but not in first or second grade, for a significant but small meaningful effect size of +0.10. Averaging the two studies, the effect size was +0.13.

Number of Studies: 2

Average Effect Size: 0.13

Full Report
How many additional research studies are potentially eligible for NCII review?
0
Citations for Additional Research Studies :

Disclaimer

Most tools and programs evaluated by the NCII are branded products which have been submitted by the companies, organizations, or individuals that disseminate these products. These entities supply the textual information shown above, but not the ratings accompanying the text. NCII administrators and members of our Technical Review Committees have reviewed the content on this page, but NCII cannot guarantee that this information is free from error or reflective of recent changes to the product. Tools and programs have the opportunity to be updated annually or upon request.