Leveled Literacy Intervention System
Study: Ransford-Kaldon et al. (2010)

Describe how students were selected to participate in the study:

The study employed a randomized controlled trial, mixed-methods design, which included both quantitative and qualitative data and allowed students to be randomly selected for the treatment (i.e., LLI in the first semester) or control (i.e., LLI in the second semester, if needed) condition. A matched-pair design was also utilized to ensure equivalency between treatment and control groups, and pre-post comparisons of student achievement in literacy were conducted. (see details in the study ie. Summary, pg 2 and Full pgs 16-24)

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

At the beginning of the 2009-2010 school year, each district provided the researchers with a list of first and second grade students that they had identified as eligible for LLI using their own selection criteria, and whose parents had provided consent to participate in the study. Pre-testing of these students with the LLI Benchmarks and DIBELS began during the first three weeks of school. Kindergarten children were selected in the late winter of 2010 in the same manner as noted above.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Children identified for the treatment group received Leveled Literacy Intervention (LLI) as their intervention in addition to regular classroom instruction. No other pull-out intervention was provided during this period.

Specify which condition is the control condition:

Children identified in the control group received only regular classroom instruction. During the study, no additional pull-out intervention was provided. Control group students did not receive LLI until the first and second grade evaluation period ended (they are referred to as delayed-LLI/control).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Subsequently, CREP conducted the randomization of the matched pairs of first and second graders based on demographic characteristics (i.e., gender, ethnicity, ELL status, special education status, and free/reduced lunch status) and pre-test LLI benchmark scores of instructional reading level. Students in the treatment group were then placed in LLI groups by LLI teachers, and the planned 90 days of LLI instruction for first and second graders began. Control group students did not receive LLI until the first and second grade evaluation period ended, and neither treatment nor control students received any additional pull-out literacy interventions during the study period.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The LLIOT, developed by CREP researchers for the purposes of the study, involves a targeted, 30 minute observation of a randomly selected LLI lesson. The LLIOT is used to rate LLI teachers’ fidelity to the LLI program model as well as the quality of their literacy instructional strategies and learning environment of the lesson. Ratings are provided using a 4-point scale that ranges from 0 (Not Observed) to 3 (Excellent). Containing 20 items, the LLIOT is comprised of 3 subscales: Quality of LLI Implementation, which is designed to measure LLI teachers’ implementation of the 10 main LLI lesson components; Literacy Instructional Strategies, which is designed to assess LLI teachers’ use of general teaching strategies that should be present in a successful literacy intervention; and Learning Environment, which is designed to assess the quality of lesson factors such as organization, pacing, and the availability of materials. On-site researchers trained by CREP conducted observations of two intervention sessions with each participating LLI group, one near the beginning of the study period and one near the end, using the LLIOT. This observation data contributed to the evaluation of fidelity to the LLI model. To ensure the reliability of data, observers received a manual which provided definitions of terms, examples and explanations of target strategies, and a description of procedures for completing the instruments. Observers also received instruction on the instrument in a group session and participated in practice exercises. In addition, the LLI Data Management System and select focus groups supplied information on fidelity.

Describe when and how fidelity of treatment information was obtained.

The Leveled Literacy Intervention Observation Tool (LLIOT) involved a targeted, 30-minute observation of LLI program implementation and instructional strategies (n = 160 observations). See Table 30 in the 2010 publication by Ransford- Kaldon et. al to view the frequencies for each item on the LLIOT, as observed during the visits. The results from the LLIOT revealed that 7 of the 10 LLI lesson components were rated “Acceptable” or “Excellent” over 90% of the time, indicating a high level of program implementation fidelity across both districts. The highest rated lesson components (i.e., those demonstrating the highest degree of implementation fidelity) included writing about reading, phonics/word work, and reading a new book, which were rated “Acceptable” or “Excellent” 98.8%, 97.6%, and 95.7% of the time, respectively. The lowest rated lesson components (i.e., those demonstrating the lowest degree of implementation fidelity) included classroom and home connections, which were not observed 51.9% and 22.5% of the time, respectively. Teachers were also rated highly on their use of literacy instructional strategies, such as modeling and encouraging fluent oral reading (96.9% “Acceptable” or “Excellent”) and appropriate reading strategies (95.7%) and assisting students in problem-solving (95.6%). Further, in the majority of observed lessons, instructional materials were readily available; the lesson was well-organized; and students were engaged and attentive (100.0%, 99.4%, and 98.1% “Acceptable” or “Excellent,” respectively). Overall, observers perceived that the lesson was delivered as designed 96.3% of the time.

What were the results on the fidelity-of-treatment implementation measure?

The Leveled Literacy Intervention Observation Tool (LLIOT) involved a targeted, 30-minute observation of LLI program implementation and instructional strategies (n = 160 observations). Table 30 illustrates the frequencies for each item on the LLIOT, as observed during the visits. The results from the LLIOT revealed that 7 of the 10 LLI lesson components were rated “Acceptable” or “Excellent” over 90% of the time, indicating a high level of program implementation fidelity across both districts. The highest rated lesson components (i.e., those demonstrating the highest degree of implementation fidelity) included writing about reading, phonics/word work, and reading a new book, which were rated “Acceptable” or “Excellent” 98.8%5, 97.6%, and 95.7% of the time, respectively. The lowest rated lesson components (i.e., those demonstrating the lowest degree of implementation fidelity) included classroom and home connections, which were not observed 51.9% and 22.5% of the time, respectively. Teachers were also rated highly on their use of literacy instructional strategies, such as modeling and encouraging fluent oral reading (96.9% “Acceptable” or “Excellent”) and appropriate reading strategies (95.7%) and assisting students in problem-solving (95.6%). Further, in the majority of observed lessons, instructional materials were readily available; the lesson was well-organized; and students were engaged and attentive (100.0%, 99.4%, and 98.1% “Acceptable” or “Excellent,” respectively). Overall, observers perceived that the lesson was delivered as designed 96.3% of the time. All items can be found in Table 30, pages 38-39.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

As the LLI benchmarks were scored in terms of alphabetic levels (i.e., pre-A, A, B, C, etc.), these outcomes first had to be recoded into numeric equivalents before analysis. Additionally, because some students were unable to reach the initial benchmark Level A as measured in the LLI benchmark system, we created a new category, pre-A benchmark level, in order to assign scores to those who were below Level A so those students could be included in the study. All benchmark outcomes were assigned numeric equivalents for each grade level before a series of mixed (i.e., “one between groups”/”one within groups”) analysis of variance (ANOVA) procedures was conducted on the transformed measures to determine whether larger gains were observed for one of the two conditions overall (i.e., LLI/treatment vs. delayed-LLI/control) and for several demographic subgroups nested within the two conditions (e.g., ethnicity, special education status, English Language Learner status). Also, variations in the sample sizes across each analysis were seen due to limited cases of missing data. In the total sample, any cases with missing data could not be included in the analysis. Missing data resulted from several situations: 1) only cases with both pre-test and post-test data were able to be included in the analyses; 2) both achievement measures had “frustration” level cut-offs, which meant some students may not have had a score if they could not meet the minimum frustration level; and 3) students were allowed to voluntarily participate in the testing. Tests for normality of data and statistical assumptions (i.e., normal distribution; independence of measures) as well as measures of central tendency (i.e., means, standard deviations) were conducted on all outcomes for each grade level prior to the series of mixed ANOVAs. A series of mixed ANOVAs was also conducted on the means of the 4 DIBELS measures in kindergarten (ISF, LNF, PSF, and NWF), the 4 DIBELS measures in 1st grade (LNF, PSF, NWF, and ORF), and the 2 DIBELS measures in 2nd grade (NWF and ORF), both overall and by demographic subgroup. Additionally, analyses were conducted on treatment and control group difference scores (i.e., pre-test to post-test difference) in order to determine if any significant gain, or rate of change over time, was found for either group. From the pre- and post-test outcomes on the benchmark tests and DIBELS measures, difference scores were computed and analyzed for treatment and control group students in the aggregate, as well as by demographic subgroup.