Burst:Reading
Study: Dubal et al. (2012)

Describe how students were selected to participate in the study:

Participants in this study were students from the 1,023 schools that had purchased and were using the Burst:Reading (formerly Burst:Early Literacy Intervention (Burst:ELI) in the 2010-2011 school year.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Within Burst:Reading, students are identified as being at risk of academic failure according to their performance on DIBELS at the beginning and middle of the school year (i.e., at the beginning of each semester). The Burst:ELI algorithm determines individual students’ intervention priority and creates intervention groups in the following manner: 1. Student assessment results are processed, yielding: a. A gross skill rating based on DIBELS Benchmark Status or risk category (i.e., Red, Yellow, and Green). The Burst Reading Assessment supplemental measures have comparable performance levels. b. A fine-grained skill rating that differentiates intervention priority within each risk category (e.g., students who are Red on DORF are further differentiated and prioritized for intervention based on DORF subscores). c. Up to two Zone of Proximal Development (ZPD) skills. A student’s highest priority ZPD skill is the skill that comes earliest in the set instructional sequence that the student tests poorly on, evidencing a need for intervention instruction. 2. Using the results from step 1, students are prioritized for intervention based on their performance relative to the rest of the students in their class or grade. 3. The teacher or other educator then selects the number of intervention groups that the Burst algorithm should create. The algorithm generates a number of possible groups and comes to a final decision on grouping based on a social welfare function. The social welfare function selects groups of students for whom the level of utility of the instruction to be delivered is the most similar among all students in the group, yielding homogenous groups. The information below demonstrates the pre-test means for treatment and control groups in the study on each measure by semester. Additionally, the score ranges associated with the At Risk and Some Risk performance levels for each DIBELS measure are provided. Lastly, scores for each measures associated with the 25th percentile for each measure are presented based on a national norming study conducted by Cummings et al. (2011). While a number of the pre-test means may be above the At Risk range, all score means are clearly below the 25th percentiles associated with national norms. K Spring: measure= PSF; treatment group=8.33; control group=8.38; at risk=0-6; some risk=7-17; score at 25th percentile=12 1 Fall: measure= NWF; treatment group=9.94; control group=10.00; at risk=0-12; some risk=13-23; score at 25th percentile=20 1 Spring: measure= NWF; treatment group=31.57; control group=32.30; at risk=0-29; some risk=30-49; score at 25th percentile=41 2 Fall: measure= ORF; treatment group=19.33; control group=19.34; at risk=0-25; some risk=26-43; score at 25th percentile=31 2 Spring: measure= ORF; treatment group=33.95; control group=34.57; at risk=0-51; some risk=52-67; score at 25th percentile=60 3 Fall: measure= ORF; treatment group=39.38; control group=41.07; at risk=0-52; some risk=53-76; score at 25th percentile=59 3 Spring: measure= ORF; treatment group=52.40; control group=53.85; at risk=0-66; some risk=67-91; score at 25th percentile=73

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Burst:Reading (formerly known as Burst:Early Literacy Intervention)

Specify which condition is the control condition:

Business as usual (BAU).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

This study used Propensity Score Matching to identify control group students who could be considered to be equivalent to the treatment students on the basis of pretest scores and other background characteristics. Based on an iterative approach, the best-fitting PSM model was determined to be one which considered student pretest scores, ethnicity, ELL status, and free or reduced-price lunch eligibility

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Prior to implementing Burst:ELI, school personnel participated in a standardized training series that included a one-day on-site session to prepare teachers or interventionists, a follow-up webinar for teachers or interventionists after 6–10 weeks, and a half-day on-site session to prepare instructional leaders. This training followed a common “see one, do one” model in the class with students, so teachers could quickly learn, through context, how the Burst:ELI instruction should be delivered. Ongoing technical training was also provided to school and district staff to help them install, manage, and troubleshoot the software.

Describe when and how fidelity of treatment information was obtained.

Burst:Reading is a software-based intervention requiring that educators access the product website to generate student grouping, review student performance reports, and download instructional materials. Fidelity of treatment information was inferred by reviewing teacher access of this website information. Additionally, the timing of assessment data collection during intervention was compared against the expected two-week assessment intervals.

What were the results on the fidelity-of-treatment implementation measure?

Fidelity of implementation analysis was not conducted for the current study. Fidelity of implementation results were mentioned briefly in the white paper though not included. The results of the analysis from an unpublished paper are provided as follows. Due to the post-hoc nature of this study, only one component of Burst: ELI intervention fidelity could be partially examined: exposure. Exposure was operationalized according to two types of implementation data that were automatically tracked by the Burst: ELI system: · Number of instructions a Burst: ELI student received in a semester or a year; and · Timeliness with which progress monitoring assessments were delivered. The Burst: ELI system uses this data to remind teachers via email to assess students or to begin instruction when they begin to fall behind schedule. Only Burst:ELI students were included in fidelity of implementation analyses. Those students missing demographic data were included in this analysis, as student characteristics were not used. There were 6,584 kindergarten students, 6,369 first grade students, 4,996 second grade students, and 3,045 third grade students included in the fidelity analysis. See attachment "BurstReading Fidelity Analyses" for additional data.

Was the fidelity measure also used in control classrooms?

No

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Subsequent to matching treatment and control students according to Propensity Score Matching values, a differences-in-differences approach was applied to the analysis of student performance on the DIBELS measures appropriate for each grade and semester. Difference scores for each grade and semester were calculated by subtracting the post-scores from pre-scores for each group. The control group difference score was then subtracted from the treatment group’s difference score and that value submitted to a cluster-corrected t-test to account for the nesting of students within schools. In addition, Student Growth Percentiles (SGPs; Betebenner, 2009) were also calculated for each grade and semester, placing student progress into a normative context. Median treatment and control group SGPs were calculated and compared to assess differences in student growth conditional on performance at the beginning of the semester. A median SGP of 50 or greater indicates overall growth better than the student sample in the study; SGPs medians for the treatment group that exceed the control group suggest that Burst:Reading had an impact on student growth. The Mann-Whitney U test was employed to evaluate the hypothesis that the distribution of SGPs in the treatment group shifted up with respect to the distribution of SGPs in the control group.