Promoting Acceleration of Comprehension and Content Through Text (PACT)
Study: Swanson et al. (2017)

Describe how students were selected to participate in the study:

This study focused on students with reading difficulties. The participating school provided a de-identified list of students with a scale score of 1639 or below on the state accountability measure of reading (just above passing on the low-bar test) in seventh grade. All incoming eighth-grade students who met this criterion were randomly assigned to the PACT treatment group or typical practice group.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students who received a scale score of 1639 or below on the state accountability measure of reading (just above passing on the low-bar test) in seventh grade.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

PACT

Specify which condition is the control condition:

Typical practice comparison condition.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

n/a – There are not more than two treatment conditions.

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

We randomly assigned a total of 108 students to the PACT intervention or typical practice conditions. Consent to participate in the study was collected after random assignment, producing a total of 78 eighth-grade students (treatment, n=45; comparison, n=33) whose parents provided consent for data collection.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

All three teachers held bachelor’s degrees, and one also held a degree in jurisprudence. . He was experienced in delivering social studies content and the PACT intervention (he had completed three 10-day units in a previous study with high levels of implementation fidelity). He also attended 10 hours of initial PACT professional development, taught the intervention for 30 days in a previous study, and attended a 6-hour refresher professional development session. He was hired and supervised by researchers. The comparison teachers held bachelor’s degrees.

Describe when and how fidelity of treatment information was obtained.

Treatment: Instruction in the PACT classroom was audio recorded daily in all rooms with the exception of assemblies and events such as fire drills. A total of 438 class sessions were recorded. Coders were trained using the gold standard method and met 90% agreement prior to coding recordings. 20% of recordings were coded (n = 82). The selection of these recordings was based on random assignment by day of instruction (i.e. Day 1, period 5 was recorded; Day 2, period 3 was recorded; and so on). The code sheet consisted of a fidelity rating scale, which identified the level of alignment of implementation with the intended intervention. The 4-point Likert-type scale ranged from very low (1) to very high (4). Comparison: Doctoral candidate-level observers conducted in-person observations one time per month in comparison classrooms. The days observed were chosen by the comparison teacher. A total of 55 observations were made (27 for one comparison teacher and 28 for the other). Two forms were used to observe comparison classrooms. Observers used the same form used for the treatment audio recordings, and they also used a form designated for observing literacy activities in social studies content area classes. Observers were trained using the gold standard method (same as the audio recording for treatment condition).

What were the results on the fidelity-of-treatment implementation measure?

Comprehension Canopy: 22.2% high, 44.4% mid-high, 33.3% mid-low, 0% low Essential Words: 12.5% high, 70.8% mid-high, 12.5% mid-low, 4.2% low Warm Up: 66.7% high, 8.3% mid-high, 20.8% mid-low, 4.2% low Critical Reading: 35.0% high, 38.3% mid-high, 20.0% mid-low, 6.7% low Team-Based Learning Comprehension Check: 23.5% high, 70.6% mid-high, 5.9% mid-low, 0% low Team-Based Learning Knowledge Application: 60% mid-high, 40% low

Was the fidelity measure also used in control classrooms?

Yes (see explanation above)

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

To estimate treatment effects, we fit a series of regression models in Mplus 7.2. Treatment status was dummy coded (1 . treatment). Pretest scores were grand mean centered (Enders & Tofighi, 2007 ) and included as a covariate. ESs were estimated as the ratio between the model-derived coefficient for treatment and the pooled within-group standard deviation across conditions at posttest. Not all participants had complete data on the variables used in the regression models. From the original sample of 78 students, 14.8%, 20%, 5%, 15%, and 4.3% of the participants had missing data for MASK knowledge acquisition, MASK reading comprehension, MASK vocabulary recall, Gates-MacGinitie reading comprehension, and STAAR social studies, respectively. We addressed missing data using multiple imputation, which uses data across the full complement of variables to estimate missing values (Collins, Schafer, & Kam, 2001 ). A total of 1,000 data sets were imputed using the routine available in Mplus 7.2.