Promoting Acceleration of Comprehension and Content Through Text (PACT)

Study: Swanson, Wanzek, Vaughn, Fall, Roberts, Hall, et al. (2017)

Swanson, E., Wanzek, J., Vaughn, S., Fall, A.M., Roberts, G., Hall, C., & Miller, V. (2017). Middle school reading comprehension and content learning intervention for below-average readers. Reading & Writing Quarterly, 22, 37–53.

Descriptive Information

Usage

Acquisition and Cost

Program Specifications and Requirements

Training

The PACT Intervention contains five components informed by the content learning model (Gersten, Baker, Smith-Johnson, Dimino, & Peterson, 2006) to improve understanding while reading text and provide opportunities for students to connect new learning to previous learning. It is a set of instructional practices implemented daily within the social studies. Evidence suggests that when these components are presented as a unified instructional approach, students gain content knowledge and content reading comprehension (Vaughn, Swanson, et al., 2013; Vaughn et al., 2015; Vaughn et al., in press).

PACT is intended for use in grade 8. The program is intended for use with any student at risk of academic failure. The academic area of focus is reading comprehension and vocabulary.

Where to Obtain: 
1912 Speedway D4900
Austin, TX 78712
Phone: 512-232-2320
 

Web Site: http://www.meadowscenter.org/projects/detail/promoting-adolescents-comprehension-of-text-pact

 

Cost: PACT materials are provided at no cost.  

It is recommended that the PACT Intervention is used in groups of 8-10 students 45 minutes per session, five days a week, all school year.

 

Training is not required for the instructor.

 

Practitioners may contact the Meadows Center for Preventing Educational Risk at the University of Texas at Austin for ongoing professional and technical support.

 

Participants: Unconvincing Evidence

Sample size: Teachers: N = 3 (treatment: n = 1; control: n = 2); students: N = 108 (treatment: n = 45; control: n = 33)

Risk Status: Students who received a scale score of 1639 or below on the state accountability measure of reading (just above passing on the low-bar test) in seventh grade.

Demographics:

 

Program

Control

Effect Size: Cox Index for Binary Differences*

Number

Percentage

Number

Percentage

Grade Level

Age 0

 

 

 

 

 

Age 1

 

 

 

 

 

Age 2

 

 

 

 

 

Age 3

 

 

 

 

 

Age 4

 

 

 

 

 

Age 5

 

 

 

 

 

Kindergarten

 

 

 

 

 

Grade 1

 

 

 

 

 

Grade 2

 

 

 

 

 

Grade 3

 

 

 

 

 

Grade 4

 

 

 

 

 

Grade 5

 

 

 

 

 

Grade 6

 

 

 

 

 

Grade 7

 

 

 

 

 

Grade 8

45

57.7

33

42.3

0.39

Grade 9

 

 

 

 

 

Grade 10

 

 

 

 

 

Grade 11

 

 

 

 

 

Grade 12

 

 

 

 

 

RaceEthnicity

African American

4

8.9

4

12.1

0.20

American Indian

NR

NR

NR

NR

 

Asian

0

0

1

3.0

 

Hispanic

36

80.0

22

66.7

0.41

White

4

8.9

5

15.2

0.35

Missing

1

2.2

1

3.0

0.25

Other

 

 

 

 

 

Socioeconomic Status

Subsidized Lunch

33

73.3

26

78.8

0.20

No Subsidized Lunch

12

26.7

13

21.2

0.20

Disability Status

Speech-Language Impairments

 

 

 

 

 

Learning Disabilities

 

 

 

 

 

Behavior Disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

Other: Any students receiving special education services

0

0

9

27.3

 

Not Identified With a Disability

45

100.0

33

72.7

 

ELL status

English Language Learner

0

0.0

8

24.2

 

Not English Language Learner

45

100.0

25

75.8

 

Gender

Female

21

46.7

18

54.5

0.19

Male

23

51.1

14

42.5

0.20

Missing

1

2.2

1

3.0

0.25

 

Training of Instructors:  All three teachers held bachelor’s degrees, and one also held a degree in jurisprudence. He was experienced in delivering social studies content and the PACT intervention (he had completed three 10-day units in a previous study with high levels of implementation fidelity). He also attended 10 hours of initial PACT professional development, taught the intervention for 30 days in a previous study, and attended a 6-hour refresher professional development session. He was hired and supervised by researchers. The comparison teachers held bachelor’s degrees.

Design: Partially Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?:  N/A

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Ye

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: N/A

Were the program and control groups demographically comparable at pretreatment?: No

Was there attrition bias1 ?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

 

 

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Treatment: Instruction in the PACT classroom was audio recorded daily in all rooms with the exception of assemblies and events such as fire drills. A total of 438 class sessions were recorded. Coders were trained using the gold standard method and met 90% agreement prior to coding recordings. 20% of recordings were coded (n = 82). The selection of these recordings was based on random assignment by day of instruction (i.e. Day 1, period 5 was recorded; Day 2, period 3 was recorded; and so on). The code sheet consisted of a fidelity rating scale, which identified the level of alignment of implementation with the intended intervention. The 4-point Likert-type scale ranged from very low (1) to very high (4).

Comparison: Doctoral candidate-level observers conducted in-person observations one time per month in comparison classrooms. The days observed were chosen by the comparison teacher. A total of 55 observations were made (27 for one comparison teacher and 28 for the other). Two forms were used to observe comparison classrooms. Observers used the same form used for the treatment audio recordings, and they also used a form designated for observing literacy activities in social studies content area classes. Observers were trained using the gold standard method (same as the audio recording for treatment condition).

 

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation:

There were 136 observed components. 104 of these were rated at mid-high to high (104/136 = 76.5%).

Comprehension Canopy: 22.2% high, 44.4% mid-high, 33.3% mid-low, 0% low

Essential Words: 12.5% high, 70.8% mid-high, 12.5% mid-low, 4.2% low

Warm Up: 66.7% high, 8.3% mid-high, 20.8% mid-low, 4.2% low

Critical Reading: 35.0% high, 38.3% mid-high, 20.0% mid-low, 6.7% low

Team-Based Learning Comprehension Check: 23.5% high, 70.6% mid-high, 5.9% mid-low, 0% low

Team-Based Learning Knowledge Application: 60% mid-high, 40% low

Measures Targeted: Partially Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

MASK Knowledge Acquisition

The MASK knowledge acquisition test was an extension of the Assessment of Social Studies Knowledge (ASK) measure used in a prior study (Vaughn, Swanson, et al., 2013). Although an extensive treatment of ASK psychometrics may be found in the original study, we note here that the alpha coefficient for the ASK content measure was 0.89 in a larger sample of students (Vaughn, Swanson, et al., 2013).

Measures content knowledge covered throughout the intervention

Measures content covered in social studies classes (content aligned with state standards)

MASK Content Reading Comprehension

The MASK reading comprehension test was an extension of the ASK measure used in a prior study (Vaughn, Swanson, et al., 2013). Although an extensive treatment of ASK psychometrics may be found in the original study, we note here that the alpha coefficient for the ASK comprehension measure was 0.85 in a larger sample of students (Vaughn, Swanson, et al., 2013).

Measures comprehension using passages related to the content taught in the intervention

Measures comprehension using passages related to the content covered in social studies classes (content aligned with state standards)

MASK Vocabulary Recall

NR

Measures vocabulary related to the content delivered during the intervention (same content as general social studies classes)

Measures vocabulary related to the content delivered in general social studies classes.

 

 

Broader  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Gates-MacGinitie Reading Comprehension Subtest (Fourth Edition)

Internal consistency ranges from 0.91 to 0.93

Alternate-forms reliability ranges from 0.80 to 0.87

Measures reading comprehension in general; allows students and opportunity to generalize the skills they learned in the program to texts consisting of different content

Measures reading comprehension in general

STAAR Eighth-Grade Social Studies

Alpha Coefficient = 0.90

Measures students’ reading abilities and content knowledge; aligned with Texas Essential Knowledge and Skills, so social studies subtest includes content similar to the content in intervention

Measures students’ reading abilities and content knowledge; social studies subtest aligned with state standards, so the content is similar to the content taught in general social studies classrooms

 

Number of Outcome Measures: 4 Reading, 1 Social Studies

Mean ES - Targeted: Data Unavailable*

Mean ES - Broader: Data Unavailable

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Reading

MASK Knowledge Acquisition

0.35

Reading

MASK Content Reading Comprehension

0.59*

Reading

MASK Vocabulary Recall

† 

Broader Measures

Construct

Measure

Effect Size

Reading

Gates-MacGinitie Reading Comprehension Subtest (Fourth Edition)

0.13

Reading

STAAR Eighth-Grade Social Studies

† 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small Group, (n=8-10)

Duration of Intervention: 45 minutes, 5 times a week, full school year

Minimum Interventionist Requirements: Professional, Training is not required

Reviewed by WWC or E-ESSA: E-ESSA

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

None of the studies considered met Evidence for ESSA's inclusion requirements.

Other Research: Potentially Eligible for NCII Review: 8 studies

Kent, S., Wanzek, J., Swanson, E.A., & Vaughn, S. (2015). Team-based learning for students with high-incidence disabilities in high school social studies classrooms. Learning Disabilities Research and Practice, 30, 3-14.

Swanson, E., Wanzek, J., Vaughn, S., Roberts, G, & Fall, A. (2015). Improving reading comprehension and social studies knowledge among middle school students with disabilities. Exceptional Children. Advance online publication. 10.1177/0014402914563704

Vaughn, S., Martinez, L. R., Wanzek, J., Roberts, G., Swanson, E., & Fall, A.-M. (2016). Improving content knowledge and comprehension for English language learners: Findings from a randomized control trial. Journal of Educational Psychology. Advance online publication.

Additional Source: Wanzek, J., Swanson, E., Vaughn, S., Roberts, G., & Fall, A.-M. (2016). English learner and non-English learner students with disabilities: Content acquisition and comprehension. Exceptional Children, 82(4), 428-442doi:10.1177/0014402915619419.

Vaughn, S., Roberts, G., Swanson, E. A., Wanzek, J., Fall, A.-M., & Stillman-Spisak, S. J. (2014). Improving middle school students’ knowledge and comprehension in social studies: A replication. Educational Psychology Review. Advanced Online Publication. doi: 10.1007/s10648-014-9274-2

Vaughn, S., Swanson, E. A., Roberts, G., Wanzek, J., Stillman-Spisak, S. J., Solis, M., & Simmons, D. (2013). Improving reading comprehension and social studies knowledge in middle school, Reading Research Quarterly, 48(1) 77-93. doi: 10.1002/rrq.039

Wanzek, J., Kent, S. C., Vaughn, S., Swanson, E., Roberts, G., & Haynes, M. (2015). Implementing team-based learning in middle school studies classes. Journal of Educational Research. Advance Online Publication. doi: 10.1080/00220671.2014.893224

Wanzek, J., Swanson, E. A., Roberts, G., Vaughn, S., & Kent, S. C. (2015). Promoting acceleration of comprehension and content through text in high school social studies classes. Journal of Research on Educational Effectiveness8(2), 169-188.

Wanzek, J., Vaughn, S., Kent, S. C., Swanson, E. A., Roberts, G., Haynes, M., Fall, A. M., Stillman-Spisak, S., & Solis, M. (2014). The effects of team-based learning on social studies knowledge acquisition in high school. Journal of Research on Educational Effectiveness, 7, 183-204.