Promoting Acceleration of Comprehension and Content Through Text (PACT)
Study: Swanson et al. (2017)

Summary

The pact intervention Evidence suggests that content approaches to reading framed within a text-processing view of comprehension that focus on content in the text as the vehicle for instruction can be effective in the general classroom (McKeown, Beck, & Blake, 2009; Vaughn, Swanson, et al., 2013; Vaughn et al., 2015) and among struggling readers included in general education social studies classes (Swanson, Wanzek, Vaughn, et al., 2015; Wanzek et al., 2015). Reading theorists and educators generally agree that proficient readers interact with the text by building a coherent representation, which involves extracting information from the text base and integrating information from multiple sources, including the text or background knowledge (Kintsch, 1998; van den Broek, 2010). This type of approach focuses on reading text to gather new information and then integrating these new ideas with previous learning (Kintsch, 1998) through discussion (Applebee, Langer, Nystrand, & Gamoran, 2003), summarization, and questioning (I. L. Beck & McKeown, 2006). To this end, we designed an intervention called PACT that contains five components informed by the content learning model (Gersten, Baker, Smith-Johnson, Dimino, & Peterson, 2006) to improve understanding while reading text and provide opportunities for students to connect new learning to previous learning. The PACT intervention is a set of instructional practices implemented daily within the social studies. Evidence suggests that when these components are presented as a unified instructional approach, students gain content knowledge and content reading comprehension (Vaughn, Swanson, et al., 2013; Vaughn et al., 2015; Vaughn et al., in press). For an overview of how these components are taught over the 10-day unit. Comprehension canopy The comprehension canopy is designed to build background knowledge and motivation. On the first day, students watch a short high-interest video. Teachers provide a purpose for viewing (e.g., “As you watch the video, write two reasons why the colonists called the First Continental Congress”) and a discussion afterward. An overarching question is introduced to unify the theme (e.g., Was the American Revolution inevitable? Why?). Each subsequent lesson begins with a review of the question and discussion of how new information informs an answer to the overarching question. At the end of the 10-day unit, students answer the comprehension canopy question during a whole-class or small-group discussion. Essential words A set of 4–5 high-utility, high-frequency concepts (e.g., revenue, petition) are taught and reviewed over the span of each 10-day unit. On Day 1, the teacher introduces each essential word using a student-friendly definition, visual representation, related words, sentences with the word in context, and question prompts for brief discussion of the word. Essential words are reinforced during warm-ups and are integrated into texts, comprehension checks, and knowledge application activities (all described below). Warm-up Throughout the unit, lessons begin with a 5-min review of an essential word using an activity that requires students to apply the meaning of the word. For example, for the word revenue, students are shown a graph listing revenues and spending for the U.S. government and asked to describe the relationship between revenue and spending over the years. Critical reading During critical reading sessions, students read and discuss information from primary and secondary sources of text related to the content in whole-class, small-group, paired, or individual reading arrangements. These grouping decisions are based on student need. Teachers engage students in classroom discourse and note taking that are designed to help students make connections to the comprehension canopy, essential words, and previously learned material. In addition, essential words are reviewed when they are used in text. Team-based learning (TBL) We adapted an instructional feature previously implemented at the postsecondary level with medical, pharmaceutical, and business students called TBL (Michaelsen & Sweet, 2011 ). TBL provides students with opportunities to engage in text-based discussions and to provide text-based evidence to support ideas. TBL comprehension checks The purposes of TBL comprehension checks are to ensure understanding of content and to provide teachers with data to guide subsequent instruction. First students complete a brief quiz individually with no access to text or notes; this quiz is turned in to the teacher for a grade. This provides an opportunity for individual accountability for content. Next students complete the check again with their heterogeneous team members. For each question, the team must agree on the answer and provide evidence from the text to support the team’s decision. Scratch-off answer sheets are provided to teams for immediate feedback on accuracy. If the team scratches the correct answer a star is revealed. If the answer is incorrect, there is no star and the team returns to discuss the question using text sources to select an alternative answer. Scratch-off cards are available at http://www.epsteineducation.com/home/about/ and cost less than 20 cents each. TBL knowledge application At the end of the unit, students work in their heterogeneous teams to complete a TBL knowledge application activity designed to apply and extend understanding of content learned from text and prior discourse by articulating new perspectives, solving problems, and presenting conclusions. For example, students may be given a set of cards with causes of the Revolutionary War. They must compare two at a time and decide which is a more compelling cause. They continue this process until they end up with the most compelling cause of the Revolutionary War. Students are then asked to write a paragraph explaining their choice.

Target Grades:
8
Target Populations:
  • Any student at risk for academic failure
Area(s) of Focus:
  • Comprehension
  • Vocabulary
Where to Obtain:
The Meadows Center for Preventing Educational Risk at the University of Texas at Austin
1912 Speedway D4900; Austin, TX 78712
512-232-2320
http://www.meadowscenter.org/projects/detail/promoting-adolescents-comprehension-of-text-pact
Initial Cost:
Free
Replacement Cost:
Free

Materials are provided at no cost

Staff Qualified to Administer Include:
  • Special Education Teacher
  • General Education Teacher
  • Reading Specialist
  • Math Specialist
  • EL Specialist
  • Interventionist
  • Student Support Services Personnel (e.g., counselor, social worker, school psychologist, etc.)
Training Requirements:
Training not required

N/A


PACT efficacy trials The efficacy of PACT was investigated on three occasions in which eighth-grade social studies classes were randomly assigned to the PACT intervention condition or the typical practice comparison condition. Students received three 10-day units of the PACT intervention. Across all three studies, students in the treatment condition outperformed students in the typical practice comparison condition at statistically significant levels on a measure of content knowledge (Vaughn, Swanson, et al., 2013; Vaughn et al., 2015; Vaughn et al., in press). Across two trials (one with English language learners), students in the PACT condition outperformed students in the typical practice condition ona measure of content reading comprehension (Vaughn, Swanson, et al., 2013; Vaughn et al., in press).Broad reading comprehension was impacted in only the first study (Vaughn, Swanson, et al., 2013).In a study of quasi-experimental design, researchers investigated the impact of PACT on the performance of students with disabilities included in the general education classrooms in which the prior studies had taken place (Swanson, Wanzek, Vaughn, et al., 2015). Students with disabilities who received the PACT intervention outperformed students with disabilities in the typical practice comparison conditions on measures of content knowledge acquisition (effect size [ES] .0.26) and content reading comprehension (ES .0.34) but not standardized reading comprehension. Findings from these efficacy trials are quite promising. However, only three 10-day units of the PACT intervention were delivered, and with varying levels of implementation fidelity across classrooms. This provides some indication that the effects of PACT may be more robust for students with reading difficulties when implemented for a longer period of time and at higher levels of implementation fidelity.

Access to Technical Support:
Practitioners may contact the Meadows Center for Preventing Educational Risk at the University of Texas at Austin for ongoing professional and technical support.
Recommended Administration Formats Include:
  • Small group of students
Minimum Number of Minutes Per Session:
45
Minimum Number of Sessions Per Week:
5
Minimum Number of Weeks:
36
Detailed Implementation Manual or Instructions Available:
Yes
Is Technology Required?
No technology is required.

Program Information

Descriptive Information

Please provide a description of program, including intended use:

The pact intervention Evidence suggests that content approaches to reading framed within a text-processing view of comprehension that focus on content in the text as the vehicle for instruction can be effective in the general classroom (McKeown, Beck, & Blake, 2009; Vaughn, Swanson, et al., 2013; Vaughn et al., 2015) and among struggling readers included in general education social studies classes (Swanson, Wanzek, Vaughn, et al., 2015; Wanzek et al., 2015). Reading theorists and educators generally agree that proficient readers interact with the text by building a coherent representation, which involves extracting information from the text base and integrating information from multiple sources, including the text or background knowledge (Kintsch, 1998; van den Broek, 2010). This type of approach focuses on reading text to gather new information and then integrating these new ideas with previous learning (Kintsch, 1998) through discussion (Applebee, Langer, Nystrand, & Gamoran, 2003), summarization, and questioning (I. L. Beck & McKeown, 2006). To this end, we designed an intervention called PACT that contains five components informed by the content learning model (Gersten, Baker, Smith-Johnson, Dimino, & Peterson, 2006) to improve understanding while reading text and provide opportunities for students to connect new learning to previous learning. The PACT intervention is a set of instructional practices implemented daily within the social studies. Evidence suggests that when these components are presented as a unified instructional approach, students gain content knowledge and content reading comprehension (Vaughn, Swanson, et al., 2013; Vaughn et al., 2015; Vaughn et al., in press). For an overview of how these components are taught over the 10-day unit. Comprehension canopy The comprehension canopy is designed to build background knowledge and motivation. On the first day, students watch a short high-interest video. Teachers provide a purpose for viewing (e.g., “As you watch the video, write two reasons why the colonists called the First Continental Congress”) and a discussion afterward. An overarching question is introduced to unify the theme (e.g., Was the American Revolution inevitable? Why?). Each subsequent lesson begins with a review of the question and discussion of how new information informs an answer to the overarching question. At the end of the 10-day unit, students answer the comprehension canopy question during a whole-class or small-group discussion. Essential words A set of 4–5 high-utility, high-frequency concepts (e.g., revenue, petition) are taught and reviewed over the span of each 10-day unit. On Day 1, the teacher introduces each essential word using a student-friendly definition, visual representation, related words, sentences with the word in context, and question prompts for brief discussion of the word. Essential words are reinforced during warm-ups and are integrated into texts, comprehension checks, and knowledge application activities (all described below). Warm-up Throughout the unit, lessons begin with a 5-min review of an essential word using an activity that requires students to apply the meaning of the word. For example, for the word revenue, students are shown a graph listing revenues and spending for the U.S. government and asked to describe the relationship between revenue and spending over the years. Critical reading During critical reading sessions, students read and discuss information from primary and secondary sources of text related to the content in whole-class, small-group, paired, or individual reading arrangements. These grouping decisions are based on student need. Teachers engage students in classroom discourse and note taking that are designed to help students make connections to the comprehension canopy, essential words, and previously learned material. In addition, essential words are reviewed when they are used in text. Team-based learning (TBL) We adapted an instructional feature previously implemented at the postsecondary level with medical, pharmaceutical, and business students called TBL (Michaelsen & Sweet, 2011 ). TBL provides students with opportunities to engage in text-based discussions and to provide text-based evidence to support ideas. TBL comprehension checks The purposes of TBL comprehension checks are to ensure understanding of content and to provide teachers with data to guide subsequent instruction. First students complete a brief quiz individually with no access to text or notes; this quiz is turned in to the teacher for a grade. This provides an opportunity for individual accountability for content. Next students complete the check again with their heterogeneous team members. For each question, the team must agree on the answer and provide evidence from the text to support the team’s decision. Scratch-off answer sheets are provided to teams for immediate feedback on accuracy. If the team scratches the correct answer a star is revealed. If the answer is incorrect, there is no star and the team returns to discuss the question using text sources to select an alternative answer. Scratch-off cards are available at http://www.epsteineducation.com/home/about/ and cost less than 20 cents each. TBL knowledge application At the end of the unit, students work in their heterogeneous teams to complete a TBL knowledge application activity designed to apply and extend understanding of content learned from text and prior discourse by articulating new perspectives, solving problems, and presenting conclusions. For example, students may be given a set of cards with causes of the Revolutionary War. They must compare two at a time and decide which is a more compelling cause. They continue this process until they end up with the most compelling cause of the Revolutionary War. Students are then asked to write a paragraph explaining their choice.

The program is intended for use in the following age(s) and/or grade(s).

not selected Age 0-3
not selected Age 3-5
not selected Kindergarten
not selected First grade
not selected Second grade
not selected Third grade
not selected Fourth grade
not selected Fifth grade
not selected Sixth grade
not selected Seventh grade
selected Eighth grade
not selected Ninth grade
not selected Tenth grade
not selected Eleventh grade
not selected Twelth grade


The program is intended for use with the following groups.

not selected Students with disabilities only
not selected Students with learning disabilities
not selected Students with intellectual disabilities
not selected Students with emotional or behavioral disabilities
not selected English language learners
selected Any student at risk for academic failure
not selected Any student at risk for emotional and/or behavioral difficulties
not selected Other
If other, please describe:

ACADEMIC INTERVENTION: Please indicate the academic area of focus.

Early Literacy

not selected Print knowledge/awareness
not selected Alphabet knowledge
not selected Phonological awareness
not selected Phonological awarenessEarly writing
not selected Early decoding abilities
not selected Other

If other, please describe:

Language

not selected Expressive and receptive vocabulary
not selected Grammar
not selected Syntax
not selected Listening comprehension
not selected Other
If other, please describe:

Reading

not selected Phonological awareness
not selected Phonics/word study
selected Comprehension
not selected Fluency
selected Vocabulary
not selected Spelling
not selected Other
If other, please describe:

Mathematics

not selected Computation
not selected Concepts and/or word problems
not selected Whole number arithmetic
not selected Comprehensive: Includes computation/procedures, problem solving, and mathematical concepts
not selected Algebra
not selected Fractions, decimals (rational number)
not selected Geometry and measurement
not selected Other
If other, please describe:

Writing

not selected Handwriting
not selected Spelling
not selected Sentence construction
not selected Planning and revising
not selected Other
If other, please describe:

BEHAVIORAL INTERVENTION: Please indicate the behavior area of focus.

Externalizing Behavior

not selected Physical Aggression
not selected Verbal Threats
not selected Property Destruction
not selected Noncompliance
not selected High Levels of Disengagement
not selected Disruptive Behavior
not selected Social Behavior (e.g., Peer interactions, Adult interactions)
not selected Other
If other, please describe:

Internalizing Behavior

not selected Depression
not selected Anxiety
not selected Social Difficulties (e.g., withdrawal)
not selected School Phobia
not selected Other
If other, please describe:

Acquisition and cost information

Where to obtain:

Address
1912 Speedway D4900; Austin, TX 78712
Phone Number
512-232-2320
Website
http://www.meadowscenter.org/projects/detail/promoting-adolescents-comprehension-of-text-pact

Initial cost for implementing program:

Cost
$0.00
Unit of cost

Replacement cost per unit for subsequent use:

Cost
$0.00
Unit of cost
Duration of license

Additional cost information:

Describe basic pricing plan and structure of the program. Also, provide information on what is included in the published program, as well as what is not included but required for implementation (e.g., computer and/or internet access)

Materials are provided at no cost

Program Specifications

Setting for which the program is designed.

not selected Individual students
selected Small group of students
not selected BI ONLY: A classroom of students

If group-delivered, how many students compose a small group?

   8-10

Program administration time

Minimum number of minutes per session
45
Minimum number of sessions per week
5
Minimum number of weeks
36
not selected N/A (implemented until effective)

If intervention program is intended to occur over less frequently than 60 minutes a week for approximately 8 weeks, justify the level of intensity:

Does the program include highly specified teacher manuals or step by step instructions for implementation?
Yes

BEHAVIORAL INTERVENTION: Is the program affiliated with a broad school- or class-wide management program?

If yes, please identify and describe the broader school- or class-wide management program:

Does the program require technology?
No

If yes, what technology is required to implement your program?
not selected Computer or tablet
not selected Internet connection
not selected Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

Training

How many people are needed to implement the program ?

Is training for the instructor or interventionist required?
No
If yes, is the necessary training free or at-cost?

Describe the time required for instructor or interventionist training:
Training not required

Describe the format and content of the instructor or interventionist training:
N/A

What types or professionals are qualified to administer your program?

selected Special Education Teacher
selected General Education Teacher
selected Reading Specialist
selected Math Specialist
selected EL Specialist
selected Interventionist
selected Student Support Services Personnel (e.g., counselor, social worker, school psychologist, etc.)
not selected Applied Behavior Analysis (ABA) Therapist or Board Certified Behavior Analyst (BCBA)
not selected Paraprofessional
not selected Other

If other, please describe:

Does the program assume that the instructor or interventionist has expertise in a given area?
No   

If yes, please describe: 


Are training manuals and materials available?
Yes

Describe how the training manuals or materials were field-tested with the target population of instructors or interventionist and students:
PACT efficacy trials The efficacy of PACT was investigated on three occasions in which eighth-grade social studies classes were randomly assigned to the PACT intervention condition or the typical practice comparison condition. Students received three 10-day units of the PACT intervention. Across all three studies, students in the treatment condition outperformed students in the typical practice comparison condition at statistically significant levels on a measure of content knowledge (Vaughn, Swanson, et al., 2013; Vaughn et al., 2015; Vaughn et al., in press). Across two trials (one with English language learners), students in the PACT condition outperformed students in the typical practice condition ona measure of content reading comprehension (Vaughn, Swanson, et al., 2013; Vaughn et al., in press).Broad reading comprehension was impacted in only the first study (Vaughn, Swanson, et al., 2013).In a study of quasi-experimental design, researchers investigated the impact of PACT on the performance of students with disabilities included in the general education classrooms in which the prior studies had taken place (Swanson, Wanzek, Vaughn, et al., 2015). Students with disabilities who received the PACT intervention outperformed students with disabilities in the typical practice comparison conditions on measures of content knowledge acquisition (effect size [ES] .0.26) and content reading comprehension (ES .0.34) but not standardized reading comprehension. Findings from these efficacy trials are quite promising. However, only three 10-day units of the PACT intervention were delivered, and with varying levels of implementation fidelity across classrooms. This provides some indication that the effects of PACT may be more robust for students with reading difficulties when implemented for a longer period of time and at higher levels of implementation fidelity.

Do you provide fidelity of implementation guidance such as a checklist for implementation in your manual?
No

Can practitioners obtain ongoing professional and technical support?
Yes

If yes, please specify where/how practitioners can obtain support:

Practitioners may contact the Meadows Center for Preventing Educational Risk at the University of Texas at Austin for ongoing professional and technical support.

Summary of Evidence Base

Please identify, to the best of your knowledge, all the research studies that have been conducted to date supporting the efficacy of your program, including studies currently or previously submitted to NCII for review. Please provide citations only (in APA format); do not include any descriptive information on these studies. NCII staff will also conduct a search to confirm that the list you provide is accurate.

Vaughn, S., Martinez, L. R., Wanzek, J., Roberts, G., Swanson, E., & Fall, A.-M. (2016). Improving content knowledge and comprehension for English language learners: Findings from a randomized control trial. Journal of Educational Psychology. Advance online publication.

 

Swanson, E., Wanzek, J., Vaughn, S., Fall, A-M., Roberts, G., Hall, C. & Miller, V. (2017). Middle school reading comprehension and content learning intervention for below average readers. Reading and Writing Quarterly: Overcoming Learning Difficulties. 33, 37-53.

 

Wanzek, J., Swanson, E., Vaughn, S., Roberts, G., & Fall, A.-M. (2016). English learner and non-English learner students with disabilities: Content acquisition and comprehension. Exceptional Children, 82(4), 428-442doi:10.1177/0014402915619419.

 

Wanzek, J., Swanson, E. A., Roberts, G., Vaughn, S., & Kent, S. C. (2015). Promoting acceleration of comprehension and content through text in high school social studies classes. Journal of Research on Educational Effectiveness8(2), 169-188.

 

Wanzek, J., Kent, S. C., Vaughn, S., Swanson, E., Roberts, G., & Haynes, M. (2015). Implementing team-based learning in middle school studies classes. Journal of Educational Research. Advance Online Publication. doi: 10.1080/00220671.2014.893224

 

Kent, S., Wanzek, J., Swanson, E.A., & Vaughn, S. (2015). Team-based learning for students with high-incidence disabilities in high school social studies classrooms. Learning Disabilities Research and Practice, 30, 3-14.

 

Swanson, E., Wanzek, J., Vaughn, S., Roberts, G, & Fall, A. (2015). Improving reading comprehension and social studies knowledge among middle school students with disabilities. Exceptional Children. Advance online publication. 10.1177/0014402914563704

 

Vaughn, S., Roberts, G., Swanson, E. A., Wanzek, J., Fall, A.-M., & Stillman-Spisak, S. J. (2014). Improving middle school students’ knowledge and comprehension in social studies: A replication. Educational Psychology Review. Advanced Online Publication. doi: 10.1007/s10648-014-9274-2

 

Wanzek, J., Vaughn, S., Kent, S. C., Swanson, E. A., Roberts, G., Haynes, M., Fall, A. M., Stillman-Spisak, S., & Solis, M. (2014). The effects of team-based learning on social studies knowledge acquisition in high school. Journal of Research on Educational Effectiveness, 7, 183-204.

 

Vaughn, S., Swanson, E. A., Roberts, G., Wanzek, J., Stillman-Spisak, S. J., Solis, M., & Simmons, D. (2013). Improving reading comprehension and social studies knowledge in middle school, Reading Research Quarterly, 48(1) 77-93. doi: 10.1002/rrq.039

 

Study Information

Study Citations

Swanson, E., Wanzek, J., Vaughn, S., Fall, A. M., Roberts, G., Hall, C. & Miller, V. (2017). Middle School Reading Comprehension and Content Learning Intervention for Below-Average Readers. Reading & Writing Quarterly, 22(1) 37-53.

Participants Empty Bobble

Describe how students were selected to participate in the study:
This study focused on students with reading difficulties. The participating school provided a de-identified list of students with a scale score of 1639 or below on the state accountability measure of reading (just above passing on the low-bar test) in seventh grade. All incoming eighth-grade students who met this criterion were randomly assigned to the PACT treatment group or typical practice group.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):
Students who received a scale score of 1639 or below on the state accountability measure of reading (just above passing on the low-bar test) in seventh grade.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:
  • below the 30th percentile on local or national norm, or
  • identified disability related to the focus of the intervention?
%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:
  • emotional disability label,
  • placed in an alternative school/classroom,
  • non-responsive to Tiers 1 and 2, or
  • designation of severe problem behaviors on a validated scale or through observation?
%

Specify which condition is the submitted intervention:
PACT

Specify which condition is the control condition:
Typical practice comparison condition.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):
n/a – There are not more than two treatment conditions.

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Age less than 1
Age 1
Age 2
Age 3
Age 4
Age 5
Kindergarten
Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8 45 33 0.00
Grade 9
Grade 10
Grade 11
Grade 12

Race–Ethnicity

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
African American 4 4 0.19
American Indian
Asian/Pacific Islander 0 1 2.08
Hispanic 36 22 0.41
White 4 5 0.35
Other

Socioeconomic Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Subsidized Lunch 33 26 0.20
No Subsidized Lunch 12 13 0.33

Disability Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Speech-Language Impairments
Learning Disabilities
Behavior Disorders
Emotional Disturbance
Intellectual Disabilities
Other 0 9 3.58
Not Identified With a Disability 45 33 0.00

ELL Status

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
English Language Learner 0 8 3.49
Not English Language Learner 45 25 3.49

Gender

Demographic Program
Number
Control
Number
Effect Size: Cox Index
for Binary Differences
Female 21 18 0.19
Male 23 14 0.22

Mean Effect Size

1.12

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences between groups in the descriptions below, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not demographic characteristics, please describe the results of those analyses here.

There were significant differences across treatment conditions in the number of English learners (EL) and the number of students with special needs who provided pretest and posttest data: EL students and students with special needs were entirely unrepresented in the treatment data, although they accounted for about a quarter of the available data in the comparison sample. (pg. 46) To evaluate the situation, we treated nonconsented students in both groups as instances of pretreatment (and postrandomization) attrition and evaluated attrition-related bias using seventh-grade STAAR scores. Using two-way analysis of variance, we compared STAAR scores across treatment groups, across consent status, and across the four groups represented by the interaction of treatment and consent status. We found no significant differences for condition (p = .08), for consent status (p =.70), or for the interaction of condition and consent (p = .3). In other words, there were no differences between the treatment groups at pretest on the STAAR, no differences between students who declined to complete the pretest battery and those who agreed, and no condition-related differences in STAAR scores for consented and nonconsented students. This pattern suggests that attrition in the groups may not have biased the sample at pretest. (pg. 46 and 47)

Design Half Bobble

What method was used to determine students' placement in treatment/control groups?
Random
Please describe the assignment method or the process for defining treatment/comparison groups.
We randomly assigned a total of 108 students to the PACT intervention or typical practice conditions. Consent to participate in the study was collected after random assignment, producing a total of 78 eighth-grade students (treatment, n=45; comparison, n=33) whose parents provided consent for data collection.

What was the unit of assignment?
Students
If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?
not selected Schools
not selected Teachers
selected Students
not selected Classes
not selected Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

Fidelity of Implementation Full Bobble

How was the program delivered?
not selected Individually
selected Small Group
not selected Classroom

If small group, answer the following:

Average group size
9
Minimum group size
8
Maximum group size
10

What was the duration of the intervention (If duration differed across participants, settings, or behaviors, describe for each.)?

Weeks
36.00
Sessions per week
5.00
Duration of sessions in minutes
45.00
What were the background, experience, training, and ongoing support of the instructors or interventionists?
All three teachers held bachelor’s degrees, and one also held a degree in jurisprudence. . He was experienced in delivering social studies content and the PACT intervention (he had completed three 10-day units in a previous study with high levels of implementation fidelity). He also attended 10 hours of initial PACT professional development, taught the intervention for 30 days in a previous study, and attended a 6-hour refresher professional development session. He was hired and supervised by researchers. The comparison teachers held bachelor’s degrees.

Describe when and how fidelity of treatment information was obtained.
Treatment: Instruction in the PACT classroom was audio recorded daily in all rooms with the exception of assemblies and events such as fire drills. A total of 438 class sessions were recorded. Coders were trained using the gold standard method and met 90% agreement prior to coding recordings. 20% of recordings were coded (n = 82). The selection of these recordings was based on random assignment by day of instruction (i.e. Day 1, period 5 was recorded; Day 2, period 3 was recorded; and so on). The code sheet consisted of a fidelity rating scale, which identified the level of alignment of implementation with the intended intervention. The 4-point Likert-type scale ranged from very low (1) to very high (4). Comparison: Doctoral candidate-level observers conducted in-person observations one time per month in comparison classrooms. The days observed were chosen by the comparison teacher. A total of 55 observations were made (27 for one comparison teacher and 28 for the other). Two forms were used to observe comparison classrooms. Observers used the same form used for the treatment audio recordings, and they also used a form designated for observing literacy activities in social studies content area classes. Observers were trained using the gold standard method (same as the audio recording for treatment condition).

What were the results on the fidelity-of-treatment implementation measure?
Comprehension Canopy: 22.2% high, 44.4% mid-high, 33.3% mid-low, 0% low Essential Words: 12.5% high, 70.8% mid-high, 12.5% mid-low, 4.2% low Warm Up: 66.7% high, 8.3% mid-high, 20.8% mid-low, 4.2% low Critical Reading: 35.0% high, 38.3% mid-high, 20.0% mid-low, 6.7% low Team-Based Learning Comprehension Check: 23.5% high, 70.6% mid-high, 5.9% mid-low, 0% low Team-Based Learning Knowledge Application: 60% mid-high, 40% low

Was the fidelity measure also used in control classrooms?
Yes (see explanation above)

Measures and Results

Measures Targeted : Half Bobble
Measures Broader : Full Bobble

Study measures are classified as targeted, broader, or administrative data according to the following definitions:

  • Targeted measures
    Assess outcomes, such as competencies or skills that the program was directly targeted to improve.
    • In the academic domain, targeted measures typically are not the very items taught but rather novel items structured similarly to the content addressed in the program. For example, if a program taught word-attack skills, a targeted measure would be decoding of pseudo words. If a program taught comprehension of cause-effect passages, a targeted measure would be answering questions about cause-effect passages structured similarly to those used during intervention, but not including the very passages used for intervention.
    • In the behavioral domain, targeted measures evaluate aspects of external or internal behavior the program was directly targeted to improve and are operationally defined.
  • Broader measures
    Assess outcomes that are related to the competencies or skills targeted by the program but not directly taught in the program.
    • In the academic domain, if a program taught word-level reading skill, a broader measure would be answering questions about passages the student reads. If a program taught calculation skill, a broader measure would be solving word problems that require the same kinds of calculation skill taught in the program.
    • In the behavioral domain, if a program taught a specific skill like on-task behavior in one classroom, a broader measure would be academic performance in that setting or on-task behavior in another setting.
  • Administrative data measures apply only to behavioral intervention tools and are measures such as office discipline referrals (ODRs) and graduation rates which do not have psychometric properties as do other, more traditional targeted or broader measures.

Click here for more information on effect size.


What populations are you submitting outcome data for?
not selected Full sample
not selected Students at or below the 20th percentile
not selected English language learners
not selected Racial/ethnic subgroups
not selected Economically disadvantaged students (low socioeconomic status)
Targeted Measure Reverse Coded? Reliability Relevance Exposure
Broader Measure Reverse Coded? Reliability Relevance Exposure
Administrative Data Measure Reverse Coded? Relevance

Posttest Data

Targeted Measures (Full Sample)

Measure Sample Type Effect Size P

Broader Measures (Full Sample)

Measure Sample Type Effect Size P

Administrative Measures (Full Sample)

Measure Sample Type Effect Size P

Targeted Measures (Subgroups)

Measure Sample Type Effect Size P

Broader Measures (Subgroups)

Measure Sample Type Effect Size P

Administrative Measures (Subgroups)

Measure Sample Type Effect Size P
For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.
Please explain any missing data or instances of measures with incomplete pre- or post-test data.
If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:
Describe the analyses used to determine whether the intervention produced changes in student outcomes:
To estimate treatment effects, we fit a series of regression models in Mplus 7.2. Treatment status was dummy coded (1 . treatment). Pretest scores were grand mean centered (Enders & Tofighi, 2007 ) and included as a covariate. ESs were estimated as the ratio between the model-derived coefficient for treatment and the pooled within-group standard deviation across conditions at posttest. Not all participants had complete data on the variables used in the regression models. From the original sample of 78 students, 14.8%, 20%, 5%, 15%, and 4.3% of the participants had missing data for MASK knowledge acquisition, MASK reading comprehension, MASK vocabulary recall, Gates-MacGinitie reading comprehension, and STAAR social studies, respectively. We addressed missing data using multiple imputation, which uses data across the full complement of variables to estimate missing values (Collins, Schafer, & Kam, 2001 ). A total of 1,000 data sets were imputed using the routine available in Mplus 7.2.

Additional Research

Is the program reviewed by WWC or E-ESSA?
WWC & E-ESSA
Summary of WWC / E-ESSA Findings :

What Works Clearinghouse Review

WWC only reviewed the report “Middle School Reading Comprehension and Content Learning Intervention for Below-Average Readers.” The findings from this review do not reflect the full body of research evidence on Promoting Acceleration of Comprehension and Content Through Text (PACT).

 

WWC Rating: Does not meet WWC standards because the measures of effectiveness cannot be attributed solely to the intervention.

 

Full Report  

 

Evidence for ESSA

None of the studies considered met Evidence for ESSA's inclusion requirements.

How many additional research studies are potentially eligible for NCII review?
8
Citations for Additional Research Studies :

Kent, S., Wanzek, J., Swanson, E.A., & Vaughn, S. (2015). Team-based learning for students with high-incidence disabilities in high school social studies classrooms. Learning Disabilities Research and Practice, 30, 3-14.

Swanson, E., Wanzek, J., Vaughn, S., Roberts, G, & Fall, A. (2015). Improving reading comprehension and social studies knowledge among middle school students with disabilities. Exceptional Children. Advance online publication. 10.1177/0014402914563704

Vaughn, S., Martinez, L. R., Wanzek, J., Roberts, G., Swanson, E., & Fall, A.-M. (2016). Improving content knowledge and comprehension for English language learners: Findings from a randomized control trial. Journal of Educational Psychology. Advance online publication.

Additional Source: Wanzek, J., Swanson, E., Vaughn, S., Roberts, G., & Fall, A.-M. (2016). English learner and non-English learner students with disabilities: Content acquisition and comprehension. Exceptional Children, 82(4), 428-442doi:10.1177/0014402915619419.

Vaughn, S., Roberts, G., Swanson, E. A., Wanzek, J., Fall, A.-M., & Stillman-Spisak, S. J. (2014). Improving middle school students’ knowledge and comprehension in social studies: A replication. Educational Psychology Review. Advanced Online Publication. doi: 10.1007/s10648-014-9274-2

Vaughn, S., Swanson, E. A., Roberts, G., Wanzek, J., Stillman-Spisak, S. J., Solis, M., & Simmons, D. (2013). Improving reading comprehension and social studies knowledge in middle school, Reading Research Quarterly, 48(1) 77-93. doi: 10.1002/rrq.039

Wanzek, J., Kent, S. C., Vaughn, S., Swanson, E., Roberts, G., & Haynes, M. (2015). Implementing team-based learning in middle school studies classes. Journal of Educational Research. Advance Online Publication. doi: 10.1080/00220671.2014.893224

Wanzek, J., Swanson, E. A., Roberts, G., Vaughn, S., & Kent, S. C. (2015). Promoting acceleration of comprehension and content through text in high school social studies classes. Journal of Research on Educational Effectiveness8(2), 169-188.

Wanzek, J., Vaughn, S., Kent, S. C., Swanson, E. A., Roberts, G., Haynes, M., Fall, A. M., Stillman-Spisak, S., & Solis, M. (2014). The effects of team-based learning on social studies knowledge acquisition in high school. Journal of Research on Educational Effectiveness, 7, 183-204.

Disclaimer

Most tools and programs evaluated by the NCII are branded products which have been submitted by the companies, organizations, or individuals that disseminate these products. These entities supply the textual information shown above, but not the ratings accompanying the text. NCII administrators and members of our Technical Review Committees have reviewed the content on this page, but NCII cannot guarantee that this information is free from error or reflective of recent changes to the product. Tools and programs have the opportunity to be updated annually or upon request.