Learning Strategies Curriculum: Word Mapping Strategy
Study: Harris, Schumaker, & Deshler (2011)

Describe how students were selected to participate in the study:

Participants were 230 public school ninth graders regularly enrolled in nine inclusive English classes, including students with and without disabilities. Three of the classes were randomly selected to receive instruction in the Word Mapping Strategy. Three classes were randomly selected to receive instruction in the Vocabulary LINCS Strategy, and three of the classes were randomly selected as the normative comparison classes. All of the students' parents signed informed consent forms.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The students with disabilities had active IEPs. Their IEPs and test scores were reviewed to determine whether the students fit the criteria for having a disability. (Their standardized achievement and IQ data were collected from school records to facilitate this verification process after consent had been obtained from parents. )

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

10.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

The three classes that received instruction in the Word Mapping Strategy (the submitted intervention) was the experimental group (79 students).

Specify which condition is the control condition:

The three classes that received "business as usual" vocabulary instruction was the comparison condition (72 students).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

The three classes that received instruction in the Vocabulary LINCing Strategy took part in the third condition (79 students).

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Nine ninth-grade English classes were randomly assigned to either the Word Mapping instruction (the experimental group) or to the business as usual comparison group or to another type of vocabulary strategy instruction.

What was the unit of assignment?

Classes

If other, please specify:

Please describe the unit of assignment:

The classes were public school ninth-grade English classes in which students with and without disabilities were enrolled and expected fully to engage in the instruction.

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

Students' scores on vocabulary assessments were used in the data analyses.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The instructor of the Word Mapping Strategy ( and the Vocabulary LINCing Strategy) was a graduate student who was completing her Ph.D. She had 10 years of teaching experience at the secondary level. Her advisors provided her support.

Describe when and how fidelity of treatment information was obtained.

The fidelity of treatment information was obtained for the Word Mapping Strategy instruction with a checklist that listed the instructional practices associated with teaching the strategy. Each item on the checklist was a teacher behavior. An observer used the checklists during class sessions in which the strategy was taught. The sessions were also videotaped for reliability purposes.

What were the results on the fidelity-of-treatment implementation measure?

The primary scorer and the reliability observer for the fidelity data were blind to the purpose of the study and to the assignment of the classes to groups. Items on the fidelity checklists were compared item-by-item. The scorers agreed on 111 out of 120 opportunities to agree for 93% agreement. Results indicated that the instructor implemented an average of 96% of the required instructional behaviors across the Word Mapping Strategy lessons.

Was the fidelity measure also used in control classrooms?

No, the control classrooms were engaged in business-as-usual English classes. The vocabulary instruction on the Word Mapping checklist would not have pertained to those classes.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Pretest scores were compared across the three groups using ANOVAs. No statistically significant difference was found between the pretest scores for any of the three tests. To determine whether there were any differences across the three groups regarding their demographic data and standardized test data, chi-square tests and one-way ANOVAs were conducted. The Pearson's chi-square results showed that there were no significant difference between the groups with respect to gender or ethnicity. The ANOVAs revealed no significant differences among the groups on their vocabulary achievement standard scores on the SAT-10 and full-scale score from the WISC-III. (See the article for the statistical statements on p. 21.)

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

N/A. There were no missing data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

The variable related to the instruction in the Vocabulary LINCing Strategy was not included. This variable was the percentage of points earned on the Vocabulary LINCing Strategy Test. That test was only administered to students assigned to the group that received instruction in the Vocabulary LINCing Strategy. Also, two satisfaction questionnaires were administered to the students in the two treatment groups to determine their satisfaction with the instruction. Since these were social validity measures, they were not included here..

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

For the Word Mapping Strategy Test, which measured students' knowledge and use of the strategy, a repeated measures ANOVA with one between-subjects factor (SWDs versus non-SWDS) was used to examine changes from the pretest to the posttest. Then a separate paired sample t-test was performed to determine whether gains made by the subgroups were significant. For the Word Knowledge Test and the Morphological Analysis Test, separate repeated-measures ANOVAs were used with two between-subjects factors (condition and subgroup) to examine changes from pretest to posttest. To examine differences between the posttest scores of the three groups, a two-way ANCOVA with two between-subjects factors (condition x subgroup) was conducted for each measure with the pretest scores serving as the covariate. (See the article for descriptions of the follow-up tests that were run and levels of significance required on p. 27.) The results showed that the Word Mapping SWDs and NSWDs earned significantly higher scores on the posttest than on the pretest for the Word Mapping Test, representing a very large effect size (.925). On the Word Knowledge Test, the Word Mapping SWDS and NSWDs earned significantly higher scores than those in the respective comparison groups, again with large effect sizes. Additionally, both the SWD and NSWD subgroups made significant gains between the pretest and posttest on the Word Knowledge Test, again with large effect sizes represented. On the Morphological Analysis Test, the SWDS and NSWDS in the Word Mapping group earned significantly higher scores than SWDs and NSWDS in the Vocabulary LINCing group and in the comparison group. All of the effect sizes associated with these comparisons were large. Finally, the Word Mapping group earned significantly higher scores on the posttest than on the pretest of the Morphological Analysis Test. (See the details on these statistical tests and the statistical statements on pp. 28-29 of the article.)