Learning Strategies Curriculum: Punctuation Strategies Program
Study: Schumaker, Fisher, & Walsh (2019)

Describe how students were selected to participate in the study:

Fifty-two middle-school students with LD were recruited, and their parents signed written informed consent forms. In addition, 36 high-school students with LD were recruited, and their parents signed written informed consent forms. All of the students were recruited by a researcher who attended their class and explained the study. Interested students took the consent form home for their parents to sign. Students who had signed consent forms and whose parents signed consent forms served in the experimental and control groups. In addition, four English Language Arts general education classes at the ninth-grade level participated as a validation sample (n = 101). The students had been regularly enrolled in these classes. The school administrators allowed these students to take the two forms of one of the tests to show the equivalence of the forms. They did not allow any personal data to be collected on these students. These students' results were also used in a comparison to show how the experimental LD students' punctuation performance compared to the performance of other students in their schools.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

All of the students in the experimental and control groups had been formally classified as having a learning disability using the IQ/discrepancy method as applied according to guidelines in the states of Kansas and Missouri. All of them were receiving special education services for writing deficits as verified by reviewing their IEPs after their parents had signed consent forms.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

100.0%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

The Punctuation Strategies Program was implemented with students randomly selected for the experimental condition.

Specify which condition is the control condition:

Students randomly assigned to the control condition worked independently with computerized programs focusing on the strategic instruction of reading skills. One of the programs, Wordsmith (Lancaster & Schumaker, in prep.) focuses on teaching the meaning of prefixes, suffixes, and roots. The second program, WordSleuth (Lancaster, Harris, & Schumaker, in prep.), focuses on teaching students a strategy for detecting the meaning of words by using the meaning of prefixes, suffixes, and roots. WordSleuth is based on the Word Mapping Strategy (Harris, Schumaker, & Deshler, 2008). References Harris, M. L., Schumaker, J. B., & Deshler, D. D. (2008). The Word Mapping Strategy: Instructor’s manual. Lawrence, KS: Edge Enterprises, Inc. Lancaster, P. E., Harris, M. L., & Schumaker, J. B. (in prep.). Wordsleuth: A multimedia program for teaching students to detect the meaning of words. Lawrence, KS: Edge Enterprises, Inc. Lancaster, P. E., Schumaker, J. B. (in prep.). Wordsmith: A multimedia program for teaching the meaning of prefixes, suffixes, and roots. Lawrence, KS: Edge Enterprises, Inc.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

N/A

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Within each age group (middle school and high school), students were subgrouped according to gender. Then students in each gender and age group were given a code number, and their numbers were drawn randomly "out of a hat" for the experimental group. The remaining students served in the control group.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

Student scores on the pretests and posttests were used in the analyses.

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The instructor was an employee of Edge Enterprises, Inc. She had a Ph.D. in child psychology. She was not a certified teacher. She monitored the students' use of the computers and provided feedback to the students on their final permanent products in which they wrote complete sentences containing punctuation marks. The bulk of the instruction was supplied by the computerized multimedia program, which the students worked on individually.

Describe when and how fidelity of treatment information was obtained.

The computerized program delivered the instruction. The program was designed such that students could not "break" the program or go forward within the program without completing every activity correctly or entering a pass code when they completed a segment of the program. Pass codes were given to the students by the instructor when the student had performed at mastery. Students raised their hands when they completed a practice activity or major section of the multimedia program. At that time, the instructor recorded the score the student had achieved on a given practice activity (which the computer displayed on the screen) or scored a written practice activity. Then the instructor gave the student a passcode to continue through to the next segment. Thus, the instructor recorded the completion of each segment of the program and all the scores that each student earned on each practice activity. These scores served as fidelity of treatment measures in that they showed how the students performed on the quizzes and practice activities throughout the computerized program. (See the attached tables showing the experimental students' scores on practice activities and quizzes as measures of fidelity. )

What were the results on the fidelity-of-treatment implementation measure?

The fidelity of treatment implementation measures showed that all of the experimental middle-school and high-school students completed 100% of the segments of the multimedia program. Students earned average scores well above 90% on all of the practice activities, earned average scores above 86% on the quizzes, and met the mastery criteria on every lesson. (See the article and attached tables for more details.)

Was the fidelity measure also used in control classrooms?

The control students were present in the same classrooms as the experimental students. They worked individually on separate computers on different instructional programs. The instructor kept track of each control student's completion of activities within their assigned computerized programs and the scores they achieved on their practice activities. All of the control students completed 100% of their assigned programs.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

The experimental and control groups were compared with regard to age and standardized test scores using Independent Samples t-tests. The results showed no significant difference between the middle-school experimental and control groups and between the high-school experimental and control groups with regard to age, reading scores, and writing scores. Thus the data analyses proceeded without controlling for these variables. (See the article for the statistics showing no differences between the groups.)

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

There were no missing data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Student and teacher satisfaction results were excluded because they are social validity measures and not outcome measures. (See the article for the results.)

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

The groups’ posttest scores on the Sentence Editing Test and the Sentence Construction Test were compared using ANCOVAs for the following measures: the percentage of punctuation marks inserted correctly, the number of punctuation marks inserted incorrectly, and the percentage of correct complete sentences containing punctuation marks created by the students to match the prompts. The pretest scores served as the covariate in each case. The between-subjects factors were the condition (experimental or control) and the school level (middle school or high school). Independent Samples t-Tests were used to compare the mean posttest scores of the experimental group to the scores of the validation group on the correct and incorrect insertion of punctuation marks. The criterion for significance for all tests was set at .05. The results revealed significant differences for all comparisons, with the experimental student groups (both middle and high school) scoring significantly higher on the correct insertion of punctuation marks, significantly lower on the incorrect insertion of punctuation marks, and significantly higher on the construction of sentences containing punctuation marks than the control groups. No differences were found between the middle-school and high-school experimental students. Additionally, the experimental students earned scores significantly better than their validation peers with regard to the correct and incorrect insertion of punctuation marks. All of the effect sizes were very large. (See the article for the actual statistical numbers and the effect sizes that resulted from each comparison.)