Check, Connect, and Expect
Study: Cheney et al. (2009)

Describe how students were selected to participate in the study:

Students in 18 schools within three school districts were selected to participate in the study if (a) they were in first, second, or third grade; (b) they were determined to be at risk for developing emotional or behavior disorders; (c) for students identified with disabilities, their IEP team agreed that the program would facilitate attainment of IEP objectives; and (d) their parents and teachers consented to their participation.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students were identified as being at risk for developing emotional or behavior disorders if they passed through the first two stages of the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992). In Stage 1 of the SSBD, teachers rank all students in their classroom in terms of their internalizing and externalizing characteristics, and the three students who rank highest for display of internalizing and externalizing symptoms, respectively, pass on to the second stage. In Stage 2, teachers complete rating scales for these six students. Students for whom ratings met national normative criteria passed through Stage 2 and were considered emotionally or behaviorally at-risk.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

All students entering the program began at the Basic level. In the Basic program, coaches (i.e., non-certified paraprofessionals supervised by a master's-level behavior specialist) checked in students in the morning and checked out students in the afternoon. The check-in process consisted of five elements: (a) checked to see if a parent signed the student's daily progress report (DPR) for the prior day, (b) provided the student the DPR for the current day, (c) reminded the student of DPR expectations, and, as needed, (d) prompted the student to go to class and/or (e) redirected student's inappropriate behavior to the task. Similarly, the check-out process consisted of four steps: (a) collected the student's DPR and tallied earned points, (b) provided verbal praise for student's points, (c) provided verbal feedback for student's inappropriate behavior, and, as needed, (d) redirected student's inappropriate behavior to the task. Coaches entered DPR data into an electronic data system weekly, and reviewed DPR data in collaboration with the behavior specialist every 5-10 days to evaluate whether change in intervention was necessary. Students were deemed successful in the Basic program and entered the Self-Monitoring program when they earned more than 75% of possible points on more than 80% of days across an eight-week period. In the Self-Monitoring program, students rated their own behavior on the DPR and compared their ratings to teacher ratings. When students achieved partial agreement, defined as within one point for each expectation rating, for 10 out of 15 days, they transitioned to only self-monitoring their behaviors (i.e., not comparing ratings to teacher ratings). After two weeks of self-monitoring only, students graduated from the program. If students were not successful in Self-Monitoring, they returned to the Basic program for four weeks with an emphasis on understanding teachers' scores. If students were not deemed successful in the Basic program after 8 weeks, they received additional services in the Basic Plus program. For students who did not meet the 5% criterion for any 2 weeks of the program, their daily criterion was set to a 10% reduction of the student's current average. In addition, incentives were added for meeting criteria during five consecutive days. DPR data were evaluated to identify difficulties with academic task completion and with social skills. Students with diffiulties with academic task completion received tutoring from coaches. Those with difficulties with social skills received social skills instruction provided by the Stop and Think Social Skills Program (Knoff, 2001). Students who met their criteria on 80% of days across 8 weeks re-entered the Basic level. If students did not meet their criterion during 8 weeks in the Basic program and after 8 weeks in the Basic Plus program, they entered the Intensive program. Students were excluded from this program if they were already participating in a school-based FBA intervention or were currently being evaluated for participation in special education. In this Intensive program, students participated in a functionally based behavioral intervention. To inform this intervention, the behavior specialist conducted an FBA involving a teacher interview, a student interview, and five behavioral observations. Conditional probabilities were used to evaluate whether teacher attention, peer attention, or avoidance reinforced the inappropriate behaviors. If one consequence reliably followed the behaviors, one of three interventions was used: (a) for teacher attention, differential reinforcement for teacher attention; (b) for peer attention, the Good Behavior Game; and (c) for escape, differential reinforcement using free time after completing work tasks. If no one function of the behavior was identified as being reliable, a multicomponent intervention was implemented. Students who progressed through the Self-Monitoring program participated in the Graduate program, during which coaches provided them with feedback on their behavior at least monthly and informally interacted with them at least weekly.

Specify which condition is the control condition:

Specific information regarding behavioral supports in the comparison condition was not provided. It was assumed that students in control schools received behavioral supports in alignment with typical school practices.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

"Schools were matched in pairs on four demographic variables: school size, percentage of student population with Individualized Education Plans (IEPs), percentage of students receiving free or reduced-price lunch, and percentage of Caucasian students" (p. 228). These pairs were then randomly assigned to either the intervention or comparison condition.

What was the unit of assignment?

Schools

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The interventionists (coaches, in this case) were noncertified paraprofessionals who were supervised by master's-level behavior specialists. The interventionists' training and level of ongoing support were not described in the article.

Describe when and how fidelity of treatment information was obtained.

Treatment fidelity was assessed via three different evaluations: the Teacher Adherence and Quality Form, the Check-In Adherence Quality Form, and the Check-Out Adherence Quality Form (Cheney & Stage, 2005a,b). The Teacher Adherence and Quality Form was completed by research staff members observing teacher feedback regarding students' DPRs; however, information regarding when and how often this form was completed was not provided. The check-in/check-out forms were completed once weekly by both the coach and the behavior specialist or a research assistant. Check-in/check-out adherence averaged 92% and ranged from 70 to 100%.

What were the results on the fidelity-of-treatment implementation measure?

No differential change in treatment procedures was observed for students of the graduate and non-graduate groups. There was, however, a statistically reliable change in coaches' check-out procedures across the two years of intervention, such that their adherence increased but quality decreased.

Was the fidelity measure also used in control classrooms?

The article does not indicate whether fidelity measures were used in the control classrooms.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

Linear growth analysis within hiearchical linear modeling (level 1: time, level 2: group membership) was used to examine changes in the eight outcome measures (i.e., three SSRS scales, two TRF scales, two WJ-III scales, and AET). The random effect of group membership was evaluated by examining whether the variance associated with group membership was significant at the final intercept as well as across time. To determine whether the intervention group improved more than the control group, the linear growth analysis must demonstrate significant change across the intervention and at the final intercept of the intervention compared to the control group. In addition, an analysis of variance was conducted to determine whether outcome measures were significantly different based on year of entry into the intervention as well as an interaction with student group membership at the end of the study.