Check, Connect & Expect

Study: Cheney, Stage, Hawken, Lynass, Mielenz, & Waugh (2009)

Study Type: Group-Design

Participants: Unconvincing Evidence

Sample size: 207 students (121 program, 86 control)

Risk Status: Students were identified as being at risk for developing emotional or behavior disorders if they passed through the first two stages of the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992). In Stage 1 of the SSBD, teachers rank all students in their classroom in terms of their internalizing and externalizing characteristics, and the three students who rank highest for display of internalizing and externalizing symptoms, respectively, pass on to the second stage.  In Stage 2, teachers complete rating scales for these six students.  Students for whom ratings met national normative criteria passed through Stage 2 and were considered emotionally or behaviorally at-risk.

Demographics:

 

Program

Control

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

0

0%

0

0%

  Grade 1

9

7%

6

7%

  Grade 2

36

30%

22

25%

  Grade 3

48

40%

39

45%

  Grade 4

25

21%

15

18%

  Grade 5

1

1%

6

6%

  Grade 6

0

0%

0

0%

  Grade 7

0

0%

0

0%

  Grade 8

0

0%

0

0%

  Grade 9

0

0%

0

0%

  Grade 10

0

0%

0

0%

  Grade 11

0

0%

0

0%

  Grade 12

0

0%

0

0%

Mean Age

 

 

 

 

Race-ethnicity

  African-American

20

17%

16

19%

  American Indian

1

1%

0

0%

  Asian/Pacific Islander

9

7%

7

8%

  Hispanic

9

7%

14

16%

  White

62

51%

45

52%

  Other

 

 

 

 

Socioeconomic status

  Subsidized lunch

48

40%

47

55%

  No subsidized lunch

81

67%

39

51%

Disability status

  Speech-language impairments

       

  Learning disabilities

       

 Emotional disturbance

       

 Intellectual disability

       

  Other

 

 

 

 

  Not identified with a disability

18

15%

30

63%

ELL status

  English language learner

 

 

 

 

  Not English language learner

 

 

 

 

Gender

  Female

29

24%

26

30%

  Male

92

76%

60

70%

Training of Instructors: The interventionists (coaches, in this case) were noncertified paraprofessionals who were supervised by master's-level behavior specialists.  The interventionists' training and level of ongoing support were not described in the article.

Design: Partially Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: Not applicable

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes? Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? Not applicable

Were the program and control groups demographically comparable?: Yes

Was there differential attrition for the program and the control groups?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: No

Implemented with Fidelity: Convincing Evidence

Description of when and how fidelity of treatment information was obtained: Treatment fidelity was assessed via three different evaluations: the Teacher Adherence and Quality Form, the Check-In Adherence Quality Form, and the Check-Out Adherence Quality Form (Cheney & Stage, 2005a,b). The Teacher Adherence and Quality Form was completed by research staff members observing teacher feedback regarding students' DPRs; however, information regarding when and how often this form was completed was not provided.  The check-in/check-out forms were completed once weekly by both the coach and the behavior specialist or a research assistant.  Check-in/check-out adherence averaged 92% and ranged from 70 to 100%.

Results on the fidelity of treatment implementation measure: No differential change in treatment procedures was observed for students of the graduate and non-graduate groups.  There was, however, a statistically reliable change in coaches' check-out procedures across the two years of intervention, such that their adherence increased but quality decreased.

Measures Targeted: Convincing Evidence

Targeted Measure

Reliability statistics

Relevance to program focus

Exposure to related support among control group

Social Skills Rating System-Teacher Form (Gresham & Elliott, 1990)

Test-retest reliability: range of 0.84 to 0.93

Measurement of problem behavior as well as academic and social skills (related to DPR)

Not described

Teacher Report Form (Achenbach, 2001)

Test-retest reliability: range of 0.72 to 0.97

Measurement of problem behavior as well as academic and social skills (related to DPR)

Not described

Academic Engaged Time (Walker & Severson, 1992)

Interobserver agreement on 20% of scheduled observations: mean percentage agreement 0.918 (SD = 10)

Measurement of classroom engagement (related to DPR)

Not described

 

 

Broader Measure

Reliability statistics

Relevance to program focus

Exposure to related support among control group

Woodcock Johnson III Tests of Achievement (Woodcock, McGrew, & Mather, 2001)

Test-retest reliabilities: 0.95 (Letter-Word Identification); 0.86 (Calculation)

Increased behavioral and social competence (e.g., AET) should result in higher academic competence

Children in the control group were likely participating in similar curricular programs.

 

Mean ES Targeted Outcomes: 0.18*

Mean ES Administrative Outcomes: N/A

Effect Size:

Targeted Measures

Measure

Effect Size

SSRS Problem Behavior

0.29*

SSRS Social Skills

0.05

TRF Externalizing Problems

0.13

TRF Internalizing Problems

0.51***

SRSS Academic Competence

-0.06

AET

0.18

 

Broader Measures

Measure

Effect Size

WJ-III Calculation

0.18

WJ-III Letter Word Identification

0.29*

 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

u      Effect size is based on unadjusted means

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single-Subject Designs): N/A

Disaggregated Outcome Data Available for Demographic Subgroups: No

Target Behavior(s): Externalizing

Delivery: Individual, Small group

Fidelity of Implementation Check List Available: No

Minimum Interventionist Requirements: Paraprofessional, No training required

Intervention Reviewed by What Works Clearinghouse: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

Other Research: Potentially Eligible for NCII Review: 0 studies