Early Vocabulary Connections
Study: Nelson et al. (2011)

Describe how students were selected to participate in the study:

EL kindergarteners were recruited for participation in this study. No student repeated kindergarten during the study; and recruitment, enrollment, assignment, intervention, and testing procedures were identical across cohorts. Students were considered EL if their performance fell within the limited or non-English speaker levels of the Oral Language component of the norm-referenced, district-administered Pre-Literacy Language Assessment Scales 2000 (DeAvila & Duncan, 2000). Further, all EL children came from families who spoke Spanish in the home. In Cohort 1, n=117 children from 14 classrooms were initially enrolled in the study (i.e., students whose parents provided active consent); in Cohort 2, n=93 children from 12 classrooms were initially enrolled.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Students were considered EL if their performance fell within the limited or non-English speaker levels of the Oral Language component of the norm-referenced, district-administered Pre-Literacy Language Assessment Scales 2000 (pre-LAS 2000) (DeAvila & Duncan, 2000). Further, all EL children came from families who spoke Spanish in the home. In cohort 1, n=117 children from 14 classrooms were initially enrolled in the study (i.e., students whose parents provided active consent); in cohort 2, n=93 children from 12 classrooms were initially enrolled. In sum, standard scores of 73.73 for treatment and 76.40 for control on PPVT indicate 4th-5th percentile. Furthermore, all students meet EL status per the district-administered criteria.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

Early Vocabulary Connections was used in the treatment condition.

Specify which condition is the control condition:

The control condition was based on interactive book reading (Wasik & Bond, 2001; Wasik et al., 2006) and was employed primarily to control for instructional time and consistency. Pictures, child friendly definitions, and guiding prompts were used in the control condition to introduce the target words, engage and motivate students, support expanded explanations, and support independent use of word meanings. Materials were similar to those included in the prop boxes used in studies of interactive book reading (see Wasik et al., 2006). We selected storybooks for the control condition that contained the target treatment curriculum words (e.g., bank, nap) and tutors used a three-step instructional sequence with these books.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Within classroom, students were randomly assigned to one of two small groups (ranging in size from 2 to 5 children per group, with an average of three per group); small groups were randomly assigned to receive small-group instruction in either Early Vocabulary Connections (treatment) or a modified form of interactive book reading (control). (The control condition was used primarily to control for time and quality of instruction.)

What was the unit of assignment?

Other

If other, please specify:

Small Groups

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:
Small Groups

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

All paraeducator tutors/tutor teams (paraeducators) were recruited from their school communities based on their interest in working with children, prior tutoring and school volunteer experience, and scheduling flexibility. The assignment of small groups to tutors was wholly determined by a combination of classroom scheduling, paraeducator availability, and the number of eligible students within classrooms within sites. Each tutor was trained in only one of the treatment conditions (i.e., treatment or control, not both), and most provided tutoring to 5 small groups (range was 4 to 6 small groups per tutor) across two sites. The participating paraeducators were female and mostly non-minority (75%), and varied in their age, educational levels (88% had secondary school diploma or its recognized equivalent), and general tutoring experience (50% had previous experience as a paraprofessional). We used a five-step process to train tutors to implement the respective instructional components for the treatment and control conditions components correctly (i.e., > 98 percent): 1) a trainer provides an overview of the theory, research base, rationale, and implementation format for the programs; 2) a trainer models and practices the implementation activities with staff; 3) simulated practice conditions are conducted to ensure that a high level of skill performance is obtained by staff; 4) a trainer provides structured feedback to staff on their proficiency; and 5) the data collectors monitor treatment fidelity and a trainer provides ongoing instruction and modeling when applicable to ensure high-quality implementation. Initial training was four hours in length. The primary focus of the initial training session was to insure tutors could implement each of the treatment conditions and their separate components with integrity. Once the intervention began, the first three lessons that were delivered by tutors were observed by project staff who provided corrective feedback to insure treatment fidelity. The tutors were then observed by project staff on six additional occasions to assess treatment fidelity. Following each observation, if needed, project staff provided additional corrective feedback to the tutors.

Describe when and how fidelity of treatment information was obtained.

Each tutor was observed delivering instruction to their small groups onsite by research staff monthly on six equidistant occasions. Two types of fidelity were recorded by research staff using criteria on general teaching behaviors (comparable across conditions) and intervention component behaviors (specific to each condition). For general delivery, tutors were rated on a 5-point rating scale (0=never to 4=proficient) on three criteria: (a) whether book/props were visible to all children, (b) whether all children were responsive during the session, and (c) whether the children were appropriately kept on task.

What were the results on the fidelity-of-treatment implementation measure?

No differences among small groups (treatment, M = 3.81, SD = 0.19; control, M = 3.73, SD = 0.15) were detected between conditions, t(55) = 1.75, p > .05.For intervention component behaviors, tutors were rated on instructional components corresponding to their specific intervention condition, and thus ratings are not directly comparable. For treatment, components included Word Blending and Spelling, Word Meaning, Fast Read Passage, Sentence Completion, Word Meaning Match, and Say a Sentence; small groups averaged M = 3.87 (SD = 0.10) on the 5-point component behavior rating scale. For control, components included vocabulary introduction, interactive book reading, and extension activity; small groups averaged M = 3.74 (SD = 0.12) on the [same] 5-point component behavior rating scale. Given the maximum value of 4 on the behavior rating scales, we note that fidelity – both general and component – was extremely high: Converted into a percentage, general fidelity averaged 95% and 93% for treatment and control, respectively, and for component fidelity, the corresponding averages were 97% and 94%. Interrater reliability was conducted one time per instructional group. The interrater reliability (Pearson’s r) for the treatment and control groups were .94 and .97, respectively.

Was the fidelity measure also used in control classrooms?

See answer to 5 above. For control, components included vocabulary introduction, interactive book reading, and extension activity; small groups averaged M = 3.74 (SD = 0.12) on the [same] 5-point component behavior rating scale. Interrater reliability (Pearson’s r) for the control groups was .97.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

The study design was a cluster randomized design, therefore hierarchical multilevel modeling approach was employed to test for differences between conditions using 3-level models in which student scores (Level 1) as nested within small groups (Level 2), which were in turn nested within school (Level 3). In each model, condition was dummy coded such that 1=treatment and 0=control. In essence, our pretest models are 2-group t-tests, except that we allowed for estimation of random effects due to small groups and schools. See the Analytic Approach section on page 197 for further details and a rationale for the approach used.