Seeing Stars
Study: Bell, Worthington et al. (n.d)

Describe how students were selected to participate in the study:

Students in grades 2-6 from two comparable schools in the same school district were identified as “at risk” or “not proficient” based on test data.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

Individual screening by AIMsweb identified at risk Grade 2 students; Star testing or state standards testing from the previous year identified at risk grade 3-6 students.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

In the attached report, the treatment group is carried out at Fairmount Elementary School (FAIR) using the instructional program Seeing Stars®: Symbol Imagery for Phonemic Awareness, Sight Words and Spelling

Specify which condition is the control condition:

In the attached report, the control group is Anderson Elementary School (ANDER).

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Systematic

Please describe the assignment method or the process for defining treatment/comparison groups.

Students in grades 2-6 from two comparable schools in the same school district were identified as “at risk” or “not proficient” based on individual screening (AIMsweb or Star) or state standards testing from the previous year (for grades 3-6). All students were administered a battery of reading and oral language tests in the fall and spring. In the program school, students were placed in small groups of 2-5 (mean 4.6), matched for severity of deficit. Small group instruction was 90 min/day, five days/week throughout the school year, administered by trained reading resource teachers under supervision by a Lindamood-Bell® expert. In addition to supervised oversight by the on-site consultant, formal fidelity monitoring took place at two time points for each teacher. Student progress was monitored by AIMsweb for grade 2 and Star testing for grades 3-6. Additional monitoring was provided by resource teachers’ and the consultant’s daily notes and observations in sessions for pacing through the program and for making recommendations about students exiting or changing the program focus. The two schools used the same basal readers. The control school used two computer programs during parts of the school year, most notably iStation during the last third of the school year.

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

Tier 3 resource teachers implementing the program were certified teachers in their state; two were certified in special education. At the beginning of the study, teachers and teaching assistants at each school were asked to complete a survey. At the program school, 26 surveys were completed; at the control school, 13 surveys were completed. On those responding, 77% of the program school teachers had a bachelor’s degree and 31% had earned a master’s; 92% found teaching an effective and rewarding profession and 96% said they use data to improve instruction. At the control school, 77% of respondents had a bachelor’s degree and 38.5% had earned a master’s; 85% found teaching an effective and rewarding profession and 92% said they use data to improve instruction. Prior to the beginning of the project, professional development workshops were held for classroom teachers, reading resource teachers, and educational assistants. The six reading resource teachers who taught small groups and all 13 K-3 grade teachers attended four days of workshops, two days in Seeing Stars® for building decoding and two days for another Lindamood-Bell program (Visualizing and Verbalizing® for building comprehension). The resource teachers also attended a day of additonal training in test interpretation, student grouping and progress monitoring, instructional pacing, and lesson planning. Teachers who have not lead students through a semester or more of instruction require an expert consultant’s assistance for pacing and devising lesson plans that efficiently address the needs of the students. This assistance, with observation and modeling, was provided in the program by a Lindamood- Bell® consultant who was in the building daily.

Describe when and how fidelity of treatment information was obtained.

In addition to regular observation and modeling, the consultant completed a standard fidelity-monitoring checklist during formal observation sessions for each Tier 3 teacher in December and March, scoring compliance on 61 points during Seeing Stars® instruction and 64 points during Visualizing and Verbalizing® instruction. Each item received a score from 0 (not observed) to 3 (consistently observed) for each item of standardized, sequential instruction specific to the instruction (e.g., introducing the task, demonstrating the procedure, use of multisensory exercises, using questioning, and integrating steps).

What were the results on the fidelity-of-treatment implementation measure?

In December, the six reading resource teachers achieved an average of 102 points (range 84-125) out of the 156 points required for independent competence in Seeing Stars®. In March, the same teachers achieved an average of 140.1 points (range 128-157) in Seeing Stars®. Fidelity monitoring was quantified by scoring the resource teachers on each instructional step within these categories: introducing the task, demonstrating the procedure, use of multisensory exercises, using questioning, and integrating steps. Scoring on each item ranged from 0 (not observed) to 3 (consistently observed). The total number of cumulative points required for independent competence in Seeing Stars is 156. However, the teachers were additionally monitored and coached daily by the on-site Lindamood-Bell consultant who oversaw lesson plans for every Tier III student. In consequence, these teachers were substantially supported daily by an expert on-site practitioner.

Was the fidelity measure also used in control classrooms?

Study measures are classified as targeted, broader, or administrative data according to the following definitions:

• Targeted measures
  Assess outcomes, such as competencies or skills that the program was directly targeted to improve.
  • In the academic domain, targeted measures typically are not the very items taught but rather novel items structured similarly to the content addressed in the program. For example, if a program taught word-attack skills, a targeted measure would be decoding of pseudo words. If a program taught comprehension of cause-effect passages, a targeted measure would be answering questions about cause-effect passages structured similarly to those used during intervention, but not including the very passages used for intervention.
  • In the behavioral domain, targeted measures evaluate aspects of external or internal behavior the program was directly targeted to improve and are operationally defined.

• Broader measures
  Assess outcomes that are related to the competencies or skills targeted by the program but not directly taught in the program.
  • In the academic domain, if a program taught word-level reading skill, a broader measure would be answering questions about passages the student reads. If a program taught calculation skill, a broader measure would be solving word problems that require the same kinds of calculation skill taught in the program.
  • In the behavioral domain, if a program taught a specific skill like on-task behavior in one classroom, a broader measure would be academic performance in that setting or on-task behavior in another setting.

• Administrative data measures apply only to behavioral intervention tools and are measures such as office discipline referrals (ODRs) and graduation rates which do not have psychometric properties as do other, more traditional targeted or broader measures.

Click here for more information on effect size.

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

None were excluded.

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

One-way analyses of variance (ANOVA) were used to compare group difference in the baseline to posttest scores. Program students had statistically lower baseline scores on WRAT reading (p=0.01), WRAT spelling (p=0.01), Gray Oral reading rate (p=0.01), and Gray Oral Fluency (p=0.02); Control students had a lower baseline score on Following Oral Directions (p=0.005). Adjustments were made by entering the baseline score as a covariate into the model. The program group made statistically significant gains on all measures and greater gains than the control group on measures of WRAT reading (word recognition), Woodcock word attack (nonword reading), Symbol Imagery (orthographic awareness), and LAC (phonemic awareness).