STAR
Reading

Summary

Where to Obtain:
Renaissance Learning
answers@renaissance.com
Renaissance Learning, PO Box 8036, Wisconsin Rapids, WI 54495
(800) 338-4204
http://www.renaissance.com
Initial Cost:
Contact vendor for pricing details.
Replacement Cost:
Contact vendor for pricing details.
Included in Cost:
There is a one-time setup fee along with a per student subscription fee. Total cost will depend on the number of schools and students. Please contact: answers@renaissance.com or (800) 338-4204 for specific details on pricing for your district. Star Reading is cloud-based and purchase includes the tool, software/technical manual, installation guide, testing instructions, and remote installation and setup.
Star Reading is a computer-adaptive assessment, and the difficulty of items presented is adjusted automatically to reflect the skill level of the student. Students may use the keyboard or the mouse, accommodating students with limited motor skills. Star Reading offers several accommodations for students with disabilities through the accessibility options built into a computer's operating system. For students with limited vision, the introductory screens of Star Reading respond to the "high contrast" accessibility feature within Windows and the "switch to black and white" accessibility feature in Mac OS. The assessment screens within Star Reading already provide visual contrast through a yellow background and black writing. Furthermore, Star Reading is compatible with Mac’s "zoom in" accessibility feature, which allows users to magnify nearly all Star Reading screens.
Training Requirements:
Less than one hour of training.
Qualified Administrators:
Access to Technical Support:
Renaissance Technical Support Staff, reachable by phone, email, or chat.
Assessment Format:
  • Individual
  • Small group
  • Large group
  • Computer-administered
Scoring Time:
  • Scoring is automatic OR
  • 0 minutes per student
Scores Generated:
  • Percentile score
  • Grade equivalents
  • IRT-based score
  • Normal curve equivalents
  • Equated
  • Lexile score
  • Other : Instructional Reading Level, Scaled Score
Administration Time:
  • 18 minutes per student
Scoring Method:
  • Automatically (computer-scored)
Technology Requirements:
  • Computer or tablet
  • Internet connection

Tool Information

Descriptive Information

Please provide a description of your tool:
Is your tool designed to measure progress towards an end-of-year goal (e.g., oral reading fluency) or progress towards a short-term skill (e.g., letter naming fluency)?
not selected
selected
The tool is intended for use with the following grade(s).
not selected Preschool / Pre - kindergarten
not selected Kindergarten
selected First grade
selected Second grade
selected Third grade
selected Fourth grade
selected Fifth grade
selected Sixth grade
selected Seventh grade
selected Eighth grade
selected Ninth grade
selected Tenth grade
selected Eleventh grade
selected Twelfth grade

The tool is intended for use with the following age(s).
not selected 0-4 years old
not selected 5 years old
not selected 6 years old
not selected 7 years old
not selected 8 years old
not selected 9 years old
not selected 10 years old
not selected 11 years old
not selected 12 years old
not selected 13 years old
not selected 14 years old
not selected 15 years old
not selected 16 years old
not selected 17 years old
not selected 18 years old

The tool is intended for use with the following student populations.
not selected Students in general education
not selected Students with disabilities
not selected English language learners

ACADEMIC ONLY: What dimensions does the tool assess?

Reading
selected Global Indicator of Reading Competence
not selected Listening Comprehension
selected Vocabulary
not selected Phonemic Awareness
not selected Decoding
selected Passage Reading
not selected Word Identification
selected Comprehension

Spelling & Written Expression
not selected Global Indicator of Spelling Competence
not selected Global Indicator of Writting Expression Competence

Mathematics
not selected Global Indicator of Mathematics Comprehension
not selected Early Numeracy
not selected Mathematics Concepts
not selected Mathematics Computation
not selected Mathematics Application
not selected Fractions
not selected Algebra

Other
Please describe specific domain, skills or subtests:


BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.
BEHAVIOR ONLY: Which category of behaviors does your tool target?

Acquisition and Cost Information

Where to obtain:
Email Address
answers@renaissance.com
Address
Renaissance Learning, PO Box 8036, Wisconsin Rapids, WI 54495
Phone Number
(800) 338-4204
Website
http://www.renaissance.com
Initial cost for implementing program:
Cost
Unit of cost
Replacement cost per unit for subsequent use:
Cost
Unit of cost
Duration of license
Additional cost information:
Describe basic pricing plan and structure of the tool. Provide information on what is included in the published tool, as well as what is not included but required for implementation.
There is a one-time setup fee along with a per student subscription fee. Total cost will depend on the number of schools and students. Please contact: answers@renaissance.com or (800) 338-4204 for specific details on pricing for your district. Star Reading is cloud-based and purchase includes the tool, software/technical manual, installation guide, testing instructions, and remote installation and setup.
Provide information about special accommodations for students with disabilities.
Star Reading is a computer-adaptive assessment, and the difficulty of items presented is adjusted automatically to reflect the skill level of the student. Students may use the keyboard or the mouse, accommodating students with limited motor skills. Star Reading offers several accommodations for students with disabilities through the accessibility options built into a computer's operating system. For students with limited vision, the introductory screens of Star Reading respond to the "high contrast" accessibility feature within Windows and the "switch to black and white" accessibility feature in Mac OS. The assessment screens within Star Reading already provide visual contrast through a yellow background and black writing. Furthermore, Star Reading is compatible with Mac’s "zoom in" accessibility feature, which allows users to magnify nearly all Star Reading screens.

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?
not selected
not selected
not selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: What is the administration format?
not selected
not selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: What is the administration setting?
not selected
not selected
not selected
not selected
not selected
not selected
not selected
If other, please specify:

Does the program require technology?

If yes, what technology is required to implement your program? (Select all that apply)
selected
selected
not selected

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?
selected
selected    If small group, n=
selected    If large group, n=
selected
not selected
If other, please specify:

What is the administration time?
Time in minutes
18
per (student/group/other unit)
student

Additional scoring time:
Time in minutes
0
per (student/group/other unit)
student

How many alternate forms are available, if applicable?
Number of alternate forms
per (grade/level/unit)

ACADEMIC ONLY: What are the discontinue rules?
selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: Can multiple students be rated concurrently by one administrator?
If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?
Yes
Describe the time required for administrator training, if applicable:
Less than one hour of training.
Please describe the minimum qualifications an administrator must possess.
not selected No minimum qualifications
Are training manuals and materials available?
Yes
Are training manuals/materials field-tested?
Yes
Are training manuals/materials included in cost of tools?
Yes
If No, please describe training costs:
Can users obtain ongoing professional and technical support?
Yes
If Yes, please describe how users can obtain support:
Renaissance Technical Support Staff, reachable by phone, email, or chat.

Scoring

BEHAVIOR ONLY: What types of scores result from the administration of the assessment?
Score
Observation Behavior Rating
not selected Frequency
not selected Duration
not selected Interval
not selected Latency
not selected Raw score
Conversion
Observation Behavior Rating
not selected Rate
not selected Percent
not selected Standard score
not selected Subscale/ Subtest
not selected Composite
not selected Stanine
not selected Percentile ranks
not selected Normal curve equivalents
not selected IRT based scores
Interpretation
Observation Behavior Rating
not selected Error analysis
not selected Peer comparison
not selected Rate of change
not selected Dev. benchmarks
not selected Age-Grade equivalent
How are scores calculated?
not selected Manually (by hand)
selected Automatically (computer-scored)
not selected Other
If other, please specify:

Do you provide basis for calculating performance level scores?
Yes

What is the basis for calculating performance level and percentile scores?
not selected Age norms
selected Grade norms
not selected Classwide norms
not selected Schoolwide norms
not selected Stanines
not selected Normal curve equivalents

What types of performance level scores are available?
not selected Raw score
not selected Standard score
selected Percentile score
selected Grade equivalents
selected IRT-based score
not selected Age equivalents
not selected Stanines
selected Normal curve equivalents
not selected Developmental benchmarks
not selected Developmental cut points
selected Equated
not selected Probability
selected Lexile score
not selected Error analysis
not selected Composite scores
not selected Subscale/subtest scores
selected Other
If other, please specify:
Instructional Reading Level, Scaled Score

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.
All scores are calculated automatically by the software. The software calculates a maximum likelihood Rasch ability estimate based on the calibrated difficulty of the items that were administered to the student, and the pattern of the student’s right and wrong responses to those items. Star Reading uses a proprietary, Rasch-based, 1-parameter logistic response model to calculate scores. The scaled score is a non-linear, monotonic transformation of the Rasch ability estimate resulting from the adaptive test. From the scaled scores and the student’s current grade placement are derived grade equivalent, percentile, and normal curve equivalent scores. No cluster, composite, or raw scores are reported.
Do you provide basis for calculating slope (e.g., amount of improvement per unit in time)?
Yes
ACADEMIC ONLY: Do you provide benchmarks for the slopes?
Yes
ACADEMIC ONLY: Do you provide percentile ranks for the slopes?
Yes
What is the basis for calculating slope and percentile scores?
not selected Age norms
selected Grade norms
not selected Classwide norms
not selected Schoolwide norms
not selected Stanines
not selected Normal curve equivalents

Describe the tool’s approach to progress monitoring, behavior samples, test format, and/or scoring practices, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.
Star Reading’s computer adapative multiple-choice item format assesses vocabulary, fluency, and comprehension skills in five broad Common Core State Standard domains: Word Knowledge and Skills; Comprehension Strategies and Constructing Meaning; Analyzing Literary Text; Understanding Author’s Craft; and Analyzing Argument and Evaluating. Star Reading’s items assess 475 grade-specific skills; multiple items are available to measure each skill. The assessment’s computer-adaptive structure matches students to items of appropriate difficulty, which in turn may help to reduce student frustration during testing. The item bank is demographically and contextually balanced. Test blueprint goals are established and tracked to ensure appropriate balance in items addressing use of fiction and nonfiction text, subject and topic areas, geographic region, gender, ethnicity, occupation, age, and disability. Items are free of stereotyping, representing different groups of people in non-stereotypical settings. Items do not refer to inappropriate content that includes, but is not limited to content that presents stereotypes based on ethnicity, gender, culture, economic class, or religion. Items do not present any ethnicity, gender, culture, economic class, or religion unfavorably. Items do not introduce inappropriate information, settings, or situations. All items are subjected to field testing, rigorous psychometric review and related calibration procedures before being included in operational item banks. This skill measurement provides a crucial component in progress monitoring. As students learn new skills, Star Reading can assess the level of achievement as often as weekly, although less frequent testing is recommended to avoid over-testing. All items have been calibrated, so that the difficulty of each item is expressed on a Rasch difficulty scale that spans the range of reading proficiency from grade 1 through grade 12. Rasch scores are converted to a wide range of scores including Scaled Score, Percentile Rank, Normal Curve Equivalent, etc. Student Growth Percentiles are also reported after a least two tests, to help the teacher understand the student’s velocity relative to academic peers (those in the same grade with a similar score history). For students receiving intensive intervention, the software provides a goal setting tool that presents teachers with several goal options including alignment with state assessment proficiency levels, growth norms (Student Growth Percentile), and score norms (Percentile Rank). Once a goal has been set for a student, the student’s progress on Star Reading is graphed against the goal line on the progress monitoring report.

Rates of Improvement and End of Year Benchmarks

Is minimum acceptable growth (slope of improvement or average weekly increase in score by grade level) specified in your manual or published materials?
Yes
If yes, specify the growth standards:
Star Reading reports Student Growth Percentile for students with two or more tests. SGP describes a student’s velocity (slope) relative to a national sample of academic peers – those students in the same grade with a similar score history. SGPs work like Percentile Ranks (1-99 scale) but once an SGP goal has been set, it is converted to a Scaled Score goal at the end date specified by the teacher. Then it is converted into an average weekly increase in a Scaled Score metric. We recommend SGP 50 (which indicates typical or expected growth) as minimum acceptable growth, though teachers are of course free to establish their own goals, as they best understand both the student and the intensity of services to be provided.
Are benchmarks for minimum acceptable end-of-year performance specified in your manual or published materials?
Yes
If yes, specify the end-of-year performance standards:
Renaissance provides default risk benchmarks at the 10th (urgent intervention), 25th (intervention) and 40th (low risk, lower-bound threshold for working at grade level) Percentile Ranks. Those default cutpoints can be adjusted by educators but are based on expert guidance. For educators wanting to base decisions on likely performance on end-of-year state summative assessments, Renaissance also provides links to those state proficiency categories. Educators select from the default PR benchmarks, the state proficiency projections, or their own custom benchmarks.
What is the basis for specifying minimum acceptable growth and end of year benchmarks?
selected
selected
not selected Other
If other, please specify:
False

If norm-referenced, describe the normative profile.

National representation (check all that apply):
Northeast:
selected New England
selected Middle Atlantic
Midwest:
selected East North Central
selected West North Central
South:
selected South Atlantic
selected East South Central
selected West South Central
West:
selected Mountain
selected Pacific

Local representation (please describe, including number of states)
Data from all 50 states plus the District of Columbia are included in the sample used to develop both growth norms (SGP) and score norms.
Date
Both types of norms updated just prior to the 2017-18 school year. Criterion-referenced benchmarks are based on scale linkages between state summative reading tests and Star Reading. Completion dates vary by state.
Size
SGP: 9.8 million students. Score norms: 5.8 million students. Criterion-referenced benchmarks (state proficiency indicators): Sample size varies by state. The responses below are for the score norms. SGP and criterion-referenced standards involve different samples.
Gender (Percent)
Male
50
Female
50
Unknown
SES indicators (Percent)
Eligible for free or reduced-price lunch
(FRL data collected at a school level)
Other SES Indicators
Race/Ethnicity (Percent)
White, Non-Hispanic
53.5
Black, Non-Hispanic
Hispanic
American Indian/Alaska Native
Asian/Pacific Islander
Other
Unknown
Disability classification (Please describe)
Not available.

First language (Please describe)
Not available.

Language proficiency status (Please describe)
Not available.
Do you provide, in your user’s manual, norms which are disaggregated by race or ethnicity? If so, for which race/ethnicity?
not selected White, Non-Hispanic
not selected Black, Non-Hispanic
not selected Hispanic
not selected American Indian/Alaska Native
not selected Asian/Pacific Islander
not selected Other
not selected Unknown

If criterion-referenced, describe procedure for specifying criterion for adequate growth and benchmarks for end-of-year performance levels.
For many districts, end-of-year performance on the state summative test is of extreme importance, and they may choose to define intervention cateogires based on state proficiency levels (e.g., students likely to place in the lowest proficiency category may be identified for intenstive intervention). Star Reading predicts performance on end-of-year summative reading/ELA assessments, allowing educators to estimate where students are likely to place as they are being progress monitored throughout the year. This is accomplished by (1) a concurrent scale linking between Star Reading and the state’s reading/ELA test scale (and accompanying proficiency cut points), and (2) growth norms that project students forward in time, to the time of the state assessment. Technical reports are available for each state linking study. The process used to project growth is Student Growth Percentile, and information on that approach can be found here. If educators choose to use their state reading/ELA proficiency categories as their benchmarks, a number of Star Reading dashboards and reports specify those benchmarks (equated to the Star Reading scale) and the growth necessary to achieve those benchmarks.

Describe any other procedures for specifying adequate growth and minimum acceptable end of year performance.

Performance Level

Reliability

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
*Offer a justification for each type of reliability reported, given the type and purpose of the tool.
The internal consistency reliability coefficient estimates the proportion of variability within a single administration of a test that is due to inconsistency among the items that comprise the test.
*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.
For each grade, a large sample (n = 100,000) of students completed Star Reading assessments throughout the 2012–2013 school year.
*Describe the analysis procedures for each reported type of reliability.
Reliability was defined as the proportion of test score variance that is attributable to true variation in the trait the test measures, where σ2 error is the variance of the errors of measurement, and σ2 total is the variance of test scores. The variance of the test scores was calculated from Scaled Score data. The variance of the errors of measurement was estimated from the conditional standard error of measurement (CSEM) statistics that accompany each of the IRT-based test scores, including the Scaled Scores. The conditional standard error of measurement (CSEM) was calculated along with the IRT ability estimate and Scaled Score. Squaring and summing the CSEM values yielded an estimate of total squared error; dividing by the number of observations yielded an estimate of error variance. Generic reliability was calculated by subtracting the ratio of error variance to Scaled Score variance from 1.

*In the table(s) below, report the results of the reliability analyses described above (e.g., model-based evidence, internal consistency or inter-rater reliability coefficients). Include detail about the type of reliability data, statistic generated, and sample size and demographic information.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
Yes
Provide citations for additional published studies.
Renaissance Learning (2016). Star Assessments™ for Reading Abridged Technical Manual. Wisconsin Rapids, WI: Author. Available by request to research@renaissance.com.
Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
No

If yes, fill in data for each subgroup with disaggregated reliability data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.

Validity

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.
All criterion measures were external to the screening tool system and represent widely used assessments of general reading ability including the following: • ACT Aspire: A nationally normed summative test of general reading achievement used in many states and designed to measure students' progress toward college and career readiness. • CAT-5. The California Achievement Test, is a nationally normed standardized test that measures achievement in reading. • ITBS. The Iowa Tests For Basic Skills are nationally normed standardized tests that offer educators a diagnostic look at how their students are progressing in reading comprehension. • MAP: NWEA’s Measures of Academic Progress is a nationally norm-referenced computer adaptive test of general reading achievement • PARCC. The Partnership for Assessment of Readiness for College and Careers end-of-year assessment covers reading and is intended to be used as an indicator of student needs and progress. • SBA. Smarter Balanced assessments are summative tests designed to measure student achievement and growth in reading to support teaching and learning. • ACT. The American College Testing college readiness assessment is a national standardized test for high school achievement and college admissions. • SAT EBRW. The SAT Evidence-based reading and writing assessment is a standardized test widely used for college admissions in the United States.
*Describe the sample(s), including size and characteristics, for each validity analysis conducted.
Samples included students who had taken both Star Reading and the criterion measure. The sample sizes varied across criterion and grades, ranging from 105 to 10,800 students.
*Describe the analysis procedures for each reported type of validity.
Concurrent and predictive correlations were calculated. A criterion assessment was considered concurrent if it was taken during the same school year as the Star Reading assessment. The correlation was considered predictive if the criterion assessment was taken one or more school years after the Star Reading assessment.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of validity analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.
Renaissance Learning (2016). Star Assessments™ for Reading Abridged Technical Manual. Wisconsin Rapids, WI: Author. Available by request to research@renaissance.com.
Describe the degree to which the provided data support the validity of the tool.
The provided data indicate that Star Reading results correspond strongly to other various respected measures of general reading ability.
Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
No

If yes, fill in data for each subgroup with disaggregated validity data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of validity analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.

Bias Analysis

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.
Yes
If yes,
a. Describe the method used to determine the presence or absence of bias:
Logistic regression analyses conditional on ability, group membership, and ability by group interaction were conducted to assess the presence of both uniform and non-uniform DIF simultaneously. Additionally, an effect size measure – Nagelkerke R-squared – was computed to quantify the magnitude of DIF where present.
b. Describe the subgroups for which bias analyses were conducted:
DIF analyses were conducted for gender (males and females) and race/ethnicity (Caucasian, African American, American Indian, Asian, and Hispanic subpopulations). Due to insufficient samples sizes on English Language Learner (ELLs) and students with disabilities (SWD), DIF analyses for these two subgroups were not possible at the time of the analyses.
c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.
Using a blended criterion that flagged items for uniform/non-uniform DIF if they had a p-value less than 0.01 and Nagelkerke R2 greater than or equal to 0.035, the results indicated that Star Reading is sufficiently bias-free. A total of 153 items (2% of the Star Reading items) were flagged for DIF. Those flagged items were removed from the item banks for review and recalibration.

Growth Standards

Sensitivity: Reliability of Slope

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe the sample, including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.
The sample consisted of students who took Star Reading tests during the 2016-2017 school year. Sample sizes ranged from 384 to 52,670 students depending on grade.
Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).
A student was included in the sample if they had performance below PR 30; tested at least ten times during the school year; and had 140 days or more between their first and last test.
Describe the analysis procedures.
For each student, tests were divided into two groups – odd and even numbered test depending on the chronological order in which they were taken. For each test group for each student, a slope was estimated by computing an OLS regression coefficient. The table below summarizes Pearson correlation coefficients as a measure of the strength of association between even and odd numbered test slopes.

In the table below, report reliability of the slope (e.g., ratio of true slope variance to total slope variance) by grade level (if relevant).

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.
Do you have reliability of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?
No

If yes, fill in data for each subgroup with disaggregated reliability of the slope data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.

Sensitivity: Validity of Slope

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.
Describe the sample(s), including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.
Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).
Describe the analysis procedures for each reported type of validity.

In the table below, report predictive validity of the slope (correlation between the slope and achievement outcome) by grade level (if relevant).
NOTE: The TRC suggests controlling for initial level when the correlation for slope without such control is not adequate.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published validity studies:
Provide citations for additional published studies.
Describe the degree to which the provided data support the validity of the tool.
Do you have validity of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?

If yes, fill in data for each subgroup with disaggregated validity of the slope data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published validity studies:
Provide citations for additional published studies.

Alternate Forms

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Unconvincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe the sample for these analyses, including size and characteristics:
The analysis sample was comprised of students struggling in Reading whose initial score was at or below the 25th percentile in Star Reading. The data were drawn from students’ first and second tests of the year; the testing occasions were close in time to one another.
What is the number of alternate forms of equal and controlled difficulty?
Because STAR Reading is computer adaptive and comprises several thousand items, there are at a minimum several hundred alternate forms for a student at a given ability level.
If IRT based, provide evidence of item or ability invariance
The table below presents a test-retest analysis correlating scores from students’ first and second Star Reading administrations. The results continue to reveal a high degree of consistency between the measures Correlations Between First and Second Administrations of Star Reading During the 2016-17 School Year for Students with a Percentile Rank Below 25 Grade n Pearson Coefficient 1 20,835 .42 2 49,618 .65 3 55,906 .67 4 47,822 .69 5 45,968 .70 6 35,143 .72 7 26,966 .71 8 25,209 .71 9 15,812 .71 10 12,049 .70 11 7,749 .69 12 5,733 .69
If computer administered, how many items are in the item bank for each grade level?
Star Reading Item Counts by Grade Grade Items K 362 1 567 2 752 3 1,072 4 807 5 710 6 600 7 525 8 422 9 269 10 281 11 176 12 106 13+ 50 Total 6,699 Note that adaptive tests such as Star Reading are not constrained to use items from the item bin matching the student’s assigned grade level.
If your tool is computer administered, please note how the test forms are derived instead of providing alternate forms:
Star Reading is an instance of computer-administered adaptive testing, based on item response theory. It administers reading test items selected one at a time to tailor item difficulty to the student’s own performance as it progresses during each test. The items have all been calibrated previously, using the 1-parameter logistic “Rasch” item response model. After each item is administered, the student’s score is updated, and a target Rasch difficulty level is calculated. The adaptive item selection algorithm seeks an item close to the target difficulty level, consistent with a content blueprint that specifies the number of items to be administered from each of five reading content domains and 16 general skills. The adaptive item bank contains more than 6,000 items. The content constraints ensure that each test is parallel to others at the same grade level, in terms of content. The adaptive item selection ensures comparable measurement precision from one test to another. If a student’s performance differs little from one testing occasion to another, the successive tests can be expected to be closely parallel to one another in difficulty and score level.

Decision Rules: Setting & Revising Goals

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
In your manual or published materials, do you specify validated decision rules for how to set and revise goals?
No
If yes, specify the decision rules:
The Star Reading software provides educators with guidance regarding the establishment of goals for each individual student to inform instructional decisions. There is not a single recommended goal for each grade level; instead, the goal setting approach is unique to each student and takes into consideration his/her prior achievement trajectory. The default goal for each individual student is based on national growth norms, as discussed elsewhere in the submission. Growth norms are used to understand typical and above typical rates of growth for a student based on comparisons with academic peers – those students in the same grade with a similar score history. Educators are encouraged to select either a typical rate of growth (one in which the student has about a 50% chance of attaining) or a more aggressive rate of growth (one in which the student has about a 35% chance of attaining. Educators may opt to select a different goal, but are not encouraged to select a goal below the 50% threshold, as students in intensive intervention are typically trying to catch up to grade level peers. Throughout the course of the intervention and progress monitoring, educators are encouraged to check student progress. Empircal research on Star Reading decision rules (Cormier & Bulut, 2017, manuscript submitted for publication) has determined that decisions can safely be made after a minimum of five administrations. So, after at least five administrations, educators review the Star Reading Progress Monitoring report. If actual student progress is at or above the student’s growth norm-based goal line, educators are encouraged to continue the intervention or revise the goal upward. If actual student progress is below the student’s growth norm-based goal line, educators are encouraged to alter the intervention and supports the student is receiving. A sample Star Progress Monitoring report is available from the Center upon request.
What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.
A quasi-experimental study was conducted involving more than 6.4 million students who began the year as struggling readers (defined as having Star Reading National Percentile Ranks below 30). Students were classified as treatment if they used the Star Reading goal setting recommendations, and control if they did not. All students in the study (both treatment and comparison) were intensively progress monitored during the 2014-15 school year on an approximately weekly basis. (The duration of progress monitoring varied, with decisions up to local educators and unique to each student.) All students’ beginning (fall) and end of year (spring) Star Reading Normal Curve Equivalent (NCE) scores were compared (posttest minus pretest), and are summarized by grade and condition in the table below. In every grade, 1-12, treatment students (using the Star Reading goal recommendations) experienced greater NCE change than their control counterparts who did not use the goal recommendations, providing support for the claim that using Star Reading’s evidence-based goal recommendations and decision rules is superior to other locally determined approaches. Within each grade, the differences in NCE gains favoring treatment students were significant at p < .001. The results of this study suggest that students experience greater growth when educators follow the individualized goal and decision rule guidance provided by Star Reading. The guidance is based on a combination of expert guidance and growth norms drawn from a large, diverse sample of students. A table detailing the results of this study is available from the Center upon request.

Decision Rules: Changing Instruction

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
In your manual or published materials, do you specify validated decision rules for when changes to instruction need to be made?
No
If yes, specify the decision rules:
What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.

Disclaimer

Most tools and programs evaluated by the NCII are branded products which have been submitted by the companies, organizations, or individuals that disseminate these products. These entities supply the textual information shown above, but not the ratings accompanying the text. NCII administrators and members of our Technical Review Committees have reviewed the content on this page, but NCII cannot guarantee that this information is free from error or reflective of recent changes to the product. Tools and programs have the opportunity to be updated annually or upon request.