STAR
Math

Summary

Star Math is a computer-adaptive assessment of general mathematics achievement for students in grades 1 to 12. Star Math provides information on student performance on hundreds of skills within 32 domains. Mathematics computation, mathematic application, and mathematics concepts can be assessed. The difficulty of items is adjusted automatically to reflect the skill level of all students.

Where to Obtain:
Renaissance Learning
answers@renaissance.com
Renaissance Learning, PO Box 8036, Wisconsin Rapids, WI 54495
(800) 338-4204
http://www.renaissance.com
Initial Cost:
Contact vendor for pricing details.
Replacement Cost:
Contact vendor for pricing details.
Included in Cost:
There is a one-time setup fee along with a per student subscription fee. Total cost will depend on the number of schools and students. Please contact: answers@renaissance.com or (800) 338-4204 for specific details on pricing for your district. Star Math is cloud-based and purchase includes the tool, software/technical manual, installation guide, testing instructions, and remote installation and setup.
Star Math is a computer-adaptive assessment, and the difficulty of items is adjusted automatically to reflect the skill level of the student. Students can use either the keyboard or the mouse, accommodating students with limited motor skills. Star Math offers several accommodations for students with disabilities through the accessibility options built into a computer’s operating system. For students with limited vision, the introductory screens of Star Math respond to the “high contrast” accessibility feature within Windows and the “switch to black and white” accessibility feature in MAC OS. Star Math is compatible with Mac’s “zoom in” accessibility feature, which allows users to magnify nearly all Star Math screens.
Training Requirements:
Less than one hour of training.
Qualified Administrators:
Access to Technical Support:
Renaissance Technical Support Staff, reachable by phone, email, or chat.
Assessment Format:
  • Individual
  • Small group
  • Large group
  • Computer-administered
Scoring Time:
  • Scoring is automatic OR
  • 0 minutes per student
Scores Generated:
  • Percentile score
  • Grade equivalents
  • IRT-based score
  • Normal curve equivalents
  • Other : Scaled Score
Administration Time:
  • 20 minutes per student
Scoring Method:
  • Automatically (computer-scored)
Technology Requirements:
  • Computer or tablet
  • Internet connection

Tool Information

Descriptive Information

Please provide a description of your tool:
Star Math is a computer-adaptive assessment of general mathematics achievement for students in grades 1 to 12. Star Math provides information on student performance on hundreds of skills within 32 domains. Mathematics computation, mathematic application, and mathematics concepts can be assessed. The difficulty of items is adjusted automatically to reflect the skill level of all students.
Is your tool designed to measure progress towards an end-of-year goal (e.g., oral reading fluency) or progress towards a short-term skill (e.g., letter naming fluency)?
not selected
selected
The tool is intended for use with the following grade(s).
not selected Preschool / Pre - kindergarten
not selected Kindergarten
selected First grade
selected Second grade
selected Third grade
selected Fourth grade
selected Fifth grade
selected Sixth grade
selected Seventh grade
selected Eighth grade
selected Ninth grade
selected Tenth grade
selected Eleventh grade
selected Twelfth grade

The tool is intended for use with the following age(s).
not selected 0-4 years old
not selected 5 years old
not selected 6 years old
not selected 7 years old
not selected 8 years old
not selected 9 years old
not selected 10 years old
not selected 11 years old
not selected 12 years old
not selected 13 years old
not selected 14 years old
not selected 15 years old
not selected 16 years old
not selected 17 years old
not selected 18 years old

The tool is intended for use with the following student populations.
not selected Students in general education
not selected Students with disabilities
not selected English language learners

ACADEMIC ONLY: What dimensions does the tool assess?

Reading
not selected Global Indicator of Reading Competence
not selected Listening Comprehension
not selected Vocabulary
not selected Phonemic Awareness
not selected Decoding
not selected Passage Reading
not selected Word Identification
not selected Comprehension

Spelling & Written Expression
not selected Global Indicator of Spelling Competence
not selected Global Indicator of Writting Expression Competence

Mathematics
selected Global Indicator of Mathematics Comprehension
not selected Early Numeracy
selected Mathematics Concepts
selected Mathematics Computation
selected Mathematics Application
selected Fractions
selected Algebra

Other
Please describe specific domain, skills or subtests:


BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.
BEHAVIOR ONLY: Which category of behaviors does your tool target?

Acquisition and Cost Information

Where to obtain:
Email Address
answers@renaissance.com
Address
Renaissance Learning, PO Box 8036, Wisconsin Rapids, WI 54495
Phone Number
(800) 338-4204
Website
http://www.renaissance.com
Initial cost for implementing program:
Cost
Unit of cost
Replacement cost per unit for subsequent use:
Cost
Unit of cost
Duration of license
Additional cost information:
Describe basic pricing plan and structure of the tool. Provide information on what is included in the published tool, as well as what is not included but required for implementation.
There is a one-time setup fee along with a per student subscription fee. Total cost will depend on the number of schools and students. Please contact: answers@renaissance.com or (800) 338-4204 for specific details on pricing for your district. Star Math is cloud-based and purchase includes the tool, software/technical manual, installation guide, testing instructions, and remote installation and setup.
Provide information about special accommodations for students with disabilities.
Star Math is a computer-adaptive assessment, and the difficulty of items is adjusted automatically to reflect the skill level of the student. Students can use either the keyboard or the mouse, accommodating students with limited motor skills. Star Math offers several accommodations for students with disabilities through the accessibility options built into a computer’s operating system. For students with limited vision, the introductory screens of Star Math respond to the “high contrast” accessibility feature within Windows and the “switch to black and white” accessibility feature in MAC OS. Star Math is compatible with Mac’s “zoom in” accessibility feature, which allows users to magnify nearly all Star Math screens.

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?
not selected
not selected
not selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: What is the administration format?
not selected
not selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: What is the administration setting?
not selected
not selected
not selected
not selected
not selected
not selected
not selected
If other, please specify:

Does the program require technology?

If yes, what technology is required to implement your program? (Select all that apply)
selected
selected
not selected

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?
selected
selected    If small group, n=
selected    If large group, n=
selected
not selected
If other, please specify:

What is the administration time?
Time in minutes
20
per (student/group/other unit)
student

Additional scoring time:
Time in minutes
0
per (student/group/other unit)
student

How many alternate forms are available, if applicable?
Number of alternate forms
per (grade/level/unit)

ACADEMIC ONLY: What are the discontinue rules?
selected
not selected
not selected
not selected
If other, please specify:

BEHAVIOR ONLY: Can multiple students be rated concurrently by one administrator?
If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?
Yes
Describe the time required for administrator training, if applicable:
Less than one hour of training.
Please describe the minimum qualifications an administrator must possess.
not selected No minimum qualifications
Are training manuals and materials available?
Yes
Are training manuals/materials field-tested?
Yes
Are training manuals/materials included in cost of tools?
Yes
If No, please describe training costs:
Can users obtain ongoing professional and technical support?
Yes
If Yes, please describe how users can obtain support:
Renaissance Technical Support Staff, reachable by phone, email, or chat.

Scoring

BEHAVIOR ONLY: What types of scores result from the administration of the assessment?
Score
Observation Behavior Rating
not selected Frequency
not selected Duration
not selected Interval
not selected Latency
not selected Raw score
Conversion
Observation Behavior Rating
not selected Rate
not selected Percent
not selected Standard score
not selected Subscale/ Subtest
not selected Composite
not selected Stanine
not selected Percentile ranks
not selected Normal curve equivalents
not selected IRT based scores
Interpretation
Observation Behavior Rating
not selected Error analysis
not selected Peer comparison
not selected Rate of change
not selected Dev. benchmarks
not selected Age-Grade equivalent
How are scores calculated?
not selected Manually (by hand)
selected Automatically (computer-scored)
not selected Other
If other, please specify:

Do you provide basis for calculating performance level scores?
Yes

What is the basis for calculating performance level and percentile scores?
not selected Age norms
selected Grade norms
not selected Classwide norms
not selected Schoolwide norms
not selected Stanines
not selected Normal curve equivalents

What types of performance level scores are available?
not selected Raw score
not selected Standard score
selected Percentile score
selected Grade equivalents
selected IRT-based score
not selected Age equivalents
not selected Stanines
selected Normal curve equivalents
not selected Developmental benchmarks
not selected Developmental cut points
not selected Equated
not selected Probability
not selected Lexile score
not selected Error analysis
not selected Composite scores
not selected Subscale/subtest scores
selected Other
If other, please specify:
Scaled Score

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.
All scores are calculated automatically by the software. The software calculates a maximum likelihood Rasch ability estimate based on the calibrated difficulty of the items that were administered to the student, and the pattern of the student’s right and wrong responses to those items. Star Math uses a proprietary, Rasch-based, 1-parameter logistic response model to calculate scores. The scaled score is a non-linear, monotonic transformation of the Rasch ability estimate resulting from the adaptive test. From the scaled scores and the student’s current grade placement are derived grade equivalent, percentile, and normal curve equivalent scores. No clusters, composite, or raw scores are reported.
Do you provide basis for calculating slope (e.g., amount of improvement per unit in time)?
Yes
ACADEMIC ONLY: Do you provide benchmarks for the slopes?
Yes
ACADEMIC ONLY: Do you provide percentile ranks for the slopes?
Yes
What is the basis for calculating slope and percentile scores?
not selected Age norms
selected Grade norms
not selected Classwide norms
not selected Schoolwide norms
not selected Stanines
not selected Normal curve equivalents

Describe the tool’s approach to progress monitoring, behavior samples, test format, and/or scoring practices, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.
Each Star Math test contains 34 items in a multiple-choice format. Aligned to state-specific standards and the Common Core State Standards, the test blueprint specifies operational item counts from each of the four domains: Numbers and Operations, Algebra, Geometry and Measurements and Data Analysis, Statistics and Probability. The items are administered to ensure that a balance of content is administered at each grade, appropriate to the typical curriculum for that grade. The assessments’ computer-adaptive structure matches students to items of appropriate difficulty, which in turn may help to reduce student frustration during testing. The assessments can also provide accommodations for students with hearing and visual impairments. The item bank is demographically and contextually balanced. Test blueprint goals are established and tracked to ensure appropriate balance in items addressing use of fiction and nonfiction text, subject and topic areas, geographic region, gender, ethnicity, occupation, age, and disability. Items are free of stereotyping, representing different groups of people in non-stereotypical settings. Items do not refer to inappropriate content that includes, but is not limited to content that presents stereotypes based on ethnicity, gender, culture, economic class, or religion. Items do not present any ethnicity, gender, culture, economic class, or religion unfavorably. Items do not introduce inappropriate information, settings, or situations. All items are subjected to field testing, rigorous psychometric review and related calibration procedures before being included in operational item banks. This skill measurement provides a crucial component in progress monitoring. As students learn new skills, Star Math can assess the level of achievement as often as weekly, although less frequent testing is recommended to avoid over-testing. All items have been calibrated, so that the difficulty of each item is expressed on a Rasch difficulty scale that spans the range of mathematics proficiency from grade 1 through grade 12. Rasch scores are converted to a wide range of scores including Scaled Score, Percentile Rank, Normal Curve Equivalent, etc. Student Growth Percentiles are also reported after a least two tests, to help the teacher understand the student’s velocity relative to academic peers (those in the same grade with a similar score history). For students receiving intensive intervention, the software provides a goal setting tool that presents teachers with several goal options including alignment with state assessment proficiency levels, growth norms (Student Growth Percentile), and score norms (Percentile Rank). Once a goal has been set for a student, the student’s progress on Star Math is graphed against the goal line on the progress monitoring report.

Rates of Improvement and End of Year Benchmarks

Is minimum acceptable growth (slope of improvement or average weekly increase in score by grade level) specified in your manual or published materials?
Yes
If yes, specify the growth standards:
Star Math reports Student Growth Percentile for students with two or more tests. SGP describes a student’s velocity (slope) relative to a national sample of academic peers – those students in the same grade with a similar score history. SGPs work like Percentile Ranks (1-99) but once an SGP goal has been set, it is converted to a Scaled Score goal at the end date specified by the teacher, thus it is converted into an average weekly increase in a Scaled Score metric. We recommend SGP 50 (which indicates typical or expected growth) as minimum acceptable growth, though teachers are of course free to establish their own goals, as they best understand the student and the intensity of services to be provided.
Are benchmarks for minimum acceptable end-of-year performance specified in your manual or published materials?
Yes
If yes, specify the end-of-year performance standards:
Renaissance provides default risk benchmarks at the 10th (urgent intervention), 25th (intervention) and 40th (low risk, lower-bound threshold for working at grade level) Percentile Ranks. Those default cutpoints can be adjusted by educators but are based on expert guidance. For educators wanting to base decisions on likely performance on end-of-year state summative assessments, Renaissance also provides links to those state proficiency categories. Educators select from the default PR benchmarks, the state proficiency projections, or their own custom benchmarks.
What is the basis for specifying minimum acceptable growth and end of year benchmarks?
selected
selected
not selected Other
If other, please specify:
False

If norm-referenced, describe the normative profile.

National representation (check all that apply):
Northeast:
selected New England
selected Middle Atlantic
Midwest:
selected East North Central
selected West North Central
South:
selected South Atlantic
selected East South Central
selected West South Central
West:
selected Mountain
selected Pacific

Local representation (please describe, including number of states)
Data from all 50 states plus the District of Columbia are included in the sample used to develop both growth norms (SGP) and score norms.
Date
Both types of norms updated just prior to the 2017-18 school year. Criterion-referenced benchmarks are based on scale linkages between state summative math tests and Star Math. Completion dates vary by state.
Size
SGP: 5.4 million students. Score norms: 1.9 million students. Criterion-referenced benchmarks (state proficiency indicators): Varies by state.
Gender (Percent)
Male
50
Female
50
Unknown
SES indicators (Percent)
Eligible for free or reduced-price lunch
(FRL data collected at a school level)
Other SES Indicators
Race/Ethnicity (Percent)
White, Non-Hispanic
55.2
Black, Non-Hispanic
Hispanic
American Indian/Alaska Native
Asian/Pacific Islander
Other
Unknown
Disability classification (Please describe)
Not available.

First language (Please describe)
Not available.

Language proficiency status (Please describe)
Not available.
Do you provide, in your user’s manual, norms which are disaggregated by race or ethnicity? If so, for which race/ethnicity?
not selected White, Non-Hispanic
not selected Black, Non-Hispanic
not selected Hispanic
not selected American Indian/Alaska Native
not selected Asian/Pacific Islander
not selected Other
not selected Unknown

If criterion-referenced, describe procedure for specifying criterion for adequate growth and benchmarks for end-of-year performance levels.
For many districts, end-of-year performance on the state summative test is of extreme importance, and they may choose to define intervention cateogires based on state proficiency levels (e.g., students likely to place in the lowest proficiency category may be identified for intenstive intervention). Star Math predicts performance on end-of-year summative mathematics assessments, allowing educators to estimate where students are likely to place as they are being progress monitored throughout the year. This is accomplished by (1) a concurrent scale linking between Star Math and the state’s summative mathematics test scale (and accompanying proficiency cut points), and (2) growth norms that project students forward in time, to the time of the state assessment. Technical reports are available for each state linking study. The process used to project growth is Student Growth Percentile, and information on that approach can be found here. If educators choose to use their state math proficiency categories as their benchmarks, a number of Star Math dashboards and reports specify those benchmarks (equated to the Star Math scale) and the growth necessary to achieve those benchmarks.

Describe any other procedures for specifying adequate growth and minimum acceptable end of year performance.

Performance Level

Reliability

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
*Offer a justification for each type of reliability reported, given the type and purpose of the tool.
The internal consistency reliability coefficient estimates the proportion of variability within a single administration of a test that is due to inconsistency among the items that comprise the test.
*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.
For each grade, a large sample (n = 131,103) of students completed Star Math assessments throughout the 2012–2013 and 2013–2014 school year.
*Describe the analysis procedures for each reported type of reliability.
Reliability was defined as the proportion of test score variance that is attributable to true variation in the trait the test measures, expressed analytically as:1-(σ^2 error)/(σ^2 total). where σ2 error is the variance of the errors of measurement, and σ2 total is the variance of test scores. The variance of the test scores was calculated from Scaled Score data. The variance of the errors of measurement was estimated from the conditional standard error of measurement (CSEM) statistics that accompany each of the IRT-based test scores, including the Scaled Scores. The conditional standard error of measurement (CSEM) was calculated along with the IRT ability estimate and Scaled Score. Squaring and summing the CSEM values yielded an estimate of total squared error; dividing by the number of observations yielded an estimate of error variance. Generic reliability was calculated by subtracting the ratio of error variance to Scaled Score variance from 1.

*In the table(s) below, report the results of the reliability analyses described above (e.g., model-based evidence, internal consistency or inter-rater reliability coefficients). Include detail about the type of reliability data, statistic generated, and sample size and demographic information.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
Yes
Provide citations for additional published studies.
Renaissance Learning (2016). Star Assessments™ for Math Abridged Technical Manual. Wisconsin Rapids, WI: Author. Available by request to research@renaissance.com.
Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
No

If yes, fill in data for each subgroup with disaggregated reliability data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
Provide citations for additional published studies.

Validity

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Partially convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.
All criterion measures were external to the screening tool system and represent widely used assessments of general math ability. • ACT. The American College Testing college readiness assessment is a national standardized test for high school achievement and college admissions. • ACT Aspire: A nationally normed summative test of general mathematics achievement is designed to measure students' progress toward college and career readiness. • CAT-5. The California Achievement Test, is a nationally normed standardized test that measures achievement in mathematics. • IA. Iowa Assessments provide standardized mathematics tests as a service to schools by the College of Education of the University of Iowa. • M-CAP. Pearson’s aimsweb Mathematics Concepts and Applications is a brief, standardized test of problem solving skills and elements included in typical math curriculum. • NWEA MAP. Measures of Academic Progress offers an adaptive computerized test for Mathematics. • PARCC. The Partnership for Assessment of Readiness for College and Careers end-of-year assessment covers mathematics and is intended to be used as an indicator of student needs and progress. • SBA. Smarter Balanced assessments are summative tests designed to measure student achievement and growth in math to support teaching and learning. • SAT. The College Board’s SAT is widely used for college admissions in the United States that covers a range of math practices, including problem solving, modeling, using tools strategically, and using algebraic structure.
*Describe the sample(s), including size and characteristics, for each validity analysis conducted.
Samples included students who had taken both Star Math and the criterion measure. The sample sizes varied across criterion and grades, ranging from 17 to 10,800 students.
*Describe the analysis procedures for each reported type of validity.
Concurrent and predictive correlations were calculated. A criterion assessment was considered concurrent if it was taken during the same school year as the Star Math assessment. The correlation was considered predictive if the criterion assessment was one school year or more after the Star Math assessment.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of validity analysis not compatible with above table format:
Manual cites other published reliability studies:
Yes
Provide citations for additional published studies.
Provide citations for additional published studies. Renaissance Learning (2016). Star Assessments™ for Math Abridged Technical Manual. Wisconsin Rapids, WI: Author. Available by request to research@renaissance.com.
Describe the degree to which the provided data support the validity of the tool.
The data indicate that Star Math results strongly correspond to other various respected measures of general mathematics ability.
Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
No

If yes, fill in data for each subgroup with disaggregated validity data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of validity analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.

Bias Analysis

Grade Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Rating Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.
Yes
If yes,
a. Describe the method used to determine the presence or absence of bias:
Logistic regression analyses conditional on ability, group membership, and ability by group interaction were conducted to assess the presence of both uniform and non-uniform DIF simultaneously. Additionally, an effect size measure – Nagelkerke R-squared – was computed to quantify the magnitude of DIF where present.
b. Describe the subgroups for which bias analyses were conducted:
DIF analyses were conducted for gender (males and females) and race/ethnicity (Caucasian, African American, American Indian, Asian, and Hispanic subpopulations). Due to insufficient samples sizes on English Language Learner (ELLs) and students with disabilities (SWD), DIF analyses for these two subgroups were not possible at the time of the analyses.
c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.
Using a blended criterion that flagged items for uniform/non-uniform DIF if they had a p-value less than 0.01 and Nagelkerke R2 greater than or equal to 0.035, the results indicated that Star Math is sufficiently bias-free. A total of 391 items (4% of the Star Math items) were flagged for DIF. Those flagged items were removed from the item banks for review and recalibration.

Growth Standards

Sensitivity: Reliability of Slope

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence Convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe the sample, including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.
The sample consisted of students who took Star Math tests during the 2016-2017 school year. Sample sizes ranged from 397 to 18,460 students depending on grade.
Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).
A student was included in the sample if they had performance below PR 30; tested at least ten times during the school year; and had 140 days or more between their first and last test.
Describe the analysis procedures.
For each student, tests were divided into two groups – odd and even numbered test depending on the chronological order in which they were taken. For each test group for each student, a slope was estimated by computing an OLS regression coefficient. The table below summarizes Pearson correlation coefficients as a measure of the strength of association between even and odd numbered test slopes.

In the table below, report reliability of the slope (e.g., ratio of true slope variance to total slope variance) by grade level (if relevant).

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
No
Provide citations for additional published studies.
Do you have reliability of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?
No

If yes, fill in data for each subgroup with disaggregated reliability of the slope data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published reliability studies:
Provide citations for additional published studies.

Sensitivity: Validity of Slope

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.
Describe the sample(s), including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.
Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).
Describe the analysis procedures for each reported type of validity.

In the table below, report predictive validity of the slope (correlation between the slope and achievement outcome) by grade level (if relevant).
NOTE: The TRC suggests controlling for initial level when the correlation for slope without such control is not adequate.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published validity studies:
Provide citations for additional published studies.
Describe the degree to which the provided data support the validity of the tool.
Do you have validity of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?

If yes, fill in data for each subgroup with disaggregated validity of the slope data.

Type of Subscale Subgroup Informant Age / Grade Test or Criterion n
(sample/
examinees)
n
(raters)
Median Coefficient 95% Confidence Interval
Lower Bound
95% Confidence Interval
Upper Bound
Results from other forms of reliability analysis not compatible with above table format:
Manual cites other published validity studies:
Provide citations for additional published studies.

Alternate Forms

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
Describe the sample for these analyses, including size and characteristics:
The analysis sample was comprised of students struggling in mathematics whose initial score was at or below the 25th percentile in Star Math. The data were drawn from students’ first and second tests of the year; the testing occasions were close in time to one another.
What is the number of alternate forms of equal and controlled difficulty?
Because Star Math is computer adaptive and comprises several thousand items, there are at a minimum several hundred alternate forms for a student at a given ability level.
If IRT based, provide evidence of item or ability invariance
The table below presents a test-retest analysis correlating scores from students’ first and second Star Math administrations. The results continue to reveal a high degree of consistency between the measures Correlations Between First and Second Administrations of Star Math During the 2016-17 School Year for Students with a Percentile Rank Below 25 Grade n Pearson Coefficient 1 6,274 .64 2 16,880 .61 3 15,188 .61 4 15,585 .64 5 15,808 .65 6 14,769 .67 7 13,013 .64 8 11,552 .63 9 7,469 .62 10 4,901 .60 11 3,687 .61 12 2,499 .65
If computer administered, how many items are in the item bank for each grade level?
Star Math Item Counts by Grade* Grade Items K 37 1 664 2 415 3 941 4 568 5 675 6 598 7 374 8 385 9 701 10 492 11 420 12 7 Total 6,277 *Note that adaptive tests such as Star Math are not constrained to use items from the item bin matching the student’s assigned grade level.
If your tool is computer administered, please note how the test forms are derived instead of providing alternate forms:
Star Math is an instance of computer-administered adaptive testing, based on item response theory. It administers math test items selected one at a time to tailor item difficulty to the student’s own performance as it progresses during each test. The items have all been calibrated previously, using the 1-parameter logistic “Rasch” item response model. After each item is administered, the student’s score is updated, and a target Rasch difficulty level is calculated. The adaptive item selection algorithm seeks an item close to the target difficulty level, consistent with a content blueprint that specifies the number of items to be administered from each of five math content domains and 16 general skills. The adaptive item bank contains more than 6,000 items. The content constraints ensure that each test is parallel to others at the same grade level, in terms of content. The adaptive item selection ensures comparable measurement precision from one test to another. If a student’s performance differs little from one testing occasion to another, the successive tests can be expected to be closely parallel to one another in difficulty and score level.

Decision Rules: Setting & Revising Goals

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence Partially convincing evidence
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
In your manual or published materials, do you specify validated decision rules for how to set and revise goals?
Yes
If yes, specify the decision rules:
The Star Math software provides educators with guidance regarding the establishment of goals for each individual student to inform instructional decisions. There is not a single recommended goal for each grade level; instead, the goal setting approach is unique to each student and takes into consideration his/her prior achievement trajectory. The default goal for each individual student is based on national growth norms, as discussed elsewhere in the submission. Growth norms are used to understand typical and above typical rates of growth for a student based on comparisons with academic peers – those students in the same grade with a similar score history. Educators are encouraged to select either a typical rate of growth (one in which the student has about a 50% chance of attaining) or a more aggressive rate of growth (one in which the student has about a 35% chance of attaining. Educators may opt to select a different goal, but are not encouraged to select a goal below the 50% threshold, as students in intensive intervention are typically trying to catch up to grade level peers. Throughout the course of the intervention and progress monitoring, educators are encouraged to check student progress. Empirical research on Star decision rules (Cormier & Bulut, 2017, manuscript submitted for publication) has determined that decisions can safely be made after a minimum of five administrations. So, after at least five administrations, educators review the Star Math Progress Monitoring report. If actual student progress is at or above the student’s growth norm-based goal line, educators are encouraged to continue the intervention or revise the goal upward. If actual student progress is below the student’s growth norm-based goal line, educators are encouraged to alter the intervention and supports the student is receiving. A sample Star Progress Monitoring report is available from the Center upon request.
What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.
A quasi-experimental study was conducted involving more than 3.5 million students who began the year struggling in mathematics (defined as having a Star Math National Percentile Rank score below 30). Students were classified as treatment if they used the Star Math goal setting recommendations, and control if they did not. All students in the study (both treatment and comparison) were intensively progress monitored during the 2014-15 school year on an approximately weekly basis. (The duration of progress monitoring varied, with decisions up to local educators and unique to each student.) All students’ beginning (fall) and end of year (spring) Star Math Normal Curve Equivalent (NCE) scores were compared (posttest minus pretest), and are summarized by grade and condition in the table below. In every grade, 1-12, treatment students (using the Star Math goal recommendations) experienced significantly greater NCE change than their control counterparts who did not use the goal recommendations, providing support for the claim that using Star Math’s evidence-based goal recommendations and decision rules is superior to other locally determined approaches. Within each grade, the differences in NCE gains favoring treatment students were significant at p < .001. The results of this study suggest that students experience greater growth when educators follow the individualized goal and decision rule guidance provided by Star Math. The guidance is based on a combination of expert guidance and growth norms drawn from a large, diverse sample of students. A table detailing the results of this study is available from the Center upon request.

Decision Rules: Changing Instruction

Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11
Rating Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable Data unavailable
Legend
Full BubbleConvincing evidence
Half BubblePartially convincing evidence
Empty BubbleUnconvincing evidence
Null BubbleData unavailable
dDisaggregated data available
In your manual or published materials, do you specify validated decision rules for when changes to instruction need to be made?
If yes, specify the decision rules:
What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.

Disclaimer

Most tools and programs evaluated by the NCII are branded products which have been submitted by the companies, organizations, or individuals that disseminate these products. These entities supply the textual information shown above, but not the ratings accompanying the text. NCII administrators and members of our Technical Review Committees have reviewed the content on this page, but NCII cannot guarantee that this information is free from error or reflective of recent changes to the product. Tools and programs have the opportunity to be updated annually or upon request.