Star

Mathematics

Cost

Technology, Human Resources, and Accommodations for Special Needs

Service and Support

Purpose and Other Implementation Information

Usage and Reporting

Initial Cost:

No information provided; contact vendor for details.

 

Replacement Cost:

No information provided; contact vendor for details.

 

Included in Cost:

There is a one-time setup fee along with a per student subscription fee. Total cost will depend on the number of schools and students.
Please contact: 
answers@renaissance.com or (800) 338-4204 for specific details on pricing for your district. Star Math is cloud-based, and purchase includes the tool, software/technical manual, installation guide, testing instructions, and remote installation and setup.

Technology Requirements:

  • Computer or tablet
  • Internet connection

 

Training Requirements:

  • Less than 1 hour of training

 

Qualified Administrators:

  • No minimum qualifications specified

 

Accommodations:

Star Math is a computer-adaptive assessment, and the difficulty of items is adjusted automatically to reflect the skill level of the student. Students can use either the keyboard or the mouse, accommodating students with limited motor skills. Star Math offers several accommodations for students with disabilities through the accessibility options built into a computer’s operating system. For students with limited vision, the introductory screens of Star Math respond to the “high contrast” accessibility feature within Windows and the “switch to black and white” accessibility feature in MAC OS. Star Math is compatible with Mac’s “zoom in” accessibility feature, which allows users to magnify nearly all Star Math screens.

Where to Obtain:

Website:

http://www.renaissance.com

Address:

Renaissance Learning, PO Box 8036, Wisconsin Rapids, WI 54495

Phone:
(800) 338-4204

Email: answers@renaissance.com


Access to Technical Support:

Renaissance Technical Support Staff are reachable by phone, email, or chat.

Star Math is a computer-adaptive assessment of general mathematics achievement for students in grades 1 to 12. Star Math provides information on student performance on hundreds of skills within 32 domains. Mathematics computation, mathematic application, and mathematics concepts can be assessed. The difficulty of items is adjusted automatically to reflect the skill level of all students.

Assessment Format:

  • Individual
  • Group
  • Computer-administered

 

Administration Time:

  • 20 minutes per student
  • 20 minutes per group

 

Scoring Time:

  • Scoring is automatic

 

Scoring Method:

  • Calculated automatically

 

Scores Generated:

  • Percentile Score
  • IRT-Based Score
  • Grade Equivalents
  • Normal Curve Equivalents
  • Scaled Score

 

 

 

Reliability

Grade1234567891011
RatingFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubble

Justify the appropriateness of each type of reliability reported:

The internal consistency reliability coefficient estimates the proportion of variability within a single administration of a test that is due to inconsistency among the items that comprise the test.

 

Describe the sample characteristics for each reliability analysis conducted:

For each grade, a large sample (n = 131,103) of students completed Star Math assessments throughout the 2012–2013 and 2013–2014 school year.

 

Describe the analysis procedures for each reported type of reliability:

Reliability was defined as the proportion of test score variance that is attributable to true variation in the trait the test measures. The variance of the test scores was calculated from Scaled Score data. The variance of the errors of measurement was estimated from the conditional standard error of measurement (CSEM) statistics that accompany each of the IRT-based test scores, including the Scaled Scores. The conditional standard error of measurement (CSEM) was calculated along with the IRT ability estimate and Scaled Score. Squaring and summing the CSEM values yielded an estimate of total squared error; dividing by the number of observations yielded an estimate of error variance. Generic reliability was calculated by subtracting the ratio of error variance to Scaled Score variance from 1.

 

Type of Reliability

Age or Grade

n

Coefficient

Confidence Interval

Generic

Grade 1

131,103

.90

.90-.90

Generic

Grade 2

131,103

.91

.91-.91

Generic

Grade 3

131,103

.91

.91-.91

Generic

Grade 4

131,103

.92

.92-.92

Generic

Grade 5

131,103

.92

.92-.92

Generic

Grade 6

131,103

.93

.93-.93

Generic

Grade 7

131,103

.93

.93-.93

Generic

Grade 8

131,103

.93

.93-.93

Generic

Grade 9

131,103

.93

.93-.93

Generic

Grade 10

131,103

.94

.94-.94

Generic

Grade 11

131,103

.94

.94-.94

Generic

Grade 12

131,103

.95

.95-.95

 

Validity

Grade1234567891011
RatingFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubble

Describe and justify the criterion measures used to demonstrate validity:

All criterion measures were external to the screening tool system and represent widely used assessments of general math ability.

  • CAT-5. The California Achievement Test, is a nationally normed standardized test that measures achievement in mathematics.
  • NWEA MAP. Measures of Academic Progress offers an adaptive computerized test for Mathematics.
  • PARCC. The Partnership for Assessment of Readiness for College and Careers end-of-year assessment covers mathematics and is intended to be used as an indicator of student needs and progress.
  • SBA. Smarter Balanced assessments are summative tests designed to measure student achievement and growth in math to support teaching and learning.
  • ACT. The American College Testing college readiness assessment is a national standardized test for high school achievement and college admissions.
  • IA. Iowa Assessments provide standardized mathematics tests as a service to schools by the College of Education of the University of Iowa.
  • SAT. The SAT Math Test is a standardized test widely used for college admissions in the United States that covers a range of math practices, including problem solving, modeling, using tools strategically, and using algebraic structure.
  • M-CAP. The aimsweb Mathematics Concepts and Applications is a brief, standardized test of problem solving skills and elements included in typical math curriculum.

 

Describe the sample characteristics for each validity analysis conducted:

Samples included students who had taken both Star Math and the criterion measure. The sample sizes varied across criterion and grades, ranging from 17 to 10,800 students.

 

Describe the analysis procedures for each reported type of validity:

Concurrent and predictive correlations were calculated. A criterion assessment was considered concurrent if it was taken during the same school year as the Star Math assessment. The correlation was considered predictive if the criterion assessment was one school year or more after the Star Math assessment.

 

Type of Validity

Age or Grade

Test or Criterion

n

Coefficient

Confidence Interval

Concurrent

Grade 1

California Achievement Test-5 (CAT-5)

105

0.74

0.64-0.81

Concurrent

Grade 1

Northwest Evaluation Association, Measures of Academic Progress (NWEA MAP)

230

0.74

0.68-0.79

Predictive

Grade 1

ACT Aspire (Grade 3)

3000

0.70

0.68-0.72

Concurrent

Grade 2

NWEA MAP

702

0.79

0.76-0.82

Concurrent

Grade 2

Iowa Assessments (IA)

279

0.81

0.77-0.85

Predictive

Grade 2

ACT Aspire (Grade 3)

3713

0.79

0.78-0.80

Concurrent

Grade 3

The Partnership for Assessment of Readiness for College and Careers (PARCC)

4103

0.80

0.79-0.81

Concurrent

Grade 3

Smarter Balanced Assessments (SBA)

10800

0.84

0.83-0.85

Predictive

Grade 3

SBA

17898

0.67

0.66-0.68

Concurrent

Grade 4

PARCC

4787

0.83

0.82-0.84

Concurrent

Grade 4

SBA

10582

0.86

0.85-0.86

Predictive

Grade 4

SBA

8571

0.88

0.87-0.89

Concurrent

Grade 5

PARCC

4266

0.79

0.78-0.80

Concurrent

Grade 5

SBA

9750

0.86

0.85-0.86

Predictive

Grade 5

SBA

8595

0.88

0.87-0.89

Concurrent

Grade 6

PARCC

5050

0.80

0.79-0.81

Concurrent

Grade 6

SBA

7852

0.86

0.85-0.86

Predictive

Grade 6

SBA

8575

0.88

0.87-0.89

Concurrent

Grade 7

PARCC

4368

0.77

0.85-0.87

Concurrent

Grade 7

SBA

6344

0.86

0.85-0.86

Predictive

Grade 7

SBA

4066

0.80

0.79-0.81

Concurrent

Grade 8

PARCC

5424

0.83

0.82-0.84

Concurrent

Grade 8

SBA

4196

0.75

0.82-0.84

Predictive

Grade 8

SBA

3748

0.76

0.74-0.77

Concurrent

Grade 9

Mathematics Concepts and Applications (M-CAP)

68

0.81

0.71-0.88

Predictive

Grade 9

SAT (Grade 11)

928

0.76

0.73-0.79

Predictive

Grade 10

SBA (Grade 11)

2262

0.78

0.76-0.80

Predictive

Grade 10

SAT (Grade 11)

17

0.89

0.72-0.96

Concurrent

Grade 11

SBA

3685

0.72

0.71-0.74

Predictive

Grade 11

American College Test (ACT)

7246

0.72

0.71-0.73

 

Describe the degree to which the provided data support the validity of the tool:

The data indicate that Star Math results strongly correspond to other various respected measures of general mathematics ability.

Bias Analysis Conducted

Grade1234567891011
RatingYesYesYesYesYesYesYesYesYesYesYes

Have additional analyses been conducted to establish whether the tool is or is not biased against demographic subgroups (e.g., students who vary by race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?

Bias Analysis Method:

Logistic regression analyses conditional on ability, group membership, and ability by group interaction were conducted to assess the presence of both uniform and non-uniform DIF simultaneously. Additionally, an effect size measure – Nagelkerke R-squared – was computed to quantify the magnitude of DIF where present.

 

Subgroups Included:

DIF analyses were conducted for gender (males and females) and race/ethnicity (Caucasian, African American, American Indian, Asian, and Hispanic subpopulations). Due to insufficient samples sizes on English Language Learners (ELLs) and students with disabilities (SWD), DIF analyses for these two subgroups were not possible at the time of the analyses. 

 

Bias Analysis Results:

Using a blended criterion that flagged items for uniform/non-uniform DIF if they had a p-value less than 0.01 and Nagelkerke R2 greater than or equal to 0.035, the results indicated that Star Math is sufficiently bias-free. A total of 391 items (4% of the Star Math items) were flagged for DIF. Those flagged items were removed from the item banks for review and recalibration.

Sensitivity: Reliability of the Slope

Grade1234567891011
RatingFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubbleFull bubble

Describe the sample used for analyses, including size and characteristics:

The sample consisted of students who took Star Math tests during the 2016-2017 school year. A student was included in the sample if they had performance below PR 30. Sample sizes ranged from 397 to 18,460 students depending on grade.

 

Describe the frequency of measurement:

All included students were tested at least ten times during the school year and had 140 days or more between their first and last test.

 

Describe reliability of the slope analyses conducted with a population of students in need of intensive intervention:

For each student, tests were divided into two groups – odd and even numbered tests, depending on the chronological order in which they were taken.  For each test group and for each student, a slope was estimated by computing an OLS regression coefficient. The table below summarizes Pearson correlation coefficients as a measure of the strength of association between even and odd numbered test slopes.

 

Type of Reliability

Age or Grade

n

Coefficient

Confidence Interval

Pearson correlation

Grade 1

8,987

0.92

0.92 - 0.92

Pearson correlation

Grade 2

18,460

0.93

0.93 - 0.93

Pearson correlation

Grade 3

16,696

0.93

0.92 - 0.93

Pearson correlation

Grade 4

14,738

0.93

0.92 - 0.93

Pearson correlation

Grade 5

12,411

0.93

0.93 - 0.93

Pearson correlation

Grade 6

8,627

0.94

0.93 - 0.94

Pearson correlation

Grade 7

6,379

0.93

0.93 - 0.93

Pearson correlation

Grade 8

5,317

0.93

0.93 - 0.94

Pearson correlation

Grade 9

2,129

0.94

0.94 - 0.95

Pearson correlation

Grade 10

1,265

0.94

0.93 - 0.94

Pearson correlation

Grade 11

803

0.94

0.93 – 0.95

Pearson correlation

Grade 12

397

0.94

0.93 - 0.95

 

Sensitivity: Validity of the Slope

Grade1234567891011
Ratingdashdashdashdashdashdashdashdashdashdashdash

Describe and justify the criterion measures used to demonstrate validity:

No qualifying evidence provided.

 

Describe the sample used for analyses, including size and characteristics:

No qualifying evidence provided.

 

Describe predictive validity of the slope of improvement analyses conducted with a population of students in need of intensive intervention:

No qualifying evidence provided.

 

Describe the degree to which the provided data support the validity of the tool:

No qualifying evidence provided.

Alternate Forms

Grade1234567891011
RatingHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubble

Describe the sample for these analyses, including size and characteristics:

The analysis sample was comprised of students struggling in mathematics whose initial score was at or below the 25th percentile in Star Math. The data were drawn from students’ first and second tests of the year; the testing occasions were close in time to one another.

 

Evidence that alternate forms are of equal and controlled difficulty or, if IRT based, evidence of item or ability invariance:

Star Math is an instance of computer-administered adaptive testing, based on item response theory. It administers math test items selected one at a time to tailor item difficulty to the student’s own performance as it progresses during each test.  The items have all been calibrated previously, using the 1-parameter logistic “Rasch” item response model.  After each item is administered, the student’s score is updated, and a target Rasch difficulty level is calculated. The adaptive item selection algorithm seeks an item close to the target difficulty level, consistent with a content blueprint that specifies the number of items to be administered from each of five math content domains and 16 general skills.  The adaptive item bank contains more than 6,000 items. The content constraints ensure that each test is parallel to others at the same grade level, in terms of content. The adaptive item selection ensures comparable measurement precision from one test to another.  If a student’s performance differs little from one testing occasion to another, the successive tests can be expected to be closely parallel to one another in difficulty and score level.

The table below presents a test-retest analysis correlating scores from students’ first and second Star Math administrations.

Correlations Between First and Second Administrations of Star Math During the 2016-17 School Year for Students with a Percentile Rank Below 25

Grade

n

Pearson Correlation Coefficient

1

6,274

.64

2

16,880

.61

3

15,188

.61

4

15,585

.64

5

15,808

.65

6

14,769

.67

7

13,013

.64

8

11,552

.63

9

7,469

.62

10

4,901

.60

11

3,687

.61

12

2,499

.65

 

Number of alternate forms of equal and controlled difficulty:

Because Star Math is computer adaptive and comprises several thousand items, there are at a minimum several hundred alternate forms for a student at a given ability level. See table below for item counts by grade level.

Star Math Item Counts by Grade*

Grade

Item Count

K

37

1

664

2

415

3

941

4

568

5

675

6

598

7

374

8

385

9

701

10

492

11

420

12

7

Total

6,277

*Note that adaptive tests such as Star Math are not constrained to use items from the item bin matching the student’s assigned grade level. 

Decision Rules: Setting and Revising Goals

Grade1234567891011
RatingHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubbleHalf-filled bubble

Specification of validated decision rules for when goals should be set or revised:

Star Math software provides educators with guidance regarding the establishment of goals for each individual student to inform instructional decisions. There is not a single recommended goal for each grade level; instead, the goal setting approach is unique to each student and takes into consideration his/her prior achievement trajectory. The default goal for each individual student is based on national growth norms, as discussed elsewhere in the submission. Growth norms are used to understand typical and above typical rates of growth for a student based on comparisons with academic peers – those students in the same grade with a similar score history. Educators are encouraged to select either a typical rate of growth (one in which the student has about a 50% chance of attaining) or a more aggressive rate of growth (one in which the student has about a 35% chance of attaining. Educators may opt to select a different goal, but are not encouraged to select a goal below the 50% threshold, as students in intensive intervention are typically trying to catch up to grade level peers.

Throughout the course of the intervention and progress monitoring, educators are encouraged to check student progress. Empirical research on Star decision rules (Cormier & Bulut, 2017, manuscript submitted for publication) has determined that decisions can safely be made after a minimum of five administrations. So, after at least five administrations, educators review the Star Math Progress Monitoring report. If actual student progress is at or above the student’s growth norm-based goal line, educators are encouraged to continue the intervention or revise the goal upward. If actual student progress is below the student’s growth norm-based goal line, educators are encouraged to alter the intervention and supports the student is receiving.

 

Evidentiary basis for these rules:

A quasi-experimental study was conducted involving more than 3.5 million students who began the year struggling in mathematics (defined as having a Star Math National Percentile Rank score below 30). Students were classified as treatment if they used the Star Math goal setting recommendations, and control if they did not. All students in the study (both treatment and comparison) were intensively progress monitored during the 2014-15 school year on an approximately weekly basis. (The duration of progress monitoring varied, with decisions up to local educators and unique to each student.) All students’ beginning (fall) and end of year (spring) Star Math Normal Curve Equivalent (NCE) scores were compared (posttest minus pretest), and are summarized by grade and condition in the table below. In every grade, 1-12, treatment students (using the Star Math goal recommendations) experienced significantly greater NCE change than their control counterparts who did not use the goal recommendations, providing support for the claim that using Star Math’s evidence-based goal recommendations and decision rules is superior to other locally determined approaches. Within each grade, the differences in NCE gains favoring treatment students were significant at p < .001. The results of this study suggest that students experience greater growth when educators follow the individualized goal and decision rule guidance provided by Star Math. The guidance is based on a combination of expert guidance and growth norms drawn from a large, diverse sample of students.

Change in Fall-Spring NCE Scores for Struggling Students Who Used and Did Not Use the Personalized Star Math Goal Setting Recommendations

Grade

Comparison: Not Using Star Math Recommended Goals or Decision Rules

Treatment: Using Star Math Recommended Goals and Decision Rules

Cohen’s d

n

Mean NCE Change (post-pre)

SD

n

Mean NCE Change (post-pre)

SD

1

310,501

+3.05

15.05

5,126

+15.45

17.66

0.76

2

469,609

+6.01

14.93

13,838

+12.49

16.68

0.41

3

485,690

+1.09

14.73

10,469

+10.40

15.43

0.62

4

471,672

+3.38

13.98

10,494

+9.58

15.36

0.42

5

451,281

+1.97

13.37

10,403

+7.01

14.10

0.37

6

358,718

+0.36

12.50

8,448

+5.35

13.14

0.39

7

305,995

+0.47

11.73

7,315

+5.22

12.80

0.39

8

281,941

-0.09

11.10

6,552

+4.80

12.08

0.42

9

121,238

+0.23

11.54

2,148

+4.01

12.90

0.31

10

86,874

-0.40

11.70

1,204

+5.43

13.17

0.47

11

54,627

+1.00

12.57

784

+6.73

13.90

0.43

12

28,406

-0.08

13.99

396

+5.34

15.62

0.37

* All treatment-control differences within grade significant at p < 0.001.

Decision Rules: Changing Instruction

Grade1234567891011
Ratingdashdashdashdashdashdashdashdashdashdashdash

Specification of validated decision rules for when changes to instruction should be made:

No qualifying evidence provided.

 

Evidentiary basis for these rules:

No qualifying evidence provided.

Administration Format

Grade1234567891011
Data
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Individual
  • Group
  • Computer-administered
  • Administration Format:

    Individual

    Group

    Computer-administered

    Administration & Scoring Time

    Grade1234567891011
    Data
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • 20 minutes
  • Administration Time:

    20 minutes

    Scoring Time:

    Not applicable

    Scoring Format

    Grade1234567891011
    Data
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Computer-scored
  • Scoring Format:

    Computer-scored

    ROI & EOY Benchmarks

    Grade1234567891011
    Data
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • ROI & EOY Benchmarks Available
  • Specify the minimum acceptable rate of growth/improvement:

    Star Math reports Student Growth Percentile for students with two or more tests. SGP describes a student’s velocity (slope) relative to a national sample of academic peers – those students in the same grade with a similar score history. SGPs work like Percentile Ranks (1-99) but once an SGP goal has been set, it is converted to a Scaled Score goal at the end date specified by the teacher, thus it is converted into an average weekly increase in a Scaled Score metric. Renaissance recommends SGP 50 (which indicates typical or expected growth) as minimum acceptable growth, though teachers are of course free to establish their own goals, as they best understand the student and the intensity of services to be provided.

     

    Specify the benchmarks for minimum acceptable end-of-year performance:

    Renaissance provides default risk benchmarks at the 10th (urgent intervention), 25th (intervention) and 40th (low risk, lower-bound threshold for working at grade level) Percentile Ranks. Those default cutpoints can be adjusted by educators but are based on expert guidance. For educators wanting to base decisions on likely performance on end-of-year state summative assessments, Renaissance also provides links to those state proficiency categories. Educators select from the default PR benchmarks, the state proficiency projections, or their own custom benchmarks.