NumberShire Level 1

Study: Fien, Doabler, Nelson, Kosty, & Clarke (submitted for publication)

Fien, H., Doabler, C. T., Nelson, N., Kosty, D., & Clarke, B. (submitted for publication). An examination of the promise of the NumberShire Level 1 gaming intervention for improving student mathematics outcomes. Journal of Research on Educational Effectiveness.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

NumberShire Level 1 (NS1) is a browser-based, Tier 2 mathematics intervention designed to support first grade students with or at-risk for mathematics difficulties in developing proficiency with whole numbers concepts and skills. Three mathematics domains represented in the Common Core State Standards for Mathematics (CCSS-M, 2010) are targeted in NS1: (a) Counting and Cardinality, (b) Number and Operations in Base Ten, and (c) Operations and Algebraic Thinking.

NS1 consists of 48 sessions, themed into 12 weeks of instruction (4 sessions per week). Each session is designed to provide 15 minutes of instruction. In total, the intervention offers students 12 hours of individualized, instructional game play. In addition, student mastery data reports are generated that depict student game performance in independent practice activities according to the CCSS-M, and are available to teachers for use in data based decision-making.

NS1 sessions utilize an explicit instructional format and contain three instructional phases: explicit modeling, supported practice, and independent practice. Embedded within each session are four mathematics mini-games, including a Teaching Event (i.e., a mini-lesson targeting a new instructional objective), Assessment Event (i.e., review of a previously mastered objective), Warm-up, and Wrap-up. Mini-games include clear explanations to introduce new material and high quality feedback, and a differentiated learning pathway is used to direct students to additional instruction and practice activities when game performance indicates a need for support. A variety of virtual mathematical representations (e.g., number lines, base-10 blocks) and frequent practice opportunities are employed to facilitate procedural fluency and build a robust and enduring conceptual understanding of whole number concepts.

NumberShire Level 1 is intended for use in grades K-2. The program is intended for use with students with learning disabilities, English language learners, any student at risk of academic failure, or students at risk in mathematics. The academic area of focus is mathematics (including concepts and/or word problems, whole number arithmetic, computation/procedures, problem solving, and mathematical concepts).

Where to obtain:
CTL Marketplace

5292 University of Oregon

Eugene, OR 97403-5292

Phone: 888-497-4290

Website: https://dibels.uoregon.edu/market/numbershire

Cost: $4 per student per year

It is recommended that NumberShire Level 1 is used with individual students or small groups equaling the number of available computers.

NumberShire Level 1 takes 15 minutes per session with a recommended 4 sessions per week for 12 weeks.

The web-based version requires a computer and internet connection. The app-based version requires an iPhone or iPad. Students play independently. Although no teacher-led instruction is required, teachers can utilize data reports to assist in determining a student’s needs and intervene with appropriate instruction as needed. The home-based version can be played on an iPhone/iPad.   

Training is not required for the instructor.

Facilitators are recommended for the program. These facilitators should become familiar with the implementation manual.  The program does not assume that the instructor has expertise in a given area.

Training manuals and materials are available.

Ongoing professional and technical support are available at support@dibels.uoregon.edu and 888-497-4290. In addition, information about NumberShire is available on-line: https://dibels.uoregon.edu/market/numbershire,

 

Participants: Convincing Evidence

Sample size: 238 (117 program, 121 control)

Risk Status: Prior to randomization, all participating first grade students completed a math screening assessment, the EasyCBM-CCSS fall benchmark. Based on performance on this assessment, the 10 lowest students in each class were identified as “NumberShire eligible.” On average, students in the intervention and control groups performed at the 20th percentile on the EasyCBM-CCSS fall benchmark relative to all students screened.

Demographics: Fien et al.

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

125

100% 

125

100% 

 

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

6

5%

10

8%

0.30

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

5

4%

5

4%

0.00

  Hispanic

26

21%

29

23%

0.07

  White

77

62%

76

61%

0.03

  Other

10

8%

5

4%

0.45

Socioeconomic status

  Subsidized lunch

 

 

 

 

 

  No subsidized lunch

 

 

 

 

 

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

11

9%

12

10%

0.07

  Not identified with a disability

114

91%

113

90%

0.07

ELL status

  English language learner

31

25%

28

22%

0.10

  Not English language learner

94

75%

97

78%

0.10

Gender

Female

64

51%

61

49%

0.05

Male

61

49%

64

51%

0.05

Training of Instructors: Interventionists were 9 district-employed instructional assistants and one parent volunteer. The interventionists were predominantly female, and just over half were White, while the remaining were Hispanic or Latino. They had an average of 6.9 years of experience working in schools, and 3.3 years providing interventions for students at risk for mathematics difficulties. The majority did not have formal training in mathematics or education. They had varying levels of experience with technology. One-third of interventionists reported using technology-based math interventions prior to participating in the study.

Design: Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: N/A

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: N/A

Were the program and control groups demographically comparable at pretreatment?: Yes

Was there attrition bias1?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Project staff observed one NumberShire session in each classroom in the final two weeks of implementation. In addition, research staff tracked student progress through game sessions on a weekly basis. Metrics gathered during NS1 gameplay served as an additional measure of fidelity of implementation, including number of sessions completed, number of items completed, and latency and accuracy in responding.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: Project staff directly observed each NS1 intervention group once during the eight-week study using the Technology Observation Tool (TOT: Nelson & Doabler, 2013). The TOT is a researcher-developed, standardized protocol designed to assess fidelity of implementation of the NS1 intervention. Project staff observed and rated each of the intervention session sites (e.g., computer labs) on six items of implementation fidelity: (a) use of effective procedures at start of gameplay, (b) students use of headphones during gameplay, (c) student engagement, (d) active monitoring and classroom management, (e) troubleshooting of technological issues, (f) use of effective procedures at end of gameplay. All items were rated on a 4-point scale (1 = not present, 4 = highly present) and were averaged to compute an overall implementation fidelity score. The average fidelity ratings for interventionists’ use of effective procedures at the start of the session, active monitoring during student gameplay, and use of effective procedures at the conclusion of the session were 3.5 (SD = 1.1), 3.6 (SD = 0.7), and 3.2 (SD = 1.2), respectively. Interventionists also received an average rating of 2.6 (SD = 1.1) for troubleshooting technology issues during sessions. Observers rated students’ engagement during gameplay and use of headphones, a critical component of NS1, as 3.7 (SD = 0.7) and 3.2 (SD = 0.8), respectively. The average overall fidelity score was 3.3 (SD = 0.8), indicating moderate overall fidelity with substantial variability between NS1 groups.

Metrics gathered during NS1 gameplay served as an additional measure of fidelity of implementation, including number of sessions completed, number of items completed, and latency and accuracy in responding. Between pretest and posttest treatment students, on average, completed 18.6 game sessions or 4.7 weeks of game play (SD = 8.1 sessions, range = 2 to 33 sessions) and repeated 12.8 game sessions (SD = 5.9, range = 2 to 24). During game play treatment students completed an average of 499.8 practice opportunities (SD = 269, range = 41 to 1,156) and completed 69% of the practice opportunities correctly (SD = 12%, range = 37% to 90%). Project staff used gameplay metrics to track student progress through game sessions on a weekly basis, and corresponded regularly with interventionists to provide support when needed (e.g., when a student’s gameplay progression deviated from the standard schedule of four sessions per week)

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure Reliability Statistics Relevance to Program Instructional Content Exposure to Related Content Among Control Group

ProFusion-R

Cronbach’s alpha: 0.89 Highly relevant. ProFusion-R assesses the skills taught in the NumberShire Level 1 intervention Given NumberShire’s alignment with CCSS-M, it’s reasonable to expect that the control group was exposed to the skills assessed.

 

Broader Measure Reliability Statistics Relevance to Program Instructional Content Exposure to Related Content Among Control Group

 Group Administered Quantity Discrimination (GA-QD)

Test-retest: 0.87

The quantity discrimination task measures a student’s fluency at evaluating which printed number is greater than another.

Given that whole number concepts, such as magnitude comparisons, are a focus in grade 1 it’s reasonable to expect that the control group was exposed to this skill.

Group Administered Missing Number (GA-MN)

Test-retest: 0.85 The missing number task measures a student’s fluency at determining the number that’s missing from a series of three ordered numbers. Given that whole number concepts, such as strategic counting, are typically a focus in grade 1 it’s reasonable to expect that the control group was exposed to this skill.
easyCBM-NCTM Cronbach's alpha: 0.82 This measure focuses on the NCTM grade 1 focal points of number and operations, geometry, and algebra Given that the measure addresses NCTM grade 1 focal points, it’s reasonable to expect that the control group was exposed to this content.

 

Number of Outcome Measures: 4 Math

Mean ES - Targeted: 0.33*

Mean ES - Broader: 0.18

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Math

Profusion-R

0.30*

Math

2-week interim mastery

0.22

Math

4-week interim mastery

0.08

Math

6-week interim mastery

0.06

 

Broader Measures

Construct

Measure

Effect Size

Math

easyCBM-NCTM total raw

-0.13

Math

Group Administered Quantity Discrimination

0.07

Math

Group Administered Missing Number

0.08

 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

u      Effect size is based on unadjusted means

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Individual, Small Group

Duration of Intervention: 15 minutes, 4 times a week, 12 weeks

Minimum Interventionist Requirements: Facilitators, Training is not required

Reviewed by WWC or E-ESSA: E-ESSA

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

No studies considered met Evidence for ESSA's inclusion requirements.

Other Research: Potentially Eligible for NCII Review: 0 studies