Reflex

Study: Rudel (2016)

Rudel, D. I., (n.d.). Effect of Reflex on Math Fact Fluency in Grades 2 & 3. Unpublished technical report. Retrieve from: https://el-gizmos.s3.amazonaws.com/docs/Florida_RCT_2016.pdf and https://el-gizmos.s3.amazonaws.com/docs/Florida_RCT_2016_Addendum.pdf

Descriptive Information

Usage

Acquisition and Cost

Program Specifications and Requirements

Training

Reflex is a game-based adaptive system for math fact acquisition. Individual students login using separate accounts and receive assessment, instruction, and practice tailored to their particular needs based on an internal student model. There is an in-product indicator confirming recommended daily use. Model usage is 3 days a week until students are 100% fluent. Time required to complete individual recommended usage varies by age. For students in grade 2, the median time is approximately 16 minutes a day. The product is intended for use by students in grades 2+. It uses a Fact Family approach that leverages the relationships between addition/subtraction and multiplication/division.

Reflex is intended for students in grades 2-8, particularly students with learning disabilities, intellectual disabilities, English language learners, and any student with basic understanding of numeracy.

Where to obtain:

110 Avon Street

Charlottesville, VA 22902

Phone Number: 866-882-4141

Website:  www.reflexmath.com

Cost: $3,295 per school per year

Reflex is available in annual licenses that range from per-student teacher licenses, site licenses, or district licenses. All licenses come bundled with professional development. Multi-year licenses are also available. Reflex also has a grant program that teachers can apply to to get Reflex free for 1 year.

Recommended usage for a second or third grade student would be 3 times a week for 15-20 minutes a day for the bulk of the school year. Students in 4th or 5th grade typically require less time.

Anticipated usage is for students to use Reflex until they reach 100% fluency, at which point the intervention is complete.

To use Reflex students must have access to the internet and a computer.

Instructors may be teachers or support staff. The program does not assume that the instructor has expertise in a given area.

A free 90-minute training is offered but not required to use Reflex since it is delivered individually straight to the students. the training does equip teachers or other personnel to know how to get the best results and understand the reporting and admin features of their teacher accounts.

Users can undertake annual professional development, and there is also a live customer service division that teachers can call, email, or chat with.

 

Participants: Unconvincing Evidence

Sample size: 130 second- and third-grade students. The sample was comprised of 66 program students and 64 control students.

Risk Status: Students who had fewer than 10 digits correct per minute at pretest (across 3 1-minute probes) were identified as at-risk for failure in mathematics. This is the threshold for M-CBM screening cutoff provided by the National Institute on Response to Intervention (Fuchs and Fuchs). PLEASE NOTE: The school did not provide data on at-risk students or percentiles on normed achievement tests, so we are only using the above criterion when reporting percentage of at-risk students in the population. However, given the large percentage of traditionally under-served populations in the study, we expect the percentage of students who would be deemed "at-risk" based on achievement test percentiles would be far higher than the percentage tendered below. We are providing subpopulation analyses on Hispanic, LEP, and SES students to show the effect of Reflex on these populations.

Demographics:

 

Program Number

Program Percentage

Control Number

Control Percentage

Cox Index

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

34

52

33

52

0.00

  Grade 3

31

47

31

48

0.02

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

2

3

4

6

0.44

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

10

15

12

19

0.17

  Hispanic

36

55

34

53

0.05

  White

15

28

14

22

0.03

  Other

2

3

0

0

2.08

Socioeconomic status

  Subsidized lunch

12

18

14

22

0.15

  No subsidized lunch

43

65

40

63

0.05

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Mental retardation

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

 

 

 

 

 

ELL status

  English language learner

10

15

16

25

0.39

  Not English language learner

 

 

 

 

 

Gender

Female

39

59

30

47

0.29

Male

26

39

34

53

0.34

 

Training of Instructors: 4 of the 7 teachers had master's degrees (3 classes in treatment, 1 class in comparison). The other 3 teachers had bachelor's degrees. The number of years of teaching experience among treatment classes were {21, 13, 2, "15 or more"}. The number of years of teaching experience across comparison groups were {5, 15 or more, 15 or more, 15 or more}. (We relied on testimony from the math coach for those teachers who did not return information on background, and several teachers were at the school prior to her being there, so "15 or more" was all the information we could get for those teachers.) All teachers in the program condition received training on using Reflex.

Design: Convincing Evidence

Did the study use random assignment? Yes

If not, was it a tenable quasi-experiment? NA

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes? Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? NA

Were the program and control groups demographically comparable at pretreatment? No but it was addressed in HLM analysis. 

Was there attrition bias1? No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)? No but it  was addressed in HLM analysis. 

 

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Fidelity of Implementation: Unconvincing Evidence

Describe when and how fidelity of treatment information was obtained: All work in Reflex occurs within the Reflex system, so session logs were consulted to derive information on usage.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: One class had poor usage in general (average 1.5 days/week), one class had reasonable usage (2.7 days/week), and two had good usage (3.3 and 3.4 days/week). However, these averages can be misleading since some students elected to play outside of school and had many days of usage, skewing the average. Overall, approximately 30% of the students in the study had poor usage (<1.6 days/week), 15% had reasonable-to-good usage (1.7-2.5 days/week) and the rest had near-model to model usage. (>2.5 days/week).

Measures Targeted: Convincing Evidence

Measures Broader: Data Unavailable

Targeted Measure

 

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

M-CBM math Fact probes (Similar to AIMSweb math fact probes)

 

M-CBM has been independently validated by a number of research designs (see page 11 of full text). The raw data in this study (prior to Box-Cox normalization) showed Cronbach Alphas of 0.94-

0.97

Reflex is an intervention aimed at developing math fact fluency, which is measured using M-CBM math fact probes.

Classes in the control group also worked on developing math fact fluency, based on surveys returned by comparison group teachers. It was a straight business-as-usual comparison group.

 

Broader Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

None

 

 

 

 

Number of Outcome Measures: 1 Math

Mean ES - Targeted: 0.84*

Mean ES - Broader: Data Unavailable

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Math

M-CBM math Fact probes

0.84***

 

Broader Measures

Construct

Measure

Effect Size

 

None

 

 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

u      Effect size is based on unadjusted means

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: Yes

Targeted Measures

Construct

Measure

Effect Size

Math

M-CBM math Fact probes – ELLs

0.75

Math

M-CBM math Fact probes – Free or reduced lunch

0.88*

Math

M-CBM math Fact probes – Hispanic Students

0.52*

 

Broader Measures

Construct

Measure

Effect Size

 

None

 

 

Key

*        p ≤ .05

**      p ≤ .01

***    p ≤ .001

–      Developer was unable to provide necessary data for NCII to calculate effect sizes

u      Effect size is based on unadjusted means

†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Disaggregated Data for <20th Percentile: No

Administration Group Size: Individual

Duration of Intervention: 10 minutes, 2 times a week, Until effective

Minimum Interventionist Requirements: Professional, Training is not required

Reviewed by WWC or E-ESSA: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

Other Research: Potentially Eligible for NCII Review: 1 study

Rudel, David (2018) Evaluating Effect of Reflex on Math Fact Fluency in Grades 2 & 3. Retrieved from https://elgizmos.s3.amazonaws.com/docs/Florida_RCT_2016.pdf

Rudel, David (2018) Additional Subpopulation Analysis. Retrieved from https://elgizmos.s3.amazonaws.com/docs/Florida_RCT_2016_Addendum.pdf