Pirate Math Individual Tutoring

Study: Fuchs, Powell, Seethaler, Cirino, Fletcher, et al. (2009)

Fuchs, L.S., Powell, S.R., Seethaler, P.M., Cirino, P.T., Fletcher, J.M., Fuchs, D., Hamlett, C.L., & Zumeta, R.O. (2009). Remediating number combination and word problem deficits among students with mathematics difficulties: A randomized control trial. Journal of Educational Psychology, 101, 561-576.(PMC2768320 NIH136631; PMID 19865600)
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

Pirate Math Individual Tutoring is a tutoring program for remediating the calculation and word-problem deficits and for promoting the algebraic cognition of third-grade students at-risk or with identified math disability. The program is based on schema-broadening instruction, with which students learn to think about word problems in terms of problem types and learn to classify word problems in terms of those problem types, even when the problems incorporate irrelevant information or involve money or incorporate relevant information in pictures, charts, or figures or combine problem types so that problems require 2-step solutions.

Students learn to represent the structure of each type of word problem using an algebraic equation and learn to solve those algebraic equations. The program is organized in terms of 4 units. The first unit teaches skills that are foundational skills to word problems (counting strategies for answering math facts, 2-step procedural calculations, solving algebraic equations, and checking word-problem solutions). Then, each of the next 3 units focuses on one problem type, with systematic cumulative review embedded throughout the units. The three problem types are two problems (where two quantities are combined to make a new amount), difference problems (where two quantities are compared), and change problems (where an event occurs to increase or decrease an initial amount). Sessions occur 3 times per week for 20-30 minutes per sessions across 16 weeks on a one-to-one basis. Tutors can be certified or noncertified teachers, each of whom is trained in a 1-day session and then supervised with bi-weekly meetings. Pirate Math has been shown to improve outcomes on math fact fluency, procedural computational skill, word-problem skill, and algebra skill.

Pirate Math Individual Tutoring is intended for use in third grade. It is designed for use with students with disabilities (including learning disabilities) and any student at risk of academic failure. The academic area of focus is math (computation, concepts, word problems, and algebra).

Pirate Math was tested at 2 sites: one proximal to the developers (Nashville) and the other distal to the developers (Houston), with results showing comparable benefits at both sites (even though Houston students were generally higher performing in math compared to Nashville). In addition, as reflected by Pirate Math sales, Pirate Math has been distributed to 48 school districts located across the country.

Where to obtain:
Lynn Davies
228 Peabody
Vanderbilt University
Nashville, TN 37203
Phone: 615-343-4782

Lynn.a.davies@vanderbilt.edu


Web Site: www.kc.vanderbilt.edu/pals

Cost:
Initial cost per student for implementing program: $40.00 for all students within one school building

Replacement cost per student for subsequent use: None. Worksheets must be copied at local site for use with more than one student.

Each Pirate Math Individual Tutoring manual costs $40. The Pirate Math manual contains 48 lessons. Each lesson has a tutor script (which is reviewed but not read verbatim) and templates for student worksheets. The manual also has templates for math fact flash cards, word-problem sorting cards, graphs to monitor student progress, and treasure maps for motivation. All templates required for implementation are included in the manual. (The only additional materials required are pencils and highlighters).

Order form for tutoring manuals:  http://kc.vanderbilt.edu/pals/pdfs/pirate_math.pdf

Pirate Math Individual Tutoring is designed for use with individual students.

Pirate Math Individual Tutoring takes 20-30 minutes per session with a recommended three sessions per week for 16 weeks.

The program includes a highly specified teacher’s manual. No special technology is required.

Training requirements: One full day of training, plus follow-up by school or district staff with weekly supervision of tutors.

In a one-day training workshop for tutors, (a) an overview of the tutoring program, goals, and topics is presented, and (b) tutoring procedures are modeled and practiced for each activity in the first set of tutoring topics. Following demonstration by the trainer, tutors practice techniques and activities in pairs and receive feedback. Additional consultation with the trainer is available by email or phone following training. Tutors attend bi-weekly meetings to learn about and practice upcoming program topics and to discuss challenges. These meetings are supervised by a building or district instructional support person.

Instructors may be certified teachers or paraprofessionals. Users report high levels of satisfaction with tutoring manuals.

To schedule Pirate Math tutor training, contact Lynn.A.Davies@vanderbilt.edu

 

Participants: Convincing Evidence

Sample size: 133 third grade students across 18 schools. (924 students were screened; 42 Pirate Math, 44 NC Tutoring, 47 Control).

Risk Status: The study was conducted at two sites, both large urban school districts. Houston was distal and Nashville was proximal to the developers of the tutoring protocols. Third-grade students (n=924) were screened for inclusion in 63 classrooms in 18 schools. Seven schools and 23 classrooms were in Houston; 11 schools and 40 classrooms were in Nashville. Because tutoring focused on NCs or on WPs, we included students with low performance on a calculations screening measure or a WP screening measure. (Screening occurred in step-wise fashion; so students did not receive every measure.) The criterion applied for low performance on the calculations measure was < the 26th percentile. The criterion applied to the 5-item word-problem measure was a score of 0 or 1. (See measures for description of the screening measures.) All 924 students were administered the calculations measure; 302 (33%) scored < the 26th percentile. We administered the 5-item WP screener to 598 students; 170 (28%) scored 0 or 1. Of the 598 students who took the calculations and the WP screening measures, 291 (49%) did not meet the inclusion criterion on either measure; 67 (11%) met only the WP criterion; 137 (23%) met only the calculations criterion; and 103 (17%) met both criteria. 

The 307 students who met either or both criteria were eligible for further screening on a reading and an abbreviated IQ measure. We excluded students who scored between the 25th and 40th percentiles in reading and students with a T-score below 30 on both IQ subtests. Students scoring < the 26th percentile on the reading measure were classified as having math and reading difficulty (MDRD). Those scoring > 39th percentile were classified as math difficulty alone (MD). Two hundred and two students took all measures. Of these students, 32 (16%) were excluded due to reading scores between the 25th and 40th percentiles; two students were excluded due to low IQ scores; and one student was excluded for both reasons. Thus, 165 students were eligible for tutoring. However, 162 students comprised the actual assignment sample because three students who met all criteria were accidentally not included in the assignment sample.

Blocking on site, type of screening difficulty (WPs, calculations, or both), and difficulty status (MD or MDRD), we randomly assigned students to one of three treatment conditions (NC tutoring, WP tutoring, or control). So, the composition of each treatment group was similar in terms of the three blocking variables. Of the 162 students, 13 (8%) moved after randomization, but prior to the onset of tutoring, 7 (4%) moved during the school year, 5 (3%) were excluded by parents or schools prior to the onset of tutoring, and 4 (2%) were withdrawn by parents or schools during the school year, leaving 133 who were evaluated at posttest.

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 

 

 

 

 

  Grade 3

42

47

47

53

 

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

 

57

 

70

 

  American Indian

 

0

 

0

 

  Asian/Pacific Islander

 

0

 

0

 

  Hispanic

 

26

 

19

 

  White

 

7

 

9

 

  Other

 

10

 

2

 

Socioeconomic status

  Subsidized lunch

 

76

 

77

 

  No subsidized lunch

 

24

 

23

 

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

17

 

17

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

 

83

 

83

 

ELL status

  English language learner

 

19

 

15

 

  Not English language learner

 

81

 

85

 

Gender

Female

 

45

 

34

 

Male

 

55

 

66

 

Training of Instructors: The tutors were part-time or full-time employees of the research project. In Houston, tutors were not certified teachers, and they were drawn from the community. Each tutor had an undergraduate degree, but the degrees were not necessarily in education-related fields. In Nashville, tutors were graduate students across departments at Vanderbilt University (3 were certified teachers; 9 were not). The graduate students were not necessarily in education programs. Tutors were trained as follows: 1 session of instruction; practice implementing the procedures alone and with each other during the subsequent week; a practice session conducted with a supervisor who provides corrective feedback; tutors studying (not reading) scripts; and meeting among tutors and the supervisor every 2-3 weeks to address problems or questions as they arise.

Design: Convincing Evidence

Did the study use random assignment?: Yes.

If not, was it a tenable quasi-experiment?: Not applicable.

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes.

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? Not applicable.

Were the program and control groups demographically comparable at pretreatment?: Yes.

Was there attrition bias1? No.

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes.

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Every tutoring session was audiotaped. Four research assistants independently listened to tapes while completing a checklist to identify the percentage of essential points in that lesson. We sampled 16.8 of tapes such that treatments, tutors, and lesson types at each site were examined comparably.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: At the site where the protocols had been developed (Nashville), the mean percentage of points addressed was 98.1 (SD=2.06) for number combinations (competing) tutoring and 98.4 (SD=2.79) for WP (Pirate Math) tutoring. In Houston, the mean percentage of points addressed was 99.5 (SD=0.47) for number combinations (competing) tutoring and 99.2 (SD=0.68) for WP (Pirate Math) tutoring.

Tutors also recorded the duration of each session. In Nashville, tutoring min averaged 1032 (SD=85.08) for number combinations (competing) tutoring and 997 (SD=130.25) for WP (Pirate Math) tutoring. In Houston, total tutoring min averaged 1155 (SD=130.09) for number combinations (competing) tutoring and 1158 (SD=184.69) for WP (Pirate Math) tutoring. Analysis of variance revealed a significant effect for site, F(1,82)=15.68, p<.001, with more time in Houston than Nashville. More pertinently, the effect for treatment condition was not significant, F(1,82)=0.18, p=.669; neither was the interaction between treatment and site, F(1,82)=0.27, p =.603.

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Number Combinations

(NC)

Each subtest comprises 25 NCs presented vertically.
Students have 1 min to write answers. The score is the number of correct answers.

Agreement was assessed on 100% of protocols by two independent scorers; alpha was computed on this sample.

Addition Fact Fluency 0-12 comprises addition NCs with sums of 0-12; Subtraction Fact Fluency 0-12, subtraction NCs with minuends of 0-12; Addition Fact Fluency 0-18, addition NCs with sums of 0-18; and Subtraction Fact Fluency 0-18, subtraction NCs with minuends of 0-18.

For the four subtests, respectively, percentage of agreement was 98.9, 98.3, 99.9, and 99.3; alpha was 0.88, 0.91, 0.86 and 0.89

 

Procedural Calculations

To assess procedural calculation learning, the study used two measures. With Double-Digit Mixed Addition and Subtraction in the Grade 3 Math Battery, 3 students have 5 min to complete 20 two-digit addition and subtraction problems with and without regrouping. The score is the number of correct answers.

With Curriculum-Based Measurement–Computation, 4 students have 3 min to complete 25 addition and subtraction items sampling the typical second-grade curriculum. The score is the number of problems correct.

Double-Digit Mixed Addition and Subtraction in the Grade 3 Math Battery:

Agreement, calculated on 100% of protocols by two independent scorers, was 99.3%.
Alpha on this sample was 0.93.

Curriculum-Based Measurement–Computation:

Two independent scorers reentered item-by-item responses into a computerized scoring program on an item-by-item basis. Agreement was 99.7%. Alpha on this sample was 0.95.

 

Find X

With Find X, 5 students solve algebraic equations (a _ b _ c or d _ e _ f) that vary the position of X across all three slots. The tester demonstrates how to find X with a sample problem.

All protocols were independently rescored; agreement was 99.3%. Alpha was 0.93.

 

Number Sentences

With Number Sentences, the tester reads eight WPs aloud; students have 30 s to write the algebraic equation representing the problem model (students do not find solutions). The score is the number of correct equations.

All protocols were independently rescored; agreement was 99.5%. Alpha was 0.84.

 

 

Broader Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Key Math

Key Math– Revised Problem Solving includes 18 WPs of increasing difficulty, which involve all four operations representing taught and untaught problem types. Administration is one to one; items are read aloud; responses are constructed. Testing is discontinued after three consecutive errors.

Split-half reliability at third grade is 0.72. Correlations with the Total Mathematics score of the Iowa at Grades 1–8 is 0.60.

 

Vanderbilt Story Problems

With Vanderbilt Story Problems, 7 students complete 18 novel problems representing the three taught problem types (Total, Difference, and Change relationships with missing information in all three positions), with and without irrelevant information and with and without charts or graphs. In small groups, the tester reads each WP aloud; students have 1 min to write a constructed response. Credit is earned for correct math and labels in answers.

Alpha was 0.86.

 

Iowa Test of Basic Skills, for Problem Solving and Data Interpretation

With the Iowa Test of Basic Skills, for Problem Solving and Data Interpretation students solve 22 WPs representing taught and untaught problem types; data in tables and graphs are required to solve items. The test is administered in small groups, with a multiple-choice response format.

At Grades 1–5, Kuder–Richardson 20 is 0.83–0.87

 

 

Number of Outcome Measures: 6 Math

Mean ES - Targeted: 0.65*

Mean ES - Broader: 0.56*

Effect Size:

Targeted Measures

Construct Measure Effect Size
Math Number Combinations 0.52*
Math Procedural Calculations 0.70**
Math Word Problems: Find X 0.50*
Math Word Problems: Number Sentences 0.87***

 Broader Measures

Construct Measure Effect Size
Math Word Problems: Key Math 0.35
Math Word Problems: Vanderbilt Story Problems 0.77***

 

Key
*      p ≤ 0.05
**    p ≤ 0.01
***  p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: Yes

 

Disaggregated  Outcome Data Available for Students at 20th Percentile or Below

Targeted Measures

Construct Measure Effect Size
Math Number Combinations 0.44
Math Procedural Calculations 0.74*
Math Word Problems: Find X 0.65
Math Word Problems: Number Sentences 0.58

 Broader Measures

Construct Measure Effect Size
Math Word Problems: Key Math 0.35
Math Word Problems: Vanderbilt Story Problems 0.44

 

Key
*      p ≤ 0.05
**    p ≤ 0.01
***  p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Administration Group Size: Individual

Duration of Intervention: 20-30 minutes, 3 times a week, 16 weeks

Minimum Interventionist Requirements: Paraprofessional, 8 hours of training plus weekly follow-up

Reviewed by WWC or E-ESSA: E-ESSA

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

Program Outcomes: One qualifying study of Pirate Math took place in Nashville and Houston. Low-achieving students in Pirate Math were compared to similar control students. On Key Math, students in Pirate Math scored significantly higher than controls, with an effect size of +0.37. This qualified Pirate Math for the ESSA “Strong” category.

Number of Studies: 1

Average Effect Size: 0.37

Full Report

 

Other Research: Potentially Eligible for NCII Review: 3 studies

Fuchs, L.S., Powell, S.R., Seethaler, P.R., Cirino, P.T., Fletcher, J.M., Fuchs, D., & Hamlett, C.L. (2010). The effects of strategic counting instruction, with and without deliberate practice, on number combinations skill among students with mathematics difficulties. Learning and Individual Differences, 20, 89-100. doi:10.1016/j.lindif.2009.09.003
 

Fuchs, L.S., Powell, S.R., Seethaler, P.M., Cirino, P.T., Fletcher, J.M., Fuchs, D., Hamlett, C.L., & Zumeta, R.O. (2009). Remediating number combination and word problem deficits among students with mathematics difficulties: A randomized control trial. Journal of Educational Psychology, 101, 561-576. doi:10.1037/a0014701
 

Fuchs, L.S., Seethaler, P.M., Powell, S.R., Fuchs, D., Hamlett, C.L., & Fletcher, J.M. (2008) Effects of preventative tutoring on the mathematical problem solving of third-grade students with math and reading difficulties. Exceptional Children, 74, 155-173.