Pirate Math Individual Tutoring
Study: Fuchs et al. (2009)

Describe how students were selected to participate in the study:

The study was conducted at two sites, both large urban school districts. Houston was distal and Nashville was proximal to the developers of the tutoring protocols. Third-grade students (n=924) were screened for inclusion in 63 classrooms in 18 schools. Seven schools and 23 classrooms were in Houston; 11 schools and 40 classrooms were in Nashville. Because tutoring focused on NCs or on WPs, we included students with low performance on a calculations screening measure or a WP screening measure. (Screening occurred in step-wise fashion; so students did not receive every measure.) The criterion applied for low performance on the calculations measure was < the 26th percentile. The criterion applied to the 5-item word-problem measure was a score of 0 or 1. (See measures for description of the screening measures.) All 924 students were administered the calculations measure; 302 (33%) scored < the 26th percentile. We administered the 5-item WP screener to 598 students; 170 (28%) scored 0 or 1. Of the 598 students who took the calculations and the WP screening measures, 291 (49%) did not meet the inclusion criterion on either measure; 67 (11%) met only the WP criterion; 137 (23%) met only the calculations criterion; and 103 (17%) met both criteria. The 307 students who met either or both criteria were eligible for further screening on a reading and an abbreviated IQ measure. We excluded students who scored between the 25th and 40th percentiles in reading and students with a T-score below 30 on both IQ subtests. Students scoring < the 26th percentile on the reading measure were classified as having math and reading difficulty (MDRD). Those scoring > 39th percentile were classified as math difficulty alone (MD). Two hundred and two students took all measures. Of these students, 32 (16%) were excluded due to reading scores between the 25th and 40th percentiles; two students were excluded due to low IQ scores; and one student was excluded for both reasons. Thus, 165 students were eligible for tutoring. However, 162 students comprised the actual assignment sample because three students who met all criteria were accidentally not included in the assignment sample. Blocking on site, type of screening difficulty (WPs, calculations, or both), and difficulty status (MD or MDRD), we randomly assigned students to one of three treatment conditions (NC tutoring, WP tutoring, or control). So, the composition of each treatment group was similar in terms of the three blocking variables. Of the 162 students, 13 (8%) moved after randomization but prior to the onset of tutoring, 7 (4%) moved during the school year, 5 (3%) were excluded by parents or schools prior to the onset of tutoring, and 4 (2%) were withdrawn by parents or schools during the school year, leaving 133 who were evaluated at posttest.

Describe how students were identified as being at risk for academic failure (AI) or as having emotional or behavioral difficulties (BI):

The study was conducted at two sites, both large urban school districts. Houston was distal and Nashville was proximal to the developers of the tutoring protocols. Third-grade students (n=924) were screened for inclusion in 63 classrooms in 18 schools. Seven schools and 23 classrooms were in Houston; 11 schools and 40 classrooms were in Nashville. Because tutoring focused on NCs or on WPs, we included students with low performance on a calculations screening measure or a WP screening measure. (Screening occurred in step-wise fashion; so students did not receive every measure.) The criterion applied for low performance on the calculations measure was < the 26th percentile. The criterion applied to the 5-item word-problem measure was a score of 0 or 1. (See measures for description of the screening measures.) All 924 students were administered the calculations measure; 302 (33%) scored < the 26th percentile. We administered the 5-item WP screener to 598 students; 170 (28%) scored 0 or 1. Of the 598 students who took the calculations and the WP screening measures, 291 (49%) did not meet the inclusion criterion on either measure; 67 (11%) met only the WP criterion; 137 (23%) met only the calculations criterion; and 103 (17%) met both criteria. The 307 students who met either or both criteria were eligible for further screening on a reading and an abbreviated IQ measure. We excluded students who scored between the 25th and 40th percentiles in reading and students with a T-score below 30 on both IQ subtests. Students scoring < the 26th percentile on the reading measure were classified as having math and reading difficulty (MDRD). Those scoring > 39th percentile were classified as math difficulty alone (MD). Two hundred and two students took all measures. Of these students, 32 (16%) were excluded due to reading scores between the 25th and 40th percentiles; two students were excluded due to low IQ scores; and one student was excluded for both reasons. Thus, 165 students were eligible for tutoring. However, 162 students comprised the actual assignment sample because three students who met all criteria were accidentally not included in the assignment sample. Blocking on site, type of screening difficulty (WPs, calculations, or both), and difficulty status (MD or MDRD), we randomly assigned students to one of three treatment conditions (NC tutoring, WP tutoring, or control). So, the composition of each treatment group was similar in terms of the three blocking variables. Of the 162 students, 13 (8%) moved after randomization, but prior to the onset of tutoring, 7 (4%) moved during the school year, 5 (3%) were excluded by parents or schools prior to the onset of tutoring, and 4 (2%) were withdrawn by parents or schools during the school year, leaving 133 who were evaluated at posttest.

ACADEMIC INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• below the 30th percentile on local or national norm, or
• identified disability related to the focus of the intervention?

%

BEHAVIORAL INTERVENTION: What percentage of participants were at risk, as measured by one or more of the following criteria:

• emotional disability label,
• placed in an alternative school/classroom,
• non-responsive to Tiers 1 and 2, or
• designation of severe problem behaviors on a validated scale or through observation?

%

Specify which condition is the submitted intervention:

WP tutoring or Pirate Math (referred to both ways in study)

Specify which condition is the control condition:

There was a no-tutoring (Tier 1 only) condition, which was the business-as-usual condition, as well as a competing condition, which was an active tutoring condition devoted entirely to number combination (i.e., math facts). This competing condition controlled for tutoring time and for the motivational component of Pirate Math.

If you have a third, competing condition, in addition to your control and intervention condition, identify what the competing condition is (data from this competing condition will not be used):

Using the tables that follow, provide data demonstrating comparability of the program group and control group in terms of demographics.

Grade Level

Race–Ethnicity

Socioeconomic Status

Disability Status

ELL Status

Gender

What method was used to determine students' placement in treatment/control groups?

Random

Please describe the assignment method or the process for defining treatment/comparison groups.

Blocking on site, type of screening difficulty (WPs, calculations, or both), and difficulty status (MD or MDRD), we randomly assigned students to one of three treatment conditions (NC tutoring, WP tutoring, or control).

What was the unit of assignment?

Students

If other, please specify:

Please describe the unit of assignment:

What unit(s) were used for primary data analysis?

Schools
Teachers
Students
Classes
Other
If other, please specify:

Please describe the unit(s) used for primary data analysis:

How was the program delivered?

Individually
Small Group
Classroom

What were the background, experience, training, and ongoing support of the instructors or interventionists?

The tutors were part-time or full-time employees of the research project. In Houston, tutors were not certified teachers, and they were drawn from the community. Each tutor had an undergraduate degree, but the degrees were not necessarily in education-related fields. In Nashville, tutors were graduate students across departments at Vanderbilt University (3 were certified teachers; 9 were not). The graduate students were not necessarily in education programs. Tutors were trained as follows: 1 session of instruction; practice implementing the procedures alone and with each other during the subsequent week; a practice session conducted with a supervisor who provides corrective feedback; tutors studying (not reading) scripts; and meeting among tutors and the supervisor every 2-3 weeks to address problems or questions as they arise.

Describe when and how fidelity of treatment information was obtained.

Every tutoring session was audiotaped. Four research assistants independently listened to tapes while completing a checklist to identify the percentage of essential points in that lesson. We sampled 16.8 of tapes such that treatments, tutors, and lesson types at each site were examined comparably.

What were the results on the fidelity-of-treatment implementation measure?

At the site where the protocols had been developed (Nashville), the mean percentage of points addressed was 98.1 (SD=2.06) for number combinations (competing) tutoring and 98.4 (SD=2.79) for WP (Pirate Math) tutoring. In Houston, the mean percentage of points addressed was 99.5 (SD=0.47) for number combinations (competing) tutoring and 99.2 (SD=0.68) for WP (Pirate Math) tutoring. Tutors also recorded the duration of each session. In Nashville, tutoring min averaged 1032 (SD=85.08) for number combinations (competing) tutoring and 997 (SD=130.25) for WP (Pirate Math) tutoring. In Houston, total tutoring min averaged 1155 (SD=130.09) for number combinations (competing) tutoring and 1158 (SD=184.69) for WP (Pirate Math) tutoring. Analysis of variance revealed a significant effect for site, F(1,82)=15.68, p<.001, with more time in Houston than Nashville. More pertinently, the effect for treatment condition was not significant, F(1,82)=0.18, p=.669; neither was the interaction between treatment and site, F(1,82)=0.27, p =.603.

Was the fidelity measure also used in control classrooms?

For any substantively (e.g., effect size ≥ 0.25 for pretest or demographic differences) or statistically significant (e.g., p < 0.05) pretest differences, please describe the extent to which these differences are related to the impact of the treatment. For example, if analyses were conducted to determine that outcomes from this study are due to the intervention and not pretest characteristics, please describe the results of those analyses here.

Please explain any missing data or instances of measures with incomplete pre- or post-test data.

If you have excluded a variable or data that are reported in the study being submitted, explain the rationale for exclusion:

Describe the analyses used to determine whether the intervention produced changes in student outcomes:

We conducted distributional exploration of each measure via statistical (e.g., skewness, kurtosis) and graphical (e.g., box plots, stem and leaf plots) means. Generally, the standardized variables were normally distributed at both time points. However, Vanderbilt Story Problems was positively skewed and kurtotic at both time points, and Find X was negatively skewed at pretest and bimodal at posttest. In the case of Vanderbilt Story Problems, a square-root transformation was completed, which improved distribution; however, because results for the original and the transformed variables were similar, results of the original variable are presented for ease of interpretation. Similarly, for Find X, logistic analyses (dichotomizing scores into high and low) yielded similar results as the original variables; so we retained the original form. Number Sentences at pretest and procedural computations at posttest also showed some skewness, but transformations did not generally improve distributions. Age was unrelated to pre- or posttest performance. Also, tutoring time was unrelated to all but two outcomes, and in this case, the relation was small and did not interact with other effects or change conclusions. Therefore, these variables are not reported. The factors of interest were difficulty status (MD vs. MDRD), site (Houston vs. Nashville), and tutoring condition (number combinations, i.e., competing condition, tutoring vs. WP, i.e., Pirate Math, tutoring vs. control).