Sign Up For Newsletter

Blueprints For Healthy Youth Development logo

Number Rockets

A school-based program designed to improve children's vocabulary, narrative skills, active listening, and confidence in independent speaking.

Program Outcomes

  • Academic Performance

Program Type

  • Academic Services
  • Mentoring - Tutoring
  • School - Individual Strategies

Program Setting

  • School

Continuum of Intervention

  • Selective Prevention

Age

  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • All

Endorsements

Blueprints: Promising

Program Information Contact

Vanderbilt University
Attn: Lynn Davies / NUMBER ROCKETS
PMB #228
110 Magnolia Circle, Suite MRL 418
Nashville, TN 37203-5721
Phone: (615) 343-4782
Email: lynn.a.davies@vanderbilt.edu
Website: vkc.mc.vanderbilt.edu/numberrockets/

Program Developer/Owner

Lynn S. Fuchs
Vanderbilt University


Brief Description of the Program

Number Rockets is a tutoring intervention for first-grade students identified as at risk for mathematics difficulty. The program is based on the concrete-representational-abstract model, which relies on concrete objects to promote conceptual learning. Tutors deliver the program to small groups of 2-3 students three times per week during the school day in 40-minute sessions, with 30 minutes of scripted mathematics instruction and activities, followed by 10 minutes of practice to build fluency. The 63 program lessons cover 17 topics, and topics include worksheets and manipulatives (e.g., Base-10 blocks for place value instruction). Behavior is monitored, and students earn rewards for demonstrating on-task behavior.

Outcomes

Results of Number Rockets efficacy study 1 (Fuchs et al., 2005) showed:

  • Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in computation, calculation, concepts/applications, and story problems
  • Significantly higher improvement in calculation and concepts/applications for AR tutored students than students identified as not at risk for mathematics difficulty
  • A 40% reduction in prevalence of math disability by the end of first grade for AR tutored students as compared to AR non-tutored students

Results of Number Rockets efficacy study 2 (Fuchs et al., 2013) showed:

  • Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in simple arithmetic, complex calculations, number knowledge, and word problems
  • A narrowing of the achievement gap in simple arithmetic skills between AR tutored students in the speeded practice condition and low-risk controls
  • Improvement in use of retrieval as a strategy by AR tutored students, relative to AR non-tutored controls
  • Reduction in counting errors for AR students in the speeded practice condition, relative to AR non-tutored controls

Results of the effectiveness study (Rolfhus et al., 2012) show that:

  • Students receiving Number Rockets tutoring scored significantly higher (4.28 points) than control group students in mathematics proficiency as assessed by the Test of Early Mathematics Ability - Third Edition (TEMA-3)

Brief Evaluation Methodology

All evaluation studies employed a randomized control trial design. Students in participating schools were screened to determine at-risk status. Students identified as at risk (AR) for mathematics difficulty were randomly assigned to tutoring or control conditions. AR students in the efficacy trials were compared to the control group and to students identified as not at risk (NAR) for mathematics difficulty. In the effectiveness study, the AR intervention group was compared to AR controls. Participating students were tested at baseline and again post-intervention. The original study (Fuchs et al., 2005) included 10 schools, Study 2 (Fuchs et al., 2013) included 40 schools across four cohorts, and Study 3 (Rolfus et al., 2012) included 76 schools in four states. The two efficacy studies included a large number of mathematics measures assessing computation, calculation, concepts, and applications. The primary outcome measure in the effectiveness study (Study 3) was overall math proficiency, as measured by the Test of Early Mathematics Ability (TEMA-3).

Study 1

Fuchs, L. S., Compton, D. L., Fuchs, D., Paulsen, K., Bryant, J. D., & Hamlett, C. L. (2005). The prevention, identification, and cognitive determinants of math difficulty. Journal of Educational Psychology, 97, 493-513.


Study 2

Fuchs, L. S., Geary, D. C., Compton, D. L., Fuchs, D., Schatschneider, C., Hamlett, C. L., . . . Changas, P. (2013). Effects of first-grade number knowledge tutoring with contrasting forms of practice. Journal of Educational Psychology, 105, 58-77.


Risk Factors

School: Poor academic performance*

Protective Factors

School: Instructional Practice


* Risk/Protective Factor was significantly impacted by the program

See also: Number Rockets Logic Model (PDF)

Race/Ethnicity/Gender Details

Evaluations did not parse out results by gender or race/ethnicity. However, the student populations evaluated were from ethnically diverse backgrounds. Two of the three studies had populations that were largely minority (African American and Hispanic), and the majority of main study's population was White(55%), followed by African American (34%).

Tutor training takes place in a one-day workshop at the site (in the district or school). The cost of the training is $1500, plus travel expenses. There is also an additional charge for each tutor attending the training: $64 for implementation manuals and $30 for supplemental materials.

The tutor training workshop provides detailed program information, an overview of background research and theory, trainer modeling, and tutor practice. Trainees also are provided materials highlighting program information and implementation requirements and guidelines. Following is the training agenda: overview of Number Rockets and Responsiveness to Intervention (RTI); walk-through of manuals, materials, and topic sequence; explanation of topic mastery criteria; use of behavior management and point system; use of fidelity checklists; trainer modeling and tutor practice of four representative lesson topics; availability of technical assistance. Tutors practice entire lessons using program manuals, scripts, and supporting materials. The trainer provides instruction and feedback to tutors on techniques, procedures, and lesson content coverage.

Implementers are encouraged to conduct regular meetings to cover upcoming lesson topics and address questions and challenges. The trainer is available by email and phone for technical support following the workshop. Fees are $50 per hour (pro-rated for less than one hour). On-site technical support may be scheduled and is subject to training workshop fees.

Process Monitoring:

Student progress sheets are provided in Number Rockets Tutoring Manual - Book 2. Fidelity checklists are available at no cost from the program developer and trainer. Instructions for using progress sheets and checklists, and for calculating fidelity scores, are provided during the training workshop.

Following is the workshop agenda.

Number Rockets Tutor Training Workshop Agenda

Time

Schedule

Registration and Materials

15 min

8:00 - 8:15

Introduction and Warm-up Exercise

15 min

8:15 - 8:30

Number Rockets Program and RTI Overview

45 min

8:30 - 9:15

Walk-Through of Manuals, Materials, Sequence

Explanation of Mastery Criteria

30 min

9:15 - 9:45

Break

15 min

9:45 - 10:00

Use of Behavior Management and Point System

15 min

10:00 - 10:15

Model and Practice Counting In/Up and Flash Cards

45 min

10:15 - 11:00

Model and Practice Session 1: Topic 4, Day 1

Flash Card Activity

Point System

60 min

11:00 - 12:00

Lunch

60 min

12:00 - 1:00

Model and Practice Session 2: Topic 6, Day 1

Flash Card Activity

Point System

60 min

1:00- 2:00

Break

15 min

2:00 - 2:15

Model and Practice Session 3: Topic 9, Day 1

Flash Card Activity

Point System

60 min

2:15 - 3:15

Model and Practice Session 4: Topic 15, Day 1

Flash Card Activity

Point System

60 min

3:15 - 4:15

Availability of Technical Assistance, Questions, Closure

15 min

4:15 - 4:30

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Start-Up Costs

Initial Training and Technical Assistance

The cost for an initial one-day on-site training is $1500 plus trainer's travel expenses.

Curriculum and Materials

Scripted manuals are required for program implementation. Number Rockets Manual - Book 1 (237 pages) contains lesson topics and detailed scripts for lesson content, feedback, and prompting. Book 2 (353 pages) contains worksheets, posters, review sheets, and other supporting material for each lesson. The cost for Book 1 is $29 and Book 2 $35.

$25 estimated cost per student for photocopies of supporting materials in Book 2.

Supplemental materials are estimated at $30 per tutor. These include Base-10 Blocks units/cubes (20 cubes per student) and rods (10 rods per student), available from school supply stores or online. Estimate also includes one bag of dry beans per tutor for concrete counting activities.

Licensing

$10 fee per tutor each year, including Year 1, to photocopy tutoring manuals and supporting materials.

Other Start-Up Costs

No information is available

Intervention Implementation Costs

Ongoing Curriculum and Materials

$25 per student for photocopies of supporting materials.

Staffing

School-based tutoring may be provided by student instructional support services personnel. If school personnel are not available, salaries for tutors would be needed.

Qualifications: Program generally is delivered by teachers or paraprofessionals who are supervised by teachers. Schools may fill these positions with student instructional support staff, substitute teachers, bachelor's level paraprofessionals, graduate students (typically education/social sciences), or school volunteers. Classroom teachers may implement the program in small groups outside of mathematics instructional time.

Ratios: The program is delivered in small groups of 2-3 students per tutor. A typical scenario may be a school with 4 first-grade classrooms and 4-5 small tutoring groups that are covered by 2 tutors.

Other Implementation Costs

No information is available

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

Initial training, delivered by trainers who are certified by the developer, prepares tutors to implement the program as designed. It is recommended that sites include back-up tutors in the initial training if tutor turnover is anticipated. Training costs would be incurred if new tutors are hired following the initial training workshop for the site.

On-site training or technical support is subject to training workshop fees ($1500 per full day plus trainer's travel expenses). Follow-up phone and email technical support from the trainer is available at $50/hour. Follow-up site visits are also available ($1000 per full day plus trainer's travel expenses).

Fidelity Monitoring and Evaluation

Fidelity monitoring may be completed by school or district personnel. Student progress sheets are provided in Number Rockets Tutoring Manual - Book 2. Fidelity checklists are available at no cost from the program developer and trainer. Instructions for using progress sheets and checklists, and for calculating fidelity scores, are provided during the training workshop.

Ongoing License Fees

A $10 licensing fee allows educators to make additional copies of the manual for up to one year. License expires one year after date of purchase but may be renewed annually.

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

No information is available

Year One Cost Example

In this example, Number Rockets is implemented in a school district with four elementary schools each with four first grade classrooms. Three students in each classroom are identified as at-risk for mathematics difficulty and placed in the program. Two tutors per school (8 in all) serve 48 students district wide. It is assumed that the tutors are part of the school-based instructional staff.

One-day on-site training $1,500.00
Trainer's travel expenses $1,000.00
Implementation manuals @ $64/tutor $512.00
Supplemental materials @ $30/tutor $240.00
Licensing fee to photocopy supporting materials @ $10/tutor $80.00
Photocopies of supporting materials @ $25/student $1,200.00
Total One Year Cost $4,532.00

With 48 students in the Number Rockets program, the Year 1 cost per student is $94.42.

Funding Overview

Number Rockets is a Tier 2 (targeted) intervention within a Response-to-Intervention (RTI) model. It is appropriate for students identified as at-risk and for students with a known math learning disability. Thus, the program may potentially receive funding through federal sources that support RTI.

Maximizing Federal Funds

Formula Funds:

  • The Individuals with Disabilities Education Act (IDEA) 2004 provides funding to school districts for early intervening services (EIS) that assist students who need increased academic and behavioral support to succeed in a general education environment. As such, IDEA 2004 is a potential source of funding for Number Rockets.
  • Title I funds may be used for Number Rockets. Schools with at least a 40 percent poverty rate may choose to operate a schoolwide program that allows Title I funds to be combined with other federal, state and local funds to upgrade the school's overall instructional program. All other schools must operate targeted assistance programs for the most at-risk students; these programs must supplement the regular education program normally provided by state and local educational agencies.

Foundation Grants and Public-Private Partnerships

The National Council of Teachers of Mathematics offers funding to "increase the breadth and depth of mathematics content knowledge" of elementary school teachers committed to mathematics teaching and learning. If a school or district wishes to train teachers in Number Rockets tutoring, this may be a potential source of funding.

President Obama's 2015 budget includes investments to improve science, technology, engineering, and mathematics (STEM) education in K-12 schools, with a priority on excellent teachers, rigorous courses, and regional partnerships that enable school districts to partner with local employers, museums, universities, and others to transform STEM teaching and learning by developing coordinated plans to promote student inspiration, achievement, and preparation in STEM subjects.

Program Developer/Owner

Lynn S. FuchsVanderbilt UniversityDepartment of Special Education228 PeabodyNashville, TN 37203United StatesLynn.Fuchs@vanderbilt.edu vkc.mc.vanderbilt.edu/numberrockets/

Program Outcomes

  • Academic Performance

Program Specifics

Program Type

  • Academic Services
  • Mentoring - Tutoring
  • School - Individual Strategies

Program Setting

  • School

Continuum of Intervention

  • Selective Prevention

Program Goals

A school-based program designed to improve children's vocabulary, narrative skills, active listening, and confidence in independent speaking.

Population Demographics

Number Rockets targets first grade students who are at risk for math difficulty. Participating students were screened and identified as at risk. Evaluations of the program have studied populations of children located in metropolitan and urban areas that are ethnically diverse.

Target Population

Age

  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • All

Race/Ethnicity/Gender Details

Evaluations did not parse out results by gender or race/ethnicity. However, the student populations evaluated were from ethnically diverse backgrounds. Two of the three studies had populations that were largely minority (African American and Hispanic), and the majority of main study's population was White(55%), followed by African American (34%).

Risk/Protective Factor Domain

  • School

Risk/Protective Factors

Risk Factors

School: Poor academic performance*

Protective Factors

School: Instructional Practice


*Risk/Protective Factor was significantly impacted by the program

See also: Number Rockets Logic Model (PDF)

Brief Description of the Program

Number Rockets is a tutoring intervention for first-grade students identified as at risk for mathematics difficulty. The program is based on the concrete-representational-abstract model, which relies on concrete objects to promote conceptual learning. Tutors deliver the program to small groups of 2-3 students three times per week during the school day in 40-minute sessions, with 30 minutes of scripted mathematics instruction and activities, followed by 10 minutes of practice to build fluency. The 63 program lessons cover 17 topics, and topics include worksheets and manipulatives (e.g., Base-10 blocks for place value instruction). Behavior is monitored, and students earn rewards for demonstrating on-task behavior.

Description of the Program

Number Rockets is a tutoring intervention for first-grade students identified as at risk for mathematics difficulty. The program is based on the concrete-representational-abstract model, which relies on concrete objects to promote conceptual learning. Tutors deliver the program to small groups of students during the school day in 40-minute sessions, with 30 minutes of scripted mathematics instruction and activity followed by 10 minutes of practice to build fluency. The 63 lessons cover 17 topics, and lessons include worksheets and manipulatives (e.g., Base-10 blocks for place value instruction). Topics span: identifying and writing numbers; understanding less than, greater than, and equal; sequencing numbers, skip counting; place value; identifying operations; writing number sentences; addition and subtraction facts; 2-digit addition and subtraction; and missing addends. For many topics, if all students in the group demonstrate mastery the group may skip the remaining lessons for that topic and advance to the next topic.

During the final 10 minutes of each tutoring session, students complete drill and practice activities to help develop automatic retrieval of math facts, and students are taught efficient counting strategies as backups to automatic retrieval. Student behavior is monitored throughout each session. At varied intervals, students are awarded points for on-task behavior, and after a pre-determined number of points have been earned, students may trade points for prizes.

Theoretical Orientation

  • Skill Oriented

Brief Evaluation Methodology

All evaluation studies employed a randomized control trial design. Students in participating schools were screened to determine at-risk status. Students identified as at risk (AR) for mathematics difficulty were randomly assigned to tutoring or control conditions. AR students in the efficacy trials were compared to the control group and to students identified as not at risk (NAR) for mathematics difficulty. In the effectiveness study, the AR intervention group was compared to AR controls. Participating students were tested at baseline and again post-intervention. The original study (Fuchs et al., 2005) included 10 schools, Study 2 (Fuchs et al., 2013) included 40 schools across four cohorts, and Study 3 (Rolfus et al., 2012) included 76 schools in four states. The two efficacy studies included a large number of mathematics measures assessing computation, calculation, concepts, and applications. The primary outcome measure in the effectiveness study (Study 3) was overall math proficiency, as measured by the Test of Early Mathematics Ability (TEMA-3).

Outcomes (Brief, over all studies)

Findings from the three evaluation studies show support for the Number Rockets program in elevating the math proficiency levels of students identified as at risk (AR) for math difficulties. When compared to groups of AR control students, AR students who received Number Rockets tutoring posted significant gains in a range of mathematics skills, including calculation, concepts/applications, word problems, computation, number knowledge, simple arithmetic, complex calculations, and overall math proficiency. In addition, in the original study (Fuchs et al., 2005), improvement of AR tutored students on calculation and concepts/applications exceeded improvement of students identified as not at risk for mathematics difficulty.

Outcomes

Results of Number Rockets efficacy study 1 (Fuchs et al., 2005) showed:

  • Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in computation, calculation, concepts/applications, and story problems
  • Significantly higher improvement in calculation and concepts/applications for AR tutored students than students identified as not at risk for mathematics difficulty
  • A 40% reduction in prevalence of math disability by the end of first grade for AR tutored students as compared to AR non-tutored students

Results of Number Rockets efficacy study 2 (Fuchs et al., 2013) showed:

  • Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in simple arithmetic, complex calculations, number knowledge, and word problems
  • A narrowing of the achievement gap in simple arithmetic skills between AR tutored students in the speeded practice condition and low-risk controls
  • Improvement in use of retrieval as a strategy by AR tutored students, relative to AR non-tutored controls
  • Reduction in counting errors for AR students in the speeded practice condition, relative to AR non-tutored controls

Results of the effectiveness study (Rolfhus et al., 2012) show that:

  • Students receiving Number Rockets tutoring scored significantly higher (4.28 points) than control group students in mathematics proficiency as assessed by the Test of Early Mathematics Ability - Third Edition (TEMA-3)

Effect Size

Effect sizes were calculated for each of the three studies. In the main study (Fuchs et al., 2005), multiple effect sizes were calculated. The tutoring program resulted in moderate to large effect sizes for the at-risk (AR) tutored students in comparison to the AR non-tutored controls on several measures: computation ES = .40; calculation ES = .57; story problems ES = .70; concepts/applications ES = .67. In addition, for calculation (ES = .61) and concepts/applications (ES = .45), tutored AR students' improvement exceeded the improvement of students identified as not at risk.

In study 2 (Fuchs et al., 2013), effect sizes varied by intervention condition (speeded vs. non-speeded practice) and by comparison group (non-tutored control group and low-risk group). Effect sizes for both intervention groups when compared to the non-tutored group were stronger on simple arithmetic and complex calculation (ES ranged from .38 to .87), than on number knowledge and word problems (ES ranged from .19 to .29). The speeded practice intervention group exceeded gains demonstrated by the low-risk group in one outcome measure: simple arithmetic (ES = .39). With regard to strategic behavior when solving arithmetic problems, effect sizes for counting were .17 for nonspeeded practice and .28 for speeded practice) to medium (retrieval ES = .42-.52) when compared to the non-tutored control group.

In the effectiveness study (Rolfhus et al., 2012), the effect size on math proficiency gains was slightly lower (.34) than effect sizes reported in the main study (range .40 - .70). This was expected, given the lower level of implementation fidelity typically achieved in a real world setting. In Study 3, average implementation fidelity across districts ad lessons was 85%, whereas in the efficacy studies implementation fidelity was 95% (Study 1) and 96% (Study 2).

These differences indicate that the Number Rockets tutoring program does improve math competency in first-grade students at risk for math difficulty.

Generalizability

The generalizability of this program is limited to first-grade children who are at risk for math difficulty. In all of the evaluations, students were screened to identify their at-risk status. Cut points were defined through the study depending on multiple factors, and the determination of the eligibility for intervention may vary by site. Across studies, sites were heterogeneous and ethnically diverse. he original study (Fuchs et al., 2005) included 10 schools, Study 2 (Fuchs et al., 2013) included 40 schools across four cohorts, and Study 3 (Rolfus et al., 2012) included 76 schools in four states. However, in all three studies schools and districts were targeted and agreed to participate in the studies and therefore may not be representative of the larger population.

Potential Limitations

In the main study (Fuchs et al., 2005), results may be limited on the basis of the pretreatment scores of the at-risk (AR) population being in the low normal range on a pretreatment standardized instrument. Such measures provide an inadequate floor on student performance to identify AR samples in early grades. However, local normative information placed the study's AR students substantially below normal, so this normative information may be considered. Second, the tutoring intervention was an additive to the regular class math instruction. This was necessary to generalize findings to a response-to-intervention model, where tutoring is added to regular class instruction so that students receive two tiers of instruction. Therefore, one cannot draw the conclusion that tutoring can replace regular class instruction.

None of the studies reported follow-up results, so maintenance of program effects is unknown.

Endorsements

Blueprints: Promising

Program Information Contact

Vanderbilt University
Attn: Lynn Davies / NUMBER ROCKETS
PMB #228
110 Magnolia Circle, Suite MRL 418
Nashville, TN 37203-5721
Phone: (615) 343-4782
Email: lynn.a.davies@vanderbilt.edu
Website: vkc.mc.vanderbilt.edu/numberrockets/

References

Study 1

Certified Fuchs, L. S., Compton, D. L., Fuchs, D., Paulsen, K., Bryant, J. D., & Hamlett, C. L. (2005). The prevention, identification, and cognitive determinants of math difficulty. Journal of Educational Psychology, 97, 493-513.

Study 2

Certified Fuchs, L. S., Geary, D. C., Compton, D. L., Fuchs, D., Schatschneider, C., Hamlett, C. L., . . . Changas, P. (2013). Effects of first-grade number knowledge tutoring with contrasting forms of practice. Journal of Educational Psychology, 105, 58-77.

Study 3

Gersten, R., Rolfhus, E., Clarke, B., Decker, L. E., Wilkins, C., & Dimino, J. (2015). Intervention for first graders with limited number knowledge: Large-scale replicaton of a randomized controlled trial. American Educational Research Journal, 52(3), 516-546.

Rolfhus, E., Gersten, R., Clarke, B., Decker, L. E., Wilkins, C., & Dimino, J. (2012). An evaluation of Number Rockets: A tier-2 intervention for grade 1 students at risk for difficulties in mathematics. (NCEE 2012-4007). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Study 1

Evaluation Methodology

Design: Ten schools in a metropolitan school district in the southeastern US participated in this study. Six of the schools were Title I funded. All first grade teachers in these schools and 667 students for whom parental consent was obtained were included. Teachers completed a questionnaire on whole class math instruction with the following results: teachers spent an average of 233 minutes per week on math instruction; lessons consisted of 17% review, 27% instructing on new content, 25% on guiding practice, and 24% on independent practice. There were also about 10 minutes of daily homework assigned. While primary guidance on math skill instruction came from using the district's curriculum standards, other sources of guidance included observations of student performance, classroom assessments, and the district basal. Whole-class instruction was the main method used, followed by small-group instruction, individual instruction, peer tutoring, and cooperative group work.

Students were identified as at-risk (AR) or not-at-risk (NAR) by testing all participating students in a whole-class format on the Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, and CBM Concept/Applications. Testing identified 308 lowest scoring students for individual testing, with an additional 11 nominated by teachers. Individual testing on these 319 children identified 139 as AR. Those students were then randomly assigned to control or tutoring intervention conditions, with blocking by classroom to ensure comparable distribution of AR students in each condition within classrooms. This created 4 pools of students: 1) 69 AR control students, 2) 70 AR tutored students, 3) 180 students who were tested individually then designated NAR, and 4) 348 NAR students only group-tested. Final group numbers (due to attrition) were, respectively: 63, 64, 145, and 292 (groups 3 & 4 were combined into one NAR group).

For the intervention, trained tutors worked with students in groups of two or three students (11 groups of two students, 16 groups of three students). Tutors attended a 2-day training workshop and then received weekly support throughout implementation. Lessons were taught using a scripted manual to promote fidelity. Tutoring sessions were also audiotaped. Each tutoring session lasted 30 minutes, followed by 10 minutes of student use of software called Math Flash, designed to improve automatic retrieval of math facts. There was a sequence of 17 scripted topics completed over 48 sessions (average number of sessions completed by students was 43.8), following the concrete-representational-abstract model. Each session included worksheet and manipulative activities. Each topic took approximately three sessions to complete, and mastery of each topic was assessed daily. Students also completed a cumulative review worksheet of each topic before moving on to the next topic. Student behavior was monitored during each session. Students were awarded points by the tutors for on-task behavior and were able to trade points for prizes. The Math Flash software was designed to provide students with repeated opportunities to hold associations between problem stems and their answers in working memory so the facts could be committed to long-term memory. The program targeted fact families (addition and subtraction facts with answers to 9) that increased in difficulty with mastery. Students were awarded points that translated to "prizes" in a "treasure box" on screen. While efforts were made to minimize the amount of regular math instruction time missed by participating in the tutoring intervention, students missed an average of 10.56 minutes of regular class math instruction.

Sample Characteristics: School characteristics included: teachers with an average of 17.8 years teaching experience and classrooms with an average of 18.32 students. All but one teacher were female and the majority of teachers (34 out of 41) were White (7 were African American). Student characteristics included: generally equal distribution of gender per group; majority of students were White, followed closely by African American, with small percentages of Hispanic and Other. NAR students performed higher on academic and intelligence measures, as expected, with AR students' scores across conditions being comparable.

Measures: Seven math measures were used; five were administered to intact classes and two were administered individually. The whole-class measures included the Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, First-Grade Concepts/Applications, and Story Problems. Scores indicated the number of correct answers on each measure. The two individual measures included the Woodcock-Johnson (WJ) III Applied Problems and WJ III Computation, which also yielded scores of correct answers. Measures were collected at pre- and post-test. In addition, the CBM Computation was administered weekly to whole classes. Reading skill was measured using the Woodcock Reading Mastery Test-Revised. The battery of cognitive assessments conducted in the individual assessments to determine the at risk students (including a teacher rating of attention) are not presented here but are described in the article, and included tests of intelligence, language ability, nonverbal problem solving, phonological processing, processing speed, concept formation, and working memory. Teachers also completed ratings of attention for each AR student. The cognitive variables examined to determine predictors of early math development included attention, language, nonverbal problem solving, phonological processing, processing speed, executive function, and working memory. Analyses were conducted on five of the seven outcome measures (WJ III Applied Problems and Subtraction Fact Fluency were not included because other measures in related domains provided better distributions).

Analysis: One-way ANOVAs were conducted on the pretest, posttest, and improvement scores of the seven math outcome measures, using condition as the factor. Effect sizes were also calculated. Analyses were not intent-to-treat. Multiple regressions were used to determine cognitive variables associated with early math development.

Outcomes

Implementation Fidelity: All tutoring sessions were audiotaped, and program topics 4 and 16 for each tutor were coded and checked for fidelity using checklists corresponding to the lessons. A second coder rechecked fidelity on a random 25% of the tapes, and agreement between coders was 88%. Program implementation fidelity across tutors was 96% for the first topic and 94% for the second topic checked.

Baseline Equivalence and Differential Attrition: There were baseline differences between the not at risk (NAR) and at risk (AR) groups on outcome measures. These differences were expected. There were no significant differences on outcome measures between the two AR conditions. There were a couple of significant differences between groups on the demographic variables. The NAR group had a higher proportion of White students compared to both AR groups, and the AR groups had a larger percentage of students receiving subsidized lunch, compared to students in the NAR group. There was no mention of analysis conducted to determine differences due to attrition between students who left the study (because they moved to a different school) and those who remained.

Posttest: Significant improvements were seen differentially across four of the seven math outcome measures. The greatest effect of the tutoring intervention was seen on the Woodcock Johnson III Calculation and First-Grade Concept/Application measures, where the improvements in AR intervention students exceeded even the students who were not at risk. On story problems, intervention students scored significantly better than the AR control students, although both AR groups were outscored by the NAR students. On the CBM Computation measure, AR students who participated in the intervention saw significant improvements in their scores compared to the AR control students, and their scores were comparable to students in the NAR condition. The status of the conditions on the remaining three measures did not change, with the NAR students performing higher than both AR groups, which scored comparably to each other.

Additionally,based on scores for the Woodcock-Johnson Calculations Test,the researchers identified a 40% reduction in prevalence of math disability at the end of first grade as a result of the math tutoring program (1.77% among non-tutored students vs. 1.06% among tutored students). Among a set of cognitive variables thought to be associated with early math development, the strongest predictor was attention, or distractibility, as rated by teachers. In terms of basic fact fluency, specifically, the unique predictors were attention and phonological processing. For end-of-year computation skill (WJ III Calculation and CBM Computation), attention was the strongest predictor for both, while working memory also predicted CBM Computation. Cognitive variables that predicted math measures in which students worked conceptually with numbers included attention, working memory, and nonverbal problem solving.

Study 2

Evaluation Methodology

Design: Participants were recruited from a metropolitan school district in the Southeast across four cohorts in four consecutive school years. Forty schools with 233 classes of first graders were included, and students were screened to identify subsets of at-risk students. Of the 4,141 eligible students, 2,806 were screened. Students were excluded if they did not speak English (n=359) or if their standard intelligence scores fell below 80 on (n=59). From the remaining pool, 973 students were randomly sampled (648 at-risk and 325 low-risk), stratified by risk status and classroom. The at-risk students were randomly assigned at the individual level while stratified by class to the three conditions. During the year, there was a loss of 57 at-risk students (8.9%) and 25 low-risk students (7.7%) who moved to schools outside of the district.

Participants were randomly assigned to one of three conditions: a speeded practice condition (n=195), a non-speeded practice condition (n=190), and a control condition (206). A second control group consisting of a sample of low-risk, non-tutored classmates, was also included (n=300). The program was referred to as Galaxy Math, which is related to Number Rockets, and used a space theme. In the non-speeded practice condition, students were encouraged to use a variety of number principle strategies including, but not limited to, relying on number lists, arithmetic principles (cardinality, commutativity principle, and subtraction as the inverse of addition), and efficient counting procedures. In the speeded condition, Flash cards were used and efficient counting strategies were encouraged. In this condition, there was also an emphasis on executing the counting strategies quickly, whereas in the non-speeded condition, the focus was on executing the strategies thoughtfully to emphasize number knowledge. For both intervention groups, tutoring occurred for 16 weeks, 3 times per week, in 30-minute sessions. In both conditions, 25 minutes of each session consisted of number knowledge instruction, with the last 5 minutes consisting of practice, but differing by condition as described above. Tutors monitored behavior and rewarded students for displaying on-task behavior with stickers and prizes.

There were 79 tutors who worked with between four and seven students. Tutors attended a 2-day training workshop and then received weekly support throughout implementation. Lessons were taught using a scripted manual to promote fidelity. Tutoring sessions were also audiotaped.

Sample Characteristics: The students across the three at-risk conditions were predominantly African American (67%-73%) and most participated in the subsidized lunch program (80%-87%). Twelve to seventeen percent were identified as having a disability. Scores across the at-risk sample of students on the screening measures (Wide Range Achievement Test-3 (WRAT) and Weschsler Abbreviated Intelligence Scale (WASI-IQ)) were comparable. The students in the low-risk condition differed in ethnicity, where there were more White students then African American, and a lower proportion of students were on the subsidized lunch program (59%). Only 2% of low-risk students had an identified disability, and on screening scores, low-risk children exceeded those in each of the at-risk conditions.

Measures: Measures were collected pre-intervention and post-intervention. On measures of domain-general cognitive resources, the Woodcock Diagnostic Reading Battery-Listening Comprehension was used to measure the respondent's ability to understand sentences or passages; the Wechsler Abbreviated Intelligence Scale Matrix Reasoning scale was used to measure nonverbal reasoning with pattern completion, classification, analogy, and serial reasoning tasks; the Working Memory Test Battery for Children was used to assess the central executive (listening recall, counting recall, and backward digit recall), the phonological loop (digit recall, word list recall, and nonword list recall), and the visuospatial sketchpad (block recall, mazes memory); the Woodcock-Johnson Reading Battery-III Visual Matching was used to measure processing speed; and the Strength and Weaknesses of ADHD-Symptoms and Normal-Behavior Scale was used to measure attentive behavior. Reliability on all measures was .80 or higher. On the measure of strategic arithmetic behavior, the Addition Strategy Assessment was used to measure response times on simple addition problems, as well as classify the method by which the student solved the problem: counting fingers, verbal counting, retrieval, and decomposition. On measures of mathematics performance, the First-Grade Mathematics Assessment Battery, Arithmetic Combinations was used to measure simple arithmetic problems, while the Double-Digit Addition and Subtraction subsets were used to measure more complex calculations; the Number Sets Test was used to measure number knowledge (understanding of number, numeral mapping, and partitioning sets); Word Problems was used to measure responses using simple arithmetic on word problems. Reliability on these measures was .86 or higher.

Analysis: Analysis was intent-to-treat, but the students from the original sample who lacked data because they moved to schools outside of the school district were dropped from the final sample. A two-level residualized change approach was used to analyze effects of study condition: students were nested in classrooms, and pretest performance was used as a covariate along with the main effect of study condition to predict posttest performance. Effect sizes were also calculated.

Outcomes

Implementation Fidelity: Research staff listened to a randomly selected lesson for each tutor in each condition on a weekly basis, and provided corrective feedback as needed. Additionally, about 16% of the tapes were sampled and coded for fidelity, with a 96% coder agreement. The mean percentage of points addressed was 97.6 in speeded practice and 97.7 in nonspeeded practice.

Baseline Equivalence and Differential Attrition: MCAR testing indicated that missing data occurred at random, so students who left the district prior to the end of the study were dropped without introducing significant bias. For the remaining data, there were no missing data. As noted above, students in the low-risk control condition were different than the at-risk children with regard to ethnicity and participation in the subsidized lunch program, and they exceeded the at-risk children on screening scores (reading and math ability, and intelligence), but there were no significant differences among the at-risk study groups.

Posttest: There were significant improvements among the treatment groups across all four mathematics outcomes. Both tutoring conditions exceeded students in the at-risk control condition on simple arithmetic. Students in the speeded tutoring group outperformed the students in the nonspeeded tutoring group, and they also narrowed the achievement gap with respect to low-risk classmates. Similar results were seen with complex calculations; the only difference being the speeded group did not narrow the achievement gap with the low-risk group. However, the achievement gap widened between the low-risk group and the nonspeeded tutoring and the at-risk control groups. Children in both tutoring groups exceeded the at-risk control group on gains in number knowledge and word problems, and results between the tutoring groups were comparable. However, on these two measures, the students across all three at-risk conditions lost ground with respect to the low-risk students.

On measures of children's strategic behavior when solving simple arithmetic problems, there were significant program effects on counting errors and retrieval. For counting errors, the speeded condition saw greater improvements over the at-risk no-tutoring group than the nonspeeded condition, although gains in both tutoring groups were comparable. However, these gains did not outpace the low-risk control group's gains. Gains between the two tutoring conditions with respect to retrieval were comparable, although only the gains among the speeded practice group kept pace with those of the low-risk control group, whereas the other two conditions lost ground.

Mediation analysis showed an effect of retrieval in mediating the effects of tutoring on arithmetic for both the nonspeeded and speeded practice groups. Evaluation of pretest cognitive resources in predicting improvements in simple arithmetic showed an interaction between nonverbal reasoning and condition. Variables shown to be predictors of posttest improvements were attentive behavior and central executive working memory. Furthermore, these predictors were more significant among the students in the nonspeeded practice condition.

Study 3

Evaluation Methodology

Design: Participants were first-grade students at risk for math difficulties from 76 schools in four urban school districts across four of the five Regional Educational Laboratory Southwest states, using school as the unit of assignment. Schools were matched within each district on a composite score calculated from mean school achievement scores and the percentage of students receiving free or reduced price lunch. From each pair, one school was assigned to the intervention condition while the other was assigned to the control condition. Eligible students with parental consent were screened on six measures to determine at-risk status: solving computation problems, concept/application problems, brief story problems, number sense, comparative judgments of numerical magnitude, and working memory. Students were deemed at-risk if their composite score was below the sample's 35th percentile. A total of 994 students (615 intervention, 379 control) met this criteria and participated in the study. Intervention students were assigned to tutoring groups of two to three students and tutoring groups met three or more times per week for approximately 17 weeks for a total of 45 lessons. Program implementation ran from December 2008 to May 2009. All students received regular core mathematics. Eighty-six tutors provided program instruction. All were recruited locally and were primarily retired or substitute teachers and had at least a bachelor's degree. Tutors received a one-day training followed by two 2-hour follow-up trainings. Trained district coaches were also available throughout implementation for supervision and technical assistance. Each tutor had an average of 2.8 tutoring groups.

Sample Characteristics: Study participants were predominantly minority (44% Black, 46% Hispanic). About half (49%) of the students were female and 35% qualified for free/reduced-price lunch. About 8% of students had an IEP.

Measures: The Test of Early Mathematics Ability-Third Edition (TEMA-3) was used to measure student proficiency in math, the primary outcome measure. Math skills tested include numbering, number comparisons, concepts, numeral literacy, number facts, and calculation. Reliability of this measure is alpha = .95. The Woodcock Johnson-III Letter/Word subtest was used to measure reading fluency. Measures were collected at post-intervention. Reliability of this measure is alpha = .98.

Analysis: Analysis was intent-to-treat using multiple imputation. Hierarchical linear modeling was used to determine outcome differences.

Outcomes

Implementation Fidelity
: Fidelity of implementation was measured in three ways. First, lesson fidelity checklists were used, where tutors audiotaped each session and four sessions were randomly evaluated by coding the checklists. Across all districts throughout implementation, average lesson fidelity was 85%. Next, instructional logs were completed, in which tutors reported on lessons completed and length of lesson. Across study districts, an average of 48.1 lessons was completed, with 32.4% of groups completing all 17 program topics. Finally, tutors collected classroom instruction checklists, where teachers recorded the instructional activities that intervention students would miss. This ensured that intervention students did not miss regular math instructional time, as prescribed in implementation guidelines. Checklists indicated that 11.4% of the classroom activities missed by students for program implementation were the whole-class math instruction.

Baseline Equivalence and Differential Attrition: There was a significant difference in the percentage of consent forms returned from the intervention schools compared to the control schools. There were significant differences between the intervention and control conditions with regard to race/ethnicity and for grade 1 enrollment, which was higher in intervention schools. For students who were identified as at-risk, there were no statistically significant differences found between the intervention and control groups on either demographic variables or mean screener composite scores. Posttest data was available for 90% (n=555) of the intervention group and 86% (n=326) of the control group and there were no significant baseline differences between groups for whom posttest data were available.

Posttest: There was a statistically significant difference found on the main outcome variable, TEMA-3 math proficiency scores, in which students in the intervention group scored an average of 4.28 points higher (ES=.34) than their counterparts in the control group. Exploratory analyses findings indicate that 1) there were no differential impacts of the intervention on students based on baseline math proficiency; 2) there were no differential impacts of the intervention on reading proficiency (as measure by the Woodcock Johnson-III Letter/Word subtest) due to students missing classroom reading instruction for participating in the intervention; and 3) there were no differential impacts of the intervention based on dosage, where student test scores on math proficiency were not significantly higher for students who received more sessions of Number Rockets than students who received fewer sessions of the program.

Limitations

Several study limitations were noted. First, researchers were unable to determine whether program outcomes were due specifically to the Number Rockets program or were due to just the additional math instruction time the at-risk students received. Second, parental consent was required for study participation, and there were differential return rates between intervention and control sites. While there were no significant differences between groups on observed demographic characteristics or screener composite scores, there is no way to know whether the differential consent rate influenced baseline equivalence on unobserved characteristics. Additionally, since both parents as well as the school districts volunteered to participate in the study, generalization of results to the larger population should be viewed with caution. Another limitation was the exclusion of Spanish-speaking students due to the materials not being available in Spanish. These students represented between 1-29% of students across all grades. Further, maintenance of effects is unknown, as long-term follow-up results were not examined.

Contact

Blueprints for Healthy Youth Development
University of Colorado Boulder
Institute of Behavioral Science
UCB 483, Boulder, CO 80309

Email: blueprints@colorado.edu

Sign up for Newsletter

If you are interested in staying connected with the work conducted by Blueprints, please share your email to receive quarterly updates.

Blueprints for Healthy Youth Development is
currently funded by Arnold Ventures (formerly the Laura and John Arnold Foundation) and historically has received funding from the Annie E. Casey Foundation and the Office of Juvenile Justice and Delinquency Prevention.