Sign Up For Newsletter

Blueprints For Healthy Youth Development logo

Peer Assisted Learning Strategies

A class-wide peer tutoring program for elementary school students to improve reading and mathematics skills of students through guided peer-assisted learning strategies.

Program Outcomes

  • Academic Performance

Program Type

  • Early Childhood Education
  • Peer Counseling and Mediation
  • School - Individual Strategies

Program Setting

  • School

Continuum of Intervention

  • Universal Prevention

Age

  • Early Adolescence (12-14) - Middle School
  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • Hispanic or Latino
  • All

Endorsements

Blueprints: Promising

Program Information Contact

Douglas Fuchs and Lynn Fuchs
Peabody College, Vanderbilt University
Peabody Box 328
110 Magnolia Circle, Suite 418
Nashville, TN 37203
Phone 615-343-4782
doug.fuchs@vanderbilt.edu
lynn.a.davies@vanderbilt.edu
vkc.mc.vanderbilt.edu/frg/

Program Developer/Owner

Douglas Fuchs
Peabody College, Vanderbilt University


Brief Description of the Program

Peer Assisted Learning Strategies (PALS) for children in kindergarten through sixth grade is a type of class-wide peer tutoring that is used to improve reading and math skills. Teachers pair low and high performing students, and the partners work on different activities that address the skills that are causing problems. The pairs are changed regularly, giving all students the opportunity to act as coaches and players. PALS enables teachers to address individual student needs, as well as observe students and develop individual remedial lessons. It is a complementary strategy that teachers can use to augment their existing reading and math curricula. PALS is composed of 25-35 minute activities that are implemented 2-4 times a week for 14 to 31 weeks.

Outcomes

Primary findings:

  • Compared to a control group, PALS (or its hybrid versions) significantly improved reading and mathematical skills of school age (including kindergarten) students.
  • PALS was most effective among low achieving and average achieving students.

Secondary findings:

  • Improved versions of PALS (PALS-HG, PMI-Elaborated + Conceptual) and hybrid programs that combine PALS with other educational interventions (e.g., Ladders+PALS, LST+PALS) significantly enhanced reading and mathematic skills.
  • PALS effectively improved reading skills among Hispanic children of varying degrees of English proficiency.
  • PALS significantly improved reading and math skills among students with identified learning disabilities.

Brief Evaluation Methodology

The majority of studies employed a randomized control trial procedure in which teachers (classrooms) were randomly assigned to treatment and control groups (e.g., Fuchs et al., 1999; Fuchs, Fuchs & Karns, 2001; Fuchs, Fuchs, Karns et al., 2001; Fuchs, Fuchs, Thompson et al., 2001; Fuchs et al., 2002; Mathes & Babyak, 2001; Calhoon & Fuchs, 2003; Calhoon, 2005; Calhoon et al., 2006, 2007; Stein et al., 2008). However, three additional studies employed a quasi-experimental design in which a non-random procedure was used to match control and intervention group participants (e.g., Mathes et al., 1998; Mathes, Torgesen & Allor, 2001; Mathes et al., 2003). The selection of the students for assessment was based on results from standardized test-scores and/or teacher's judgment (e.g., Fuchs et al., 1999; Fuchs, Fuchs, Karns et al., 2001; Fuchs, Fuchs, Thompson et al., 2001; Mathes & Babyak, 2001; Mathes, Torgesen & Allor, 2001; Stein et al., 2008). Studies investigated the effect of variations of PALS, including program enhancements (e.g., PALS with elaborated helping strategies, Fuchs et al., 1999), "hybrid" programs such as Ladders + PALS (Fuchs et al., 2001) and Linguistic Skills Training (LST) + PALS (Calhoon, 2005), skill-based mini-lessons (ML) + PALS (Mathes and Babyak, 2001), increasingly individualized treatments for program nonresponders (McMaster et al., 2005), as well as mediator effects such as differing levels of teacher assistance (Stein et al., 2008). In addition, a number of studies used PALS to improve student's mathematical achievments (Fuchs, Fuchs, Phillips et al., 1995; Fuchs, Fuchs & Karns 2001, Fuchs et al., 2002; Calhoon & Fuchs, 2003). Sample sizes varied widely and included between 4 and 140 teachers from 2 to 49 different schools. In most studies only a subset of students from each classroom was assessed (for each category [LA, AA, HA] 1 to 6 children) (see Calhoon & Fuchs, 2003; Fuchs et al., 2002; Calhoon et al., 2006, 2007 for exceptions). Studies assessed program effects only at posttest, between 15 and 20 weeks after pretest. One laudable exception is Fuchs et al. (2001) who assessed changes in treatment effects 5 months after posttest. The program has been implemented in geographically (e.g., Nashville, Tennessee [majority of studies], Minnesota, Southern Texas) and socio-economically diverse school settings (Title 1 and non-Title 1 schools).

Study 1

Fuchs, L. S., Fuchs, D., Kazdan, S., & Allen, S. (1999). Effects of peer-assisted learning strategies in reading with and without training in elaborated help giving. The Elementary School Journal, 99(3), 201-219.


Risk Factors

School: Poor academic performance

Protective Factors

School: Instructional Practice, Opportunities for prosocial involvement in education


* Risk/Protective Factor was significantly impacted by the program

Race/Ethnicity Specific Findings
  • Hispanic or Latino
Race/Ethnicity/Gender Details

Usually program effectiveness was assessed among students regardless of gender or racial composition of the classroom. However, program effectiveness was demonstrated also among Hispanic students enrolled in English only and dual language educational programs (Calhoon et al. 2006, Calhoon et al. 2007).

PALS Workshop Audience: Teachers, special educators, curriculum instruction specialists, school psychologists, teacher trainers, principals.

PALS Workshop Cost: The presenter's fee is estimated at $1500 plus travel expenses for a 1-day, 6-hour workshop. All workshop participants will need to purchase a PALS teacher's manual prior to the training workshop (price range $40-$45).

Sample PALS Workshop Agenda:

  1. PALS Background: Research and Development
  2. Learning About PALS
  3. Demonstration of PALS Activities
  4. Practice in PALS Activities
  5. Putting It All Together
  6. Implementation Issues

PALS Reading Workshop

Time: One-Day PALS Reading workshops are scheduled from 8:30 a.m. - 3:00 p.m.

Description:

K-PALS and First-Grade PALS promote phonological awareness, letter-sound correspondence, decoding, and fluency for beginning readers. PALS Reading for Grades 2-6 promotes reading fluency and use of comprehension strategies. In the PALS Reading workshop, participants learn how to conduct peer-mediated reading instruction. They learn how to pair students and prepare them to work together constructively on strategic reading tasks that accelerate reading performance.

Following a brief summary of research findings, presenters provide hands-on practice of all PALS activities including a simulation of a PALS classroom during training, teacher-directed lessons, and peer-mediated practice. Participants receive student practice materials that include samples of the activity worksheets. Presenters role-play as PALS teachers, and participants practice PALS as classroom students.

PALS Math Workshop

Time: 6 Hours (K-1 is 3 hours; 2-6 is 3 hours)

Description:

Participants learn how to implement PALS Math, an activity that blends the components of effective group work into a structured peer-mediated approach. PALS Math helps teachers meet the needs of diverse students and allows students to assume an active role in their own learning.

Following a brief presentation summarizing research findings, presenters provide hands-on practice of all PALS activities. Participants learn how to pair students and prepare them to work together constructively on appropriate mathematics tasks. Participants receive an introduction to the various mathematics concepts for PALS Math, and they learn the roles of coach and player for each PALS Math lesson. Participants receive a sample PALS Math folder that will enable them to confidently conduct PALS in their own classrooms.

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Start-Up Costs

Initial Training and Technical Assistance

Training is offered to groups of up to 50 teachers for one day for $1,500 plus travel.

Curriculum and Materials

Each teacher should have a manual for reading and/or math. Manuals cost between $40 and $44 each. Optional DVDs with examples of interventions covering multiple grades for both reading and math are available for $15. Math PALS for grades 2-6 require Student Materials at a cost of $30 for a class (although this cost is discretionary, as it is suggested that students bring in their favorite books, or the teacher may select from books in his or her library). Inexpensive supplies such as transparencies, kitchen timers, overhead projectors, and some copying are also required.

Materials Available in Other Language: A Spanish language version of PALS Reading Grades 2 - 6 for teachers and students is available at the same cost as the English language version.

Licensing

None.

Other Start-Up Costs

None.

Intervention Implementation Costs

Ongoing Curriculum and Materials

Copying of individual student lessons is required for some grades at a nominal cost.

Staffing

The program is implemented by teachers during the course of the regular school day. No additional staff is required.

Other Implementation Costs

None.

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

None.

Fidelity Monitoring and Evaluation

None.

Ongoing License Fees

None.

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

None.

Year One Cost Example

This example covers the initial implementation of the program in two elementary schools for grades K-5. It assumes two classes per grade in each school involving 12 teachers per school for a total of 24 teachers. Each teacher implements both reading and math PALS.

Teacher Training: Reading grades K-1 -- 8 teachers $1,500.00
Teacher Training: Reading grades 2-5 -- 16 teachers $1,500.00
Travel for Reading Trainer $2,000.00
Teacher Training: Math grades K-1 -- 8 teachers $1,500.00
Teacher Training: Math grades 2-5 -- 16 teachers $1,500.00
Travel for Math Trainer $2,000.00
32 Teacher Manuals @ $40 $1,280.00
16 Teacher Manuals @ $44 (Grades 2-5 Reading-revised) $704.00
Math Student Materials-Grades 2-5 for 16 classes @ $30 $480.00
Total One Year Cost $12,464.00

With 24 classes participating during the school year at 30 students per class, 720 students would participate at a cost of $17.31 per student.

Funding Overview

As a curriculum-enhancing program, PALS is very inexpensive to implement. School system curriculum funds could be considered for PALS. Foundations and government grants could also assist with initial implementation expenses.

Allocating State or Local General Funds

State education funds allocated to local school systems as well as locally-appropriated public school funding can support PALS particularly during regular reviews of curricula within the district.

Maximizing Federal Funds

Formula Funds: Title I can potentially support curricula purchase and teacher training in schools that are operating school wide Title I programs. In order for Title I to be allocated, PALS would have to be viewed as contributing to overall academic achievement.

Discretionary Grants: Federal discretionary grants from the Department of Education can be used to fund the initial training of teachers.

Foundation Grants and Public-Private Partnerships

Foundations, especially those with a stated interest in improving educational achievement, can provide funding for initial training and curricula purchase.

Generating New Revenue

While curricula purchase is usually viewed as a school system responsibility, fundraising can also be considered, especially when the school has many competing needs and priorities. Parent Teacher Associations, business and local civic associations can also serve as sponsors of fundraising campaigns.

Data Sources

All information comes from the responses to a questionnaire submitted by the developers, Lynn and Douglas Fuchs, to the Annie E. Casey Foundation.

Program Developer/Owner

Douglas FuchsPeabody College, Vanderbilt UniversityPeabody Box 328230 Appleton PlaceNashville, TN 37203-5701USA615-343-4782615-343-1570doug.fuchs@vanderbilt.edu http://kc.vanderbilt.edu/pals/

Program Outcomes

  • Academic Performance

Program Specifics

Program Type

  • Early Childhood Education
  • Peer Counseling and Mediation
  • School - Individual Strategies

Program Setting

  • School

Continuum of Intervention

  • Universal Prevention

Program Goals

A class-wide peer tutoring program for elementary school students to improve reading and mathematics skills of students through guided peer-assisted learning strategies.

Population Demographics

Students of both genders, varying racial/ethnic background, from different SES backgrounds (Title 1 schools and non-Title 1 schools) formed the study population. PALS has been designed for children from kindergarten to 6th grade; however, the Blueprints-certified study focused only on grades 2-4.

Target Population

Age

  • Early Adolescence (12-14) - Middle School
  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • Hispanic or Latino
  • All

Race/Ethnicity Specific Findings

  • Hispanic or Latino

Race/Ethnicity/Gender Details

Usually program effectiveness was assessed among students regardless of gender or racial composition of the classroom. However, program effectiveness was demonstrated also among Hispanic students enrolled in English only and dual language educational programs (Calhoon et al. 2006, Calhoon et al. 2007).

Risk/Protective Factor Domain

  • Individual
  • School

Risk/Protective Factors

Risk Factors

School: Poor academic performance

Protective Factors

School: Instructional Practice, Opportunities for prosocial involvement in education


*Risk/Protective Factor was significantly impacted by the program

Brief Description of the Program

Peer Assisted Learning Strategies (PALS) for children in kindergarten through sixth grade is a type of class-wide peer tutoring that is used to improve reading and math skills. Teachers pair low and high performing students, and the partners work on different activities that address the skills that are causing problems. The pairs are changed regularly, giving all students the opportunity to act as coaches and players. PALS enables teachers to address individual student needs, as well as observe students and develop individual remedial lessons. It is a complementary strategy that teachers can use to augment their existing reading and math curricula. PALS is composed of 25-35 minute activities that are implemented 2-4 times a week for 14 to 31 weeks.

Description of the Program

Peer Assisted Learning Strategies (PALS) for children in kindergarten through sixth grade is a type of class-wide peer tutoring that is used to improve reading and math skills. Teachers pair low and high performing students, and the partners work on different activities that address the skills that are causing problems. The pairs are changed regularly, giving all students the opportunity to act as coaches and players. PALS enables teachers to address individual student needs, as well as observe students and develop individual remedial lessons. It is a complementary strategy that teachers can use to augment their existing reading and math curricula. PALS is composed of 25-35 minute activities that are implemented 2-4 times a week for 14 to 31 weeks.

PALS reading: Different versions of PALS reading have been developed for different age levels. Preschool - kindergarten PALS focuses on letter names and sounds, letter-sound correspondence, phonological awareness, and decoding. First grade PALS stresses decoding and reading fluently. In grades 2-6, PALS promotes reading fluency and comprehension. In the higher grades, PALS activities include partner reading, paragraph shrinking (identifying the main idea), and prediction relay (predicting what will be learned next, reading aloud, determining if the prediction was accurate, and summarizing the main idea). PALS does not require special reading materials.

PALS math: PALS math entails coaching and practice for students in grades K-6. Students hone their approrpiate grade-level skills and concepts on gameboards (at kindergarten and first grade) and worksheets (at grades 2-6). The coach uses a sheet containing questions that guide the player. Coaching lasts about 15-20 minutes. At grades 2-6, PALS math also uses mixed-problem practice worksheets that include the problem type the students just worked on as well as easier types of problems. Students work independently on these worksheets, after which they exchange and score the practice sheets. Practice lasts 5-10 minutes.

Theoretical Rationale

PALS was developed around the idea that frequent interaction and feedback between peers increases learning success. The common activities of PALS, partner reading, summarizing, and predicting were chosen based on the following theoretical assumptions: Partner reading served as a means to improve reading accuracy and fluency. Summarizing requires readers to monitor comprehension, and trains the identification of main ideas. Prediction activities help students to develop and automatize the strategic behavior of formulating and checking predictions about the text and the logical connection of ideas, a strategy used by expert readers.

Theoretical Orientation

  • Skill Oriented

Brief Evaluation Methodology

The majority of studies employed a randomized control trial procedure in which teachers (classrooms) were randomly assigned to treatment and control groups (e.g., Fuchs et al., 1999; Fuchs, Fuchs & Karns, 2001; Fuchs, Fuchs, Karns et al., 2001; Fuchs, Fuchs, Thompson et al., 2001; Fuchs et al., 2002; Mathes & Babyak, 2001; Calhoon & Fuchs, 2003; Calhoon, 2005; Calhoon et al., 2006, 2007; Stein et al., 2008). However, three additional studies employed a quasi-experimental design in which a non-random procedure was used to match control and intervention group participants (e.g., Mathes et al., 1998; Mathes, Torgesen & Allor, 2001; Mathes et al., 2003). The selection of the students for assessment was based on results from standardized test-scores and/or teacher's judgment (e.g., Fuchs et al., 1999; Fuchs, Fuchs, Karns et al., 2001; Fuchs, Fuchs, Thompson et al., 2001; Mathes & Babyak, 2001; Mathes, Torgesen & Allor, 2001; Stein et al., 2008). Studies investigated the effect of variations of PALS, including program enhancements (e.g., PALS with elaborated helping strategies, Fuchs et al., 1999), "hybrid" programs such as Ladders + PALS (Fuchs et al., 2001) and Linguistic Skills Training (LST) + PALS (Calhoon, 2005), skill-based mini-lessons (ML) + PALS (Mathes and Babyak, 2001), increasingly individualized treatments for program nonresponders (McMaster et al., 2005), as well as mediator effects such as differing levels of teacher assistance (Stein et al., 2008). In addition, a number of studies used PALS to improve student's mathematical achievments (Fuchs, Fuchs, Phillips et al., 1995; Fuchs, Fuchs & Karns 2001, Fuchs et al., 2002; Calhoon & Fuchs, 2003). Sample sizes varied widely and included between 4 and 140 teachers from 2 to 49 different schools. In most studies only a subset of students from each classroom was assessed (for each category [LA, AA, HA] 1 to 6 children) (see Calhoon & Fuchs, 2003; Fuchs et al., 2002; Calhoon et al., 2006, 2007 for exceptions). Studies assessed program effects only at posttest, between 15 and 20 weeks after pretest. One laudable exception is Fuchs et al. (2001) who assessed changes in treatment effects 5 months after posttest. The program has been implemented in geographically (e.g., Nashville, Tennessee [majority of studies], Minnesota, Southern Texas) and socio-economically diverse school settings (Title 1 and non-Title 1 schools).

Outcomes (Brief, over all studies)

Primary findings:
Compared to a control group PALS (or its hybrid versions) significantly improved reading skills of kindergarteners (Stein et al., 2008), first-graders (Mathes et al., 1998; Mathes, Torgesen & Allor, 2001; Mathes et al., 2003; Calhoon et al., 2006, 2007), grade 2-3 students (Fuchs et al., 1999), and grade 6-8 students (Calhoon, 2005) at posttest. PALS seems to be most effective among low achieving and average achieving students (Mathes, Torgesen & Allor, 2001; Fuchs, Fuchs, Thompson et al., 2001; Mathes & Babyak, 2001) and some studies report no effects on high achieving students (Mathes et al., 1998; Mathes, Torgesen & Allor, 2001). PALS was also found to be effective in improving mathematical skills among kindergartners (Fuchs, Fuchs & Karns, 2001), first-graders (Fuchs et al., 2002), and students in grades 2 to 4 (Fuchs, Fuchs, Phillips et al., 1995).

Secondary findings:
An improved version of PALS, which used elaborated helping strategies (PALS-HG), significantly improved reading skills compared to a control group among 4th grade, but not 2nd and 3rd grade, students (Fuchs et al., 1999). The addition of conceptual training enhanced the beneficial program effects of PALS (PMI-Elaborated) for mathematical skills among students in grades 2 to 4 (Fuchs et al., 1997). Hybrid programs, combining Ladders and PALS or Linguistic Skills Training (LST) and PALS significantly improved reading skills among students (Fuchs et al., 2001; Calhoon, 2005). However, no difference was observed if PALS was amended with computer assisted instructions (CAI) compared to PALS only (Mathes, Torgesen & Allor, 2001). The level of support that teachers received was shown to impact implementation fidelity and program effectiveness (Stein et al., 2008).

Compared to a control group, PALS effectively improved reading skills among Hispanic children of varying degrees of English proficiency (Calhoon et al., 2006, 2007). PALS was also effective in improving reading and math skills among students with identified learning disabilities (Calhoon, 2005, Fuchs, Fuchs, Phillips et al., 1995; Fuchs, Fuchs & Karns, 2001; Fuchs et al., 2002).

Outcomes

Primary findings:

  • Compared to a control group, PALS (or its hybrid versions) significantly improved reading and mathematical skills of school age (including kindergarten) students.
  • PALS was most effective among low achieving and average achieving students.

Secondary findings:

  • Improved versions of PALS (PALS-HG, PMI-Elaborated + Conceptual) and hybrid programs that combine PALS with other educational interventions (e.g., Ladders+PALS, LST+PALS) significantly enhanced reading and mathematic skills.
  • PALS effectively improved reading skills among Hispanic children of varying degrees of English proficiency.
  • PALS significantly improved reading and math skills among students with identified learning disabilities.

Mediating Effects

Mediating effects are not discussed in the majority of studies. An exception is Stein et al. (2008), who showed that the beneficial effect of K-PALS on early reading achievement gains is dependent on program implementation fidelity. Teachers with the best support showed highest levels of implementation fidelity, and controls for fidelity determined the program effects.

Effect Size

Overall, PALS showed moderate to strong effects for significant outcome measures at posttest in improving the reading skills of kindergarten and school-age children. A few studies report either consistently strong (d>.80) (e.g., Stein et al., 2008; Calhoon, 2005) or medium (d=.50-.80) effect sizes (Mathes et al., 1998; Mathes et al., 2003; Mathes, Torgesen & Allor, 2001; Calhoon et al., 2007), while the majority of studies reports a mix of medium and strong effects (e.g., Fuchs et al., 1999; Fuchs, Fuchs, Thompson, 2001; Mathes & Babyak, 2001; Calhoon et al., 2006). A reoccurring finding seems to be that strongest effect sizes are observed for average achieving (AA) and low achieving (LA) students, while only small effects are found for high achieving (HA) students (e.g., Fuchs, Fuchs, Thompson et al., 2001; Mathes & Babyak, 2001).

Improvements in mathematical skills through PALS were weaker than for reading skills with mostly small effect sizes (ES=.2-.5) (e.g., Fuchs, Fuchs, Phillips et al., 1995) that reached medium effect sizes (ES=.5-.8) in a few instances (e.g., Fuchs, Fuchs & Karns, 2001; Fuchs et al., 2002).

Generalizability

The findings can be generalized to kindergarten and school-age children (grades 1-8) irrespective of gender, ethnicity, and SES background, and reading skills. PALS effectiveness was confirmed for Hispanic students of varying levels of English proficiency enrolled in English only and bilingual education programs. For PALS math, program effectiveness has been confirmed for kindergartners and students in grades 1 to 5 with mixed results for high-school students with disabilities.

Potential Limitations

The studies evaluating PALS show some reoccurring flaws.

  • None of the reviewed studies conducted a test for differential attrition (exceptions are Calhoon, 2005; Fuchs, Fuchs, Phillips et al., 1995; and Fuchs, Fuchs & Karns, 2001, for which no attrition occurred).
  • In most studies compliance with the intent-to-treat principle is unclear (attrition occurred but no explanation is provided on whether attritors were followed). In only one case non-compliance is clearly stated (Mathes, Torgesen & Allor, 2001) while only the studies that did not have any attrition clearly comply with the intent-to-treat principle.
  • Long-term program effects are not assessed (see Fuchs, Fuchs, Thompson et al., 2001, for an exception who assessed changes in treatment effects 5 months after posttest)
  • In most studies it is unclear whether the research staff that conducted the testing was blind to children's treatment condition. Only one study (Fuchs, Fuchs, Thompson et al., 2001) seems to be aware of this problem.
  • Two studies (Fuchs, Fuchs, Thompson et al., 2001; Calhoon, 2005) investigated the effect of hybrid programs (e.g., Ladders+PALS, LST+PALS), which does not allow evaluation of the isolated treatment effect of PALS.
  • The majority of studies used the wrong unit of analysis. Randomization was done at the classroom-level while the analysis was done using individual-level information (exceptions are Stein et al., 2008, who employed multi-level models, and Fuchs et al., 1999, who evaluated three students per classroom).
  • In three studies teachers were not randomized, and some of those assigned to the treatment had already successfully used the program (Mathes, Torgesen & Allor, 2001; Mathes et al., 1998; Mathes et al., 2003).

Notes

A study by Fuchs et al. (1994) employed a randomized controlled trial design to investigate the difference between three groups: (1) Curricular based measurements (CBM) with instructional recommendations (CBM-IN; n=10), (2) CBM without instructional recommendations (CBM-NoIN; n=10), and (3) a no-treatment control group (n=20). While teachers in both groups received information on student's progress, the CBM-IN group received (a) specific recommendations about what to teach, (b) recommendations on how to constitute small groups for instruction on skills on which students experienced common chronic difficulty, (c) a list of skills and the computer-assisted program each student should use for the next 2 weeks, and (d) instructions to implement classwide peer tutoring (CWPT; which is the original term for PALS). Teachers in both conditions were permitted to use CWPT, but only CBM-IN teachers were provided with the materials and information on how to implement CWPT. This evaluation methodology does not allow isolating the program effect of classwide peer tutoring (CWPT) since the CBM-IN group also adds other program components such as group instructions, computer-assisted learning. In addition, the CBM-noIN group was also permitted to use CWPT, which further prevents a clear evaluation of program effects. Based on this problems, a complete write-up of this study is not provided. A summary of this study was also provided by Phillips et al. (1993).

Another article by Phillips et al. (1994) briefly mentioned results from the Fuchs et al. (1994) study. However, the primary purpose of Phillips et al. (1994) was to report on insights into the teachers' perspective on the advantages and disadvantages of curriculum based measurement (CBM) and peer tutoring (PT). The teachers who had participated in the study reported by Fuchs et al. (1994) were interviewed regarding (a) ways in which the program had affected their students and their instruction, (b) their level of satisfaction with the program, and (c) shortcomings of the program.

References

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., & Bentz, J. (1994). Classwide curriculum-based measurement: Helping general educators meet the challenge of student diversity. Exceptional Children, 60(6), 518-537.

Phillips, N. B., Fuchs, L. S., & Fuchs, D. (1994). Effects of classwide curriculum-based measurement and peer tutoring: A collaborative researcher-practitioner interview study. Journal of Learning Disabilities, 27(7), 420-435.

Phillips, N. B., Hamlett, C. L., Fuchs, L. S., & Fuchs, D. (1993). Combining classwide curriculum-based measurement and peer tutoring to help general educators provide adaptive education. Learning Disabilities Research and Practice, 8(3), 148-156.

Endorsements

Blueprints: Promising

Peer Implementation Sites

Hugh Cole Elementary School
50 Asylum Rd, Warren, RI 02885
(401) 245-1460
Contact email: natasha.axelson@bwrsd.org

Nicola (Nikki) Scheib
Math Curriculum Coordinator
Alan Shawn Feinstein Middle School of Coventry
15 Foster Drive
Coventry, RI 02816
(401) 822-9426
scheibnicola@coventryschools.net

Program Information Contact

Douglas Fuchs and Lynn Fuchs
Peabody College, Vanderbilt University
Peabody Box 328
110 Magnolia Circle, Suite 418
Nashville, TN 37203
Phone 615-343-4782
doug.fuchs@vanderbilt.edu
lynn.a.davies@vanderbilt.edu
vkc.mc.vanderbilt.edu/frg/

References

Study 1

Certified Fuchs, L. S., Fuchs, D., Kazdan, S., & Allen, S. (1999). Effects of peer-assisted learning strategies in reading with and without training in elaborated help giving. The Elementary School Journal, 99(3), 201-219.

Study 2

Fuchs, D., Fuchs, L. S., Thompson, A., Al Otaiba, S., Yen, L., Yang, N., . . . O'Connor, R. (2001). Is reading important in reading-readiness programs? A randomized field trial with teachers as program implementers. Journal of Education Psychology, 93(2), 251-267.

Study 3

Stein, M. L., Berends, M., Fuchs, D., McMaster, K., Saenz, L., Fuchs, L. S., & Compton, D. L. (2008). Scaling up an early reading program: Relationships among teacher support, fidelity of implementation, and student performance across different sites and years. Education Evaluation and Policy Analysis, 30(4), 368-388.

Study 4

Calhoon, M., Al Otaiba, S., Cihak, D., King, A., & Avalos, A. (2007). Effects of a peer-mediated program on reading skill acquisition for two-way bilingual first-grade classrooms. Learning Disabilities Quarterly, 30(3), 169-184.

Study 5

Calhoon, M., Al Otaiba, S., Greenberg, D., King, A., & Avalos, A. (2006). Improving reading skills in predominantly Hispanic Title 1 first-grade classrooms: The promise of peer-assisted learning strategies. Learning Disabilities & Practice, 21(4), 261-272.

Study 6

Calhoon, M. (2005). Effects of a peer-mediated phonological skill and reading comprehension program on reading skill acquisition for middle school students with reading disabilities. Journal of Learning Disabilities, 38(5), 424-433.

Study 7

Mathes, P., & Babyak, A. (2001). The effects of Peer-Assisted Literacy Strategies for first-grade readers with and without additional mini-skills lesson. Learning Disabilities Research & Practice, 16(1), 28-44.

Study 8

Mathes, P., Torgesen, J. K., & Allor, J. H. (2001). The effects of Peer Assisted Literacy Strategies for first-grade readers with and without additional computer-assisted instruction in phonological awareness. American Educational Research Journal, 38(2), 371-410.

Study 9

Mathes, P. G., Howard, J. K., Allen, S. H., & Fuchs, D. (1998). Peer-assisted learning strategies for first-grade readers: Responding to the needs of diverse learners. Reading Research Quarterly, 33(1), 62-94.

Study 10

Mathes, P., Torgesen, J. K., Clancy-Menchetti, J., Santi, K., Nicholas, K., Robinson, C., & Grek, M. (2003). A comparison of teacher-directed versus peer-assisted instruction to struggling first-grade readers. The Elementary School Journal, 103(5), 459-479.

Study 11

McMaster, K. L., Fuchs, D., Fuchs, L. S., & Compton, D. L. (2005). Responding to nonresponders: An experimental field trial of identification and intervention methods. Exceptional Children, 71(4), 445-463.

Study 12

Calhoon, M., & Fuchs, L. S. (2003). The effects of peer-assisted learning strategies and curriculum-based measurement on the mathematics performance of secondary students with disabilities. Remedial and Special Education, 24(4), 235-245.

Study 13

Fuchs, L. S., Fuchs, D., & Karns, K. (2001). Enhancing kindergartners' mathematical development: Effects of peer-assisted learning strategies. The Elementary School Journal, 101(5), 495-510.

Study 14

Fuchs, L. S., Fuchs, D., Yazdian, L., & Powell, S. R. (2002). Enhancing first-grade children's mathematical development with peer-assisted learning strategies. School Psychology Review, 31(4), 569-583.

Study 15

Fuchs, L. S., Fuchs, D., Phillips, N. B., Hamlett, C. L., & Karns, K. (1995). Acquisition and transfer effects of classwide peer-assisted learning strategies in mathematics for students with varying learning histories. School Psychology Review, 24, 604-620.

Study 16

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., & Karns, K. (1995). General educators' specialized adaptation for students with learning disabilities. Exceptional Children, 61(5), 440-459.

Study 17

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., Karns, K., & Dutka, S. (1997). Enhancing students' helping behavior during peer-mediated instruction with conceptual mathematical explanations. Elementary School Journal, 97, 223-250.

Study 18

Fuchs, L. S., Fuchs, D., Karns, K., Yazdian, L., & Powell, S. (2001). Creating a strong foundation for mathematics learning with Kindergarten Peer-Assisted Learning Strategies. Teaching Exceptional Children, 33(3), 84-87.

Study 1

Evaluation Methodology

Design:

Recruitment:
It is unclear from which pool the 24 teachers were chosen (1 or more schools) or how the recruitment was conducted.
Study type/Randomization:
This study constitutes a randomized control trial. Randomization was conducted at the classroom-level by randomly assigning teachers to intervention vs. control group.
Teachers: Participants were 15 general education teachers at primary grades 2 and 3 and 9 general education teachers at intermediate grade 4 (all teachers N=24). Stratified by grade level, teachers were randomly assigned to two treatments: collaborative reading activities, operationalized with peer-assisted learning strategies (PALS) (N=10 at grades 2-3; N=6 at grades 4) or a control group with no collaborative reading activities (N=5 at grades 2-3; N=3 at grade 4). Then half the PALS teachers were randomly assigned to a treatment that included preparation in help-giving strategies (PALS-HG). Thus, two experimental groups (PALS and PALS-HG) and one control group was established.
Students: Teachers implemented their respective treatments with all students in their reading classes. However, only for a selection of students treatment effects were assessed. Each teacher identified three students: (a) one at risk (AR) student who, as judged by the teacher, was manifesting social behavior difficulties that permeated the school day, including reading instruction, (b) one average achieving (AA) student who demonstrated no chronic behavior problems and whose reading performance, as judged by the teacher, was near the middle of the class, and (c) one high achieving (HA) student who also demonstrated no chronic behavior problems and whose reading performance as judged by the teacher, was near the top of the class.

Intervention/Attrition:
PALS sessions comprised three activities: partner reading, summarization, and prediction. For 21 weeks teachers incorporated 3 PALS sessions (35 minutes) into their regular teaching activities. Teachers taught the PALS lessons to and conducted PALS with all children in their naturally constituted classes. Compared to PALS, PALS-HG teachers taught their students specific strategies designed to help partners figure out correct responses on their own, rather than directly correcting the partner. These elaborated help giving (HG) strategies include, for example, helping reader to use context, helping reader use the vowel sounds, provide additional assistance in the summarizing exercise through asking targeted questions, etc (see p.209, table 4).

Assessment:
All assessments were conducted by trained research assistants. Reading achievement tests were administered immediately before and after treatment. Observations to characterize the nature of help students provided to peers during PALS and PALS-HG were conducted during weeks 22 and 23 of the study.

Sample characteristics:
Sample characteristics are not described in the text.

Measures:
Validity of measurements:
Inter-rater reliability was assessed. The agreement between independent observers was usually high (around 80%).

Primary outcomes:
The study employed the reading comprehension subtest of the third edition of the Stanford Diagnostic Reading Test (SDRT). For grade 2 children, the SDRT included 28 items that require sentence reading and 20 items that involved paragraph comprehension (overall Kuder-Richardson reliability=.93). For grades 3 and 4, the measures included 24 literal and 24 inferential comprehension items (overall Kuder-Richardson reliability=.94).

In addition, a structured observation system was used to assess the nature of help students provided to peers. RAs counted the errors corrected and the frequency that tutors relied on HG (help giving) strategies.

Analysis:
The unit of analysis was classroom, not students. The study uses analysis of variance to contrast, at the between-classroom level, the three study groups (PALS-HG vs. PALS vs. control) and to evaluate within-classrooms differences between the three student types (AR vs. AA vs. HA). To follow up significant effects, the authors used the Fisher LSD post hoc procedure to evaluate pairwise comparisons. Since changes in scores were evaluated, baseline controls are included in the models.

Intention-to-treat: The study setup does not allow judgement of whether the intent-to-treat principle was followed, but it appears that the three students selected at baseline for each classroom were used at posttest.

Outcomes

Implementation fidelity:
A research assistant (RA) was assigned to each teacher, who served as consultant to the teacher during the study. Teachers participated in a full-day workshop, conducted separately for PALS and PALS-HG teachers. In addition, different RAs (who were not assigned as consultants) observed and rated teachers' accuracy of implementing PALS. For PALS and PALS-HG, the percentage of correctly implemented program elements was 98.2 and 97.5 respectively.

Baseline Equivalence:
Chi-square tests indicated no relation between treatment and students' race, free/reduced-price lunch status, or gender. In addition, the three treatment groups did not differ at baseline on any outcome measure for the investigated student types (AR, AA, HA).

Differential attrition:
Attrition is not mentioned and a test for differential attrition was not performed.

Post-test:
For grades 2-3, the results showed that comparing the improvement in reading skills between pretest and posttest, PALS students outperformed PALS-HG (d=1.05) and the control group (d=.76). In this comparison, PALS-HG and the control group showed comparable improvements. In contrasts, for grade 4, PALS-HG students outperformed PALS (d=.72) and control group students (d=.98), while PALS and control group students showed comparable improvements in performance.
Student Help: On average, PALS-HG students corrected peers more frequently (p<.01) than PALS students. Also, PALS-HG students employed more frequent help giving strategies than PALS students. However, no difference in the intervention effect was found comparing the two grade groups (grade 2-3 vs. grade 4) using an interaction term (intervention x grade).

Long-term effects:
The study did not investigate long term effects.

Limitations:

  • Sampling issues: The choice of students for the evaluation of PALS success was based on "teachers judgment" and thus includes a subjective, non-random component.
  • Sample characteristics are not described.
  • It is not clear whether the study was fielded in only one school or many.
  • Attrition is not mentioned and a test for differential attrition was not performed.
  • No long term effects were investigated.
  • Inconsistent evidence for the relative benefits of PALS and PALS-HG

Study 2

Evaluation Methodology

Design:
Recruitment:
The authors recruited 4 Title 1 (high poverty) and 4 non-Title I schools in the Metro-Nashville Public Schools system. The researchers discussed the study with principals and teachers of 10 schools, 1 declined and out of the remaining 9 schools the first 8 that agreed to participate were chosen.

Sample size/Attrition:
Among the eight study schools, 38 of 43 kindergarten teachers volunteered to participate. The researchers chose 33 (no rationale is given for selection criteria), ensuring that 18 taught in Title 1 schools and that 15 taught in non-Title 1 schools. In each classroom, children were selected for study participation based on the Rapid Letter Naming (RLN) Test and teacher judgment. Children with the lowest six scores in each classroom were assigned by research assistants to a low achiever (LA) group; four students with scores in the middle of the distribution were assigned average-achiever (AA) status; the four students with the highest scores were labeled high achievers (HA). In this manner, 379 children were assigned to a LA, AA, or HA category. The study does not report attrition. However, a simple calculation (33 x 14 = 462) reveals that only 82% of the assigned students were used in the final evaluation. In addition, 25 children were identified as special-education students and were also rated LA, AA, or HA, resulting in a final study sample of 404 children. The 404 children were assessed at pre- and posttest but only 312 (77%) were assessed at the follow-up test.

Study type/Randomization/Intervention:
This study uses a randomized control trial design. The 33 teachers were assigned to one of three study groups by means of stratified (school type) randomization. Eleven teachers (N=6 Title 1; N=5 non-Title 1) were assigned to each study group. The study compared three groups: 1. Phonological training using "Ladders" (based on the Ladders-To-Literacy workbook) activities; 2. Phonological training (Ladders) in combination with peer assisted learning strategies (PALS); 3. Control group for which no phonological training was provided. The Ladders intervention was based on 15 activities, designed to enhance phonological skills. Ladder activities were teacher led, directed to the whole class. The intervention "Ladders + PALS" required children to work in dyads with same-age peers in addition to the Ladder activities. In the control group, teachers continued their regular reading/language arts instructions.

Assessment:
Research staff administered pretest (fall of kindergarten school year 1), posttest (spring of school year 1) and follow-up assessments 5 months after posttest (fall of school year 2). While the pre- and posttests were administered in 2 sessions, the follow-up assessment was administered in 1 session.

Sample characteristics:
Median percentages of African American kindergarten children in the Title 1 and non-Title 1 schools were 52% and 21% respectively. For the study classrooms only, the median proportion of children receiving free or reduced lunch was 81% in Title 1 schools and 29% in non-Title 1 schools. The average price of a home in the communities surrounding the four Title 1 schools was $53,923 vs. $136,309 in the non-Title 1 school communities.

Measures:
Validity of measurements:
All measures have been used in published studies and thus validity and reliability can be assumed. However, research staff that did the testing appears to be not blind to the treatment conditions.

Primary outcomes:
A number of tests were used to evaluate children's improvement in reading skills:

  • Rapid Letter Naming (RLN) Test
  • Rapid Letter Sound (RLS) Test
  • Segmentation - assesses children's ability to deconstruct words into component sounds
  • Word Attack Subtest of the Woodcock Reading Mastery Test - Revised Form G - assesses ability to pronounce pseudowords
  • Word Identification Subtest (Word ID) of the Woodcock Reading Mastery Test - Revised Form G
  • Blending - child is required to put sounds together in order to form a word
  • Spelling Subtest of the Wechsler Individual Achievement Test

Analysis:
The study used teachers (classrooms) as unit of analysis and thus averaged the scores of each teacher for all three (LA, AA, and HA) student groups. To examine treatment effects, the study used multivariate analyses of variance, combining conceptually similar and highly correlated outcome-measures into two constructs: phonological awareness (segmenting, blending) and alphabetic (RLN, RLS, Word ID, Word Attack, Spelling measures). Two planned contrasts compared 1) Ladders and Ladders + PALS vs. control, and 2) Ladders vs. Ladders + PALS. It is not clear whether the models include baseline controls.

Intention-to-treat: The study may not have followed the intent-to-treat principle. Even though theoretically 462 students were chosen to participate in the study, the analysis is based on 379 students (plus 25 special-education students). It is not clear why the 83 students were not included in the evaluation and whether investigators made an effort to obtain data for these potential attritors. However, for the follow-up assessment researchers tested students who were now located in numerous different schools and classrooms which displays a substantial effort to apply the intent-to-treat principle.

Outcomes - Posttest, Long-term
Implementation fidelity:
All teachers received special training for their respective intervention. Research staff monitored implementation fidelity through classroom visits and rating of teachers' quality of delivering Ladder and PALS exercises (lesson clarity, lesson presentation, student involvement). Overall, the implementation fidelity was high. For example, for PALS the accuracy of implementation ranged from 77% to 87%.

Baseline Equivalence:
There were no statistically significant differences among the three study groups in the number of hours per week teachers reported devoting to language arts and reading. No significant difference was observed in teachers' class size, years of teaching experience, age, gender, education, or race. Statistical tests show that students were comparable across study groups on characteristics such as number of school absences, age, and report card grades. In addition, the three groups did not differ on the outcome measures at baseline for any of the student subgroups (LA, AA, HA).

Differential attrition:
No test of differential attrition was performed even though a substantial number of students could not be tested at the follow-up assessments.

Post-test:
Phonological awareness (segmentation, blending)
: The results showed that for all student categories, the combined intervention groups (Ladders and Ladders + PALS) did outperform the control group (LA: F=5.73, p<.01; AA: F=10.86, p<.001; HA: F=5.40, p<.01). No significant difference was observed comparing the two intervention groups (Ladders vs. Ladders + PALS).

Alphabetic (RLN, RLS, Word ID, Word Attack, Spelling measures): Comparing the combined intervention groups (Ladders and Ladders + PALS) to the control group showed significant improvements among LA students (F=6.85, p<.001) and for the AA category (F=2.75, p<.05) but not for the HA category. No significant difference was observed comparing the two intervention groups (Ladders vs. Ladders + PALS).

Long-term effects:
Phonological awareness
: Even though the phonological improvements observed at posttest decreased at the 5-month follow-up assessment, the effects remained significant comparing the combined intervention groups (Ladders and Ladders + PALS) to the control group (LA: F=3.40, p<.05; AA: F=4.78, p<.05; HA: F=5.54, p<.05).

Alphabetic: No significant findings were observed for any contrast.

Effect size:
On phonological awareness, the PALS intervention (which included also Ladder exercises) produced largely strong effects, compared to a control group, depending on the specific test and students category (LA: segmentation d=.45; blending d=1.27; AA: segmentation d=2.10; blending d=1.97; HA: segmentation d=1.07; blending d=1.21). On alphabetic performance (RLN, RLS, Word ID, Word Attack, Spelling measures), medium to strong size effects were observed, comparing Ladders + PALS to the control group. Stronger effects were observed among LA (d=.28-1.28) and AA (d=.73-1.42) students compared to HA students (d=.08-.90).

Limitations

  • Research staff that administered pretest, posttest and follow-up tests were not blind to children's treatment condition.
  • Intent-to-treat cannot be evaluated based on the given information
  • Due to the "hybrid" program approach it is not possible to evaluate the isolated impact of PALS
  • The study did not report attrition and did not conduct a test for differential attrition.
  • It is not clear whether the models include baseline controls
  • Long-term effects were evaluated only 5-months after posttest

Study 3

Evaluation Methodology

Design:
Recruitmen/Sample size/Attrition:
The authors recruited schools at three sites in Tennessee (Nashville), Minnesota (Minneapolis, St. Paul, and Bloomington), and South Texas. In year 1, 48 schools agreed to participate in the study (10 in Nashville, 19 in Minnesota, and 17 in South Texas). In year 2, 49 schools participated (14 in Nashville, 21 in Minnesota, and 14 in South Texas). Once schools agreed to participate, project staff met with teachers to explain study participation requirements and procedures. The study recruited 145 teachers in Year 1 (52 in Nashville, 42 in Minnesota, and 51 in South Texas) and 134 teachers in Year 2 (54 in Nashville, 40 in Minnesota, and 40 in South Texas).

To all kindergartners, for which parental consent was obtained, the Rapid Letter Naming and Rapid Letter Sounds (RLS) test was administered. On the basis of students' performance, 4 children for each category (low achiever LA, average achiever AA, high achiever HA) were identified to participate in the study. The authors also included all children with identified disabilities. Pretest and posttest measures were administered to 1,674 kindergarteners in Year 1 and 1,555 in Year 2. The study does not report on attrition.

Study type/Randomization/Intervention:
This study used a randomized control trial design. Within each school, kindergarten teachers were randomly assigned to one of four study conditions: 1) 1-day workshop before K-PALS implementation (workshop group); 2) workshop plus a "booster" training workshop (booster group); 3) workshop, booster, and a class-room helper (a research assistant knowledgeable in K-PALS) (helper group); and a control group that did not implemented K-PALS.

In K-PALS, same-age children work in dyads through 72 structured lessons. Based on subjective judgment of students skills, teachers pair the strongest learners with the weakest learners. Each student in a pair takes a turn as a reader (tutee) and coach (tutor). Pairs remain together for 4 to 6 weeks. K-Pals was conducted by study teachers four times per week in 35-minutes sessions for 21 weeks. For each of the 72 peer-mediated lessons, children engage in the same four activities: 1) What Sound (learning letter-sound correspondence), Sound Boxes (learning decodable words), Sight Words (learning sight words), and Reading Sentences.

Assessment:
Program effects were assessed through comparison of gains in reading ability between pretests and posttest approximately 20 weeks after the administration of the pretest. The testing was conducted by trained research assistants.

Sample characteristics:
For the Nashville schools, non-Hispanic Whites, African Americans, Hispanics, and "Others" constituted 37%, 42%, 18%, and 3%, respectively, of the student community. Sixteen percent of students received special education services. The student population at the Minnesota site constituted of 45% African Americans, 14% Asians, 11% Hispanics, 4% Native Americans, and 26% non-Hispanic Whites. More than half of the students (57%) received subsidized lunch, 14% got special education services, and 24% were English language learners. The South Texas site involved six school districts in Hidalgo County, which ranks among the poorest counties in the nation, with 36% of its population living in poverty. The school children population was 80% Hispanic, of which many children showed limited English proficiency.

Measures:
Validity of measurements:
The used measures have been used by prior studies. To ensure a high level of data quality, double-scored data were entered in separate databases and went through a series of electronic and hand checks by a data manger. Research assistants were specifically trained to administer the tests correctly. Inter-rater agreement in Nashville and Minnesota was 96%. In South Texas, pre- and posttest reliability was 99%.

Primary outcomes:
Student reading achievement gains were calculated as the pre- to post-test gain on the Rapid Letter Naming and Rapid Letter Sounds (RLS) test.

Secondary measures:
Student-level measures:
The study included dummy variables for whether student was an English language learner, received free or reduced-price lunch, or had an individualized education plan. Additional dummies for race/ethnicity and gender were used.

Teacher-level variables:

  • Teacher-Directed What Sound fidelity rubric
  • Measures of teacher's perception of school context
  • Instructional Coherence (8 items, alpha=.80)
  • Teacher Community (5 items, alpha=.85)
  • Principal Leadership (15 items, alpha=.93)
  • Teacher Efficacy (efforts and ability, attitudes and habits, school resources)
  • K-PALS Instructional Coherence (3 items, alpha=.63)
  • K-PALS Principal Leadership (2 items, alpha=.84)

School-level variables:
Dummy variables for site location and study year were used.

Analysis:
The study used a multilevel modeling approach. In the employed 3-Level, random intercept models, students are nested in teachers' classrooms, while classrooms are nested in schools. The analysis controls for baseline scores on the outcome measure.

Intention-to-treat: It appears that the study gathered and used data on all subjects who were randomized and completed the surveys. However, the lack in reporting of attrition prevents a final conclusion regarding the study's compliance to the intent-to-treat principle.

Outcomes - Posttest, Long-term

Implementation fidelity:
All teachers that implemented K-PALS were provided with a comprehensive manual that detailed the intervention and provided guidelines for its implementation. Teachers received training and supervision to different degrees, based on intervention group assignment. To evaluate implementation fidelity, the teacher and the student pairs were observed and evaluated.

Baseline Equivalence:
No test for baseline equivalence was performed.

Differential attrition:
Attrition was not reported and no test for differential attrition was performed.

Post-test:
All three program groups (workshop group, booster group, helper group) showed higher average gains in sound identification (RLS) than the control group, confirming the effectiveness of PALS regardless of teachers' training and support. Strongest improvements were obtained if teachers had obtained the workshop and the booster training (booster group: b=18.67, p<.01), followed by teachers who were assisted by helpers (helper group: b=16.11, p<.01), and teachers who had only received the workshop training (workshop group: b=11.10, p<.01).

However, once a measure for fidelity of implementation was introduced into the models, the program group effects became insignificant. This indicated that the effects of the K-PALS program are substantially mediated by the fidelity with which teachers implement the K-PALS program. Implementation fidelity, varied by teachers' support with teachers' assisted by research assistants (helper group) showing the highest levels of implementation fidelity.

Effect size:
Compared to a control group, PALS resulted in a strong effect sizes (d=1.02-1.18) that varied slightly as a function of different levels of teachers' support.

Long-term effects:
No long-term effects were investigated in this study.

Limitations

  • Attrition is not reported and an investigation of differential attrition was not performed.
  • The lack of information on attrition makes it hard to appraise the compliance with the intent-to-treat principle.
  • No long-term effects were investigated.
  • No test for baseline equivalence was performed.

Study 4

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Three Title 1 elementary schools in a south-western school district, located near the border of Mexico were recruited for the study. Six first-grade teachers were selected based on their interest in and willingness to participate in the project. All teachers utilized the Two Way Bilingual Immersion (TWBI) program offered by their schools, providing approximately equal amounts of English and Spanish instruction throughout the school day. Students from each teachers' classroom (N=94) were selected for evaluation of program effectiveness. Of the 94 students, 18 (19%) were lost over the study period due to moving or transferring to another school, reducing the effective sample size to 76 students. The PALS group comprised 43 students (English Proficient (EP): N=30; English Language Learners (ELLs): N=13) and the control group comprised 33 students (EP: N=22; ELLs: N=11).

Study type/Randomization/Intervention:
The study constitutes a randomized control trial. Teachers were randomly assigned to either the PALS (N=3) or control group (i.e., no peer-mediated reading activities) (N=3). PALS was implemented according to the published teacher training manual and training protocol developed by Fuchs et al. (2000). PALS sessions followed a structured instructional routine as follows: (a) the teacher briefly presents the activities of the day, (b) students practice the PALS activities for about 15 minutes while the teacher circulates around the room to supervise, and (c) the teacher instructs students to switch to "Story Sharing," a partner reading activity that also lasts about 15 minutes. Pairs are assigned to teams and work together for about four weeks. Students participated in 60 PALS sessions that lasted 30-35 minutes three times per week for 20 weeks. PALS was implemented during the English portion of reading instruction.

Assessment:
Students' reading skills were assessed two weeks before (Fall), midway through (Winter), and immediately following the intervention (Spring). Assessments were conducted by trained research assistants. It is not clear whether research assistants were blind to the group assignments.

Sample characteristics:
More than 80% of children in all three schools received free or reduced-cost lunch, indicating the low socio-economic status of the student population. Of the 76 students, 79% were Hispanic. Twenty-four students were determined by their schools to have limited English proficiency (ELLs) while 52 students were considered English Proficient (EP).

Measures:
The study used four Dynamic Indicators of Early Literacy Skills (DIBELS) reading subtests:

  • Letter Naming Fluency (LNF)
  • Phoneme Segmentation (PSF)
  • Nonsense Word Fluency (NWF)
  • Oral Reading Fluency (ORF)

Analysis:
Statistical methods/baseline control:
A repeated-measures ANOVA was conducted with time (Fall, Winter, Spring) and condition (PALS vs. contrast) for each reading subtest. As such, the analysis controls for baseline scores.

Intention-to-treat: It is unclear whether the study complied with the intent-to-treat principle.

Outcomes - Posttest, Long-term

Implementation fidelity:
Teachers for the treatment conditions attended a one-day workshop that provided explicit training on PALS. A research assistant was assigned to each teacher, who visited the classroom and observed implementation fidelity and provided support for the teacher. The fidelity scores for PALS implementation ranged from 94.9% to 98.7% and thus demonstrate high implementation fidelity.

Baseline Equivalence:
No significant difference was observed between groups on gender, ethnicity, and years retained. However, the PALS condition had a higher number of female students and lower number of students with special education labels. In addition, statistical tests show no differences between intervention and control group at baseline on three of the four outcome measures. Only for the NWF measure a significant difference between EP intervention and EP control group was observed.

Differential attrition:
Even though attrition was substantial (19%), no test for differential attrition was performed.

Post-test:
Primary finding
: Results show significant time x treatment interactions favoring PALS for three out of four outcome measures (Phoneme Segmentation (PSF) F=4.38, p=.01; Nonsense Word Fluency (NWF) F=4.30, p=.01; and Oral Reading Fluency (ORF): F=63.73, p=.001). No significant time x treatment interaction was found for Letter Naming Fluency (LNF).

Secondary finding: The study further investigated treatment effects for the subgroups of English Language Learners (ELLs) and English Proficient (EP) students. For ELLs students, significant time x treatment interactions were demonstrated for two out of four measures (LNF: F=4.48, p<.05; and NWF: F=5.38, p<.01). Similarly, for EP students, two out of four time x treatment interactions were found to be significant favoring PALS (PSF: F=6.20, p<.01 and ORF: F=59.53, p<.001).

Effect size:
PALS significantly improved reading skills compared to a control group with moderate effect sizes ranging from .50 to .53 depending on the outcome measure.

Long-term effects:
The study did not investigate long term effects of PALS.

Limitations

  • Even though attrition was substantial (19%) no test for differential attrition was performed.
  • It is not clear whether research assistants were blind to the group assignments.
  • No long term effects were assessed.
  • Wrong unit of analysis: Randomization was done at the classroom-level while the analysis was done using individual-level information.

Study 5

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
The study was fielded in three Title 1 elementary schools in a New Mexico border town. Even though bilingual education was available at these schools (e.g., Spanish immersion programs, Two-way bilingual programs), the students studied were all placed in English Language-only first-grade classrooms. The study started with 96 first-grade students, in 6 classrooms of which 18 students (19%) moved, resulting in a final sample of 78 students. No statistical investigation of differential attrition was performed.

Study type/Randomization/Intervention:
This study constitutes a randomized control trial. The 6 participating teachers (and their respective classrooms) were randomly assigned to either the PALS treatment or a control condition. The program was implemented in accordance with the PALS training manual (Fuchs et al. 2000). As usual, teachers created dyads by pairing one high- and one low-performing reader and then trained all students in the classes to follow procedures for first-grade PALS. Pairs worked together for about 4 weeks. Students participated in 60 PALS sessions that lasted 30-35 minutes three times per week for 20 weeks. Sessions included two types of activities, Sounds and Words and Story Sharing. In contrast, teachers in the control condition provided the regular core reading instruction that their district and school provided to all first-grade students.

Assessment:
The assessments were done by trained research assistants. Students were tested in Fall, prior to intervention, in Winter during the 10th week of PALS, and in Spring immediately following the 20-weeks of intervention. It is not clear whether research assistants were blind to students' group assignment.

Sample characteristics:
In the three schools, over 75% of students received free and reduced lunch. The majority of the students studied were of Hispanic descent. Of the 78 students, 68% were Hispanic bilingual students, while 32% were Anglo English-monolingual students. Even though children were bilingual to varying degrees, they did not require English as a second language instruction.

Measures:
Validity of measurements:
All measures have been used in prior studies and high levels of reliability have been reported (e.g., test-retest reliability ranging from .92-.97).

Primary outcomes:
Three subtests of the Dynamic Indicators of Early Literacy Skills (DIBELS) measures were used:

  • Phoneme Segmentation Fluency (PSF)
  • Nonsense Word Fluency (NWF)
  • Oral Reading Fluency (ORF) - this measure was only administered in Winter and Spring.

Analysis:
The study employed repeated measure ANOVAs that control for baseline scores, but analyzed individuals rather than classrooms.

Intention-to-treat: It is unclear whether the study complied with the intent-to-treat principle.

Outcomes - Posttest, Long-term

Implementation fidelity:
During a one-day workshop, teachers were trained to conduct PALS. A research assistant was assigned to each teacher to provide support and guidance during the implementation process. Research assistants also assessed implementation fidelity through direct observation (40-item checklist). The fidelity scores for PALS implementation ranged from 90.28% to 97.34% with a mean of 93.75%.

Baseline Equivalence:
Statistical tests did not show much variation in demographic characteristics of children between intervention and control group. The only difference was observed for special education with higher numbers of students in the control group receiving this service. There were no significant differences in the pretreatment reading performance between students in the PALS and control conditions on any of the used measures.

Differential attrition:
Even though attrition was substantial, a test for differential attrition was not performed.

Post-test:
Compared to the control group, PALS was able to significantly improve reading performance on 2 out of 3 tests. More specifically, a significant interaction effect for time x treatment was present for the measures of Phoneme Segmentation Fluency (PSF), F(2, 148) = 12.79, p < .000, and Nonsense Word Fluency (NWF), F(2,148) = 5.32, p < .01, but not for Oral Reading Fluency (ORF), F(1,74) = 110.10, p>.05.

Furthermore, the three-way interaction of time x ethnicity x treatment was not significant for either PSF, F(1, 148) = 2.42, p = .09, NWF, F(1,148) = .996, p > .05, or ORF F(1,74) = .14, p >.05, suggesting that the program is able to improve reading skills regardless of race/ethnic background of the students.

Focusing on at-risk students, the study further demonstrated that consistently higher percentages of PALS than control students initially at risk achieved grade level performance at the end of the school year. Similarly, among PALS students, there was less slippage from performance on grade level (at the beginning of the school year) to at risk (at the end of the school year). These findings held across both Hispanic and non-Hispanic subgroups.

Effect size:
Large to moderate effect sizes ranging from .58 to .93 were found for the PALS treatment for 2 out of 3 measures on reading skills (Phoneme Segmentation Fluency, Nonsense Word Fluency).

Long-term effects:
The study did not investigate long-term program effects.

Limitations

  • Even though attrition was substantial, a test for differential attrition was not performed.
  • It is not clear whether research assistants were blind to students' group assignment.
  • The study did not investigate long-term program effects.
  • Wrong unit of analysis: Randomization was done at the classroom-level while the analysis was done using individual-level information.

Study 6

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Four special education teachers from two middle schools in a southwestern school district were recruited for the study. All teachers were selected based on their interest and willingness to participate. A requirement was that teachers had students with chronic reading difficulties in their classrooms. The study population comprised 38 students with reading disabilities in the classrooms of the four teachers. No attrition occurred.

Study type/Randomization/Intervention:
This study constitutes a randomized control trial. Teachers (and their respective classrooms) were randomly assigned to one of two treatment conditions: Linguistic Skills Training (LST) in combination with PALS (N=2) or a control group (N=2).

The LST phonological skill program was designed specifically to accompany the PALS reading comprehension program. Each LST lesson begins with a scripted teacher-led lesson, followed by teacher-directed practice. Students then practice in peer tutoring pairs closely monitored by the teacher. LST systematically and explicitly teaches phonological skills through letter-sound correspondence and phonological awareness exercises. The PALS program incorporates three essential reading activities: partner reading, paragraph shrinking, and prediction relay. LST was implemented 3 days per week (51 hours across study period) while PALS was implemented 2 days a week (34 hours across study period). The control group received reading instructions using a widely implemented remedial reading program (Saxon Phonics Intervention), which, however, has no peer-mediated activities. Saxon Phonics Intervention follows a more traditional format by integrating explicit teaching of decoding skills, spelling, vocabulary, reading fluency, and reading comprehension. The Saxon Phonics Intervention program was implemented 3 days a week (51 hours across study period). Saxon Phonics Intervention was coupled with the SRA Skill Acquisition program, which aims at improving reading comprehension. The SRA (Science Research Associates) Skill Acquisition program was implemented 2 days a week (34 hours across study period). Each session lasted between 35 and 40 minutes.

Assessment:
Assessments were conducted 2 weeks before intervention (pretest) and after 31-weeks of intervention (posttest) by trained research assistants. However, it is unclear whether the research assistants were blind to the group assignment of students.

Sample characteristics:
Student participants consisted of 32 sixth, 5 seventh, and 1 eighth grader (N=38) from four special education, self-contained language arts classrooms. All students had a learning disability and were reading at least three grade levels below their current grade placement. The study population comprised more boys (66%) than girls (34%). The majority of students were of Hispanic descent (66%), followed by whites (29%) and blacks (5%). On average, students were 12 years of age and had an IQ around 90.

Measures:
Validity of measurements:
The Woodcock-Johnson Test of Achievement-3 is a widely used, norm-referenced diagnostic achievement test with good evidence for internal consistency (r exceeds .80 for all subtests).

Primary outcomes:
The following four reading subtests of the Woodcock-Johnson Test of Achievement-3 (WJ-3) were used.

  • Letter Word Identification
  • Word Attack
  • Reading Fluency
  • Passage Comprehension

Analysis:
The authors conducted two statistical tests. First, they used univariate ANOVAs to compare posttest scores between groups for each reading subtest. As such, this test does not control for baseline controls. Second, the authors used univariate ANOVAs to assess growth (pretest scores minus posttest scores), which includes baseline values.

Intention-to-treat: The study gathered and used data on all subjects who were randomized and tested.

Outcomes - Posttest, Long-term
Implementation fidelity:
Teachers attended a 1-day workshop that provided explicit training on their individual programs. A research assistant observed the program implementation and provided ongoing support and feedback. Overall, a high implementation fidelity was reported (PALS=90.3, LST=98.7, Saxon Phonics Intervention=96.6, SRA Skill Acquisition=91.7).

Baseline Equivalence:
No significant differences were found between treatment groups for gender, race, grade level, or IQ. However, a significant difference was observed for student age with the LST/PALS group being slightly older than the control group (12.11 vs. 11.71). No statistically significant difference was observed between treatment groups on any of the four outcome measures at baseline.

Differential attrition:
Because no attrition occurred, no test of differential attrition was performed.

Post-test:
The results showed statistically significant differences between groups at posttest, with the LST/PALS condition outperforming the control condition on Letter-Word identification (F=14.32, p=.001), Word Attack (F=10.03, p=.01), and Passage Comprehension (F=11.35, p=.01). However, no significant difference between conditions was found for Reading Fluency.

The results for assessing growth (pretest scores minus posttest scores) produced similar results. Significant growth for the LST/PALS treatment group over the control group was detected for Letter-Word Identification (F=14.32, p=.001), Word Attack (F=13.19, p=.001), and Passage Comprehension (F=11.35, p=.01). No significant differences were found for growth on Reading Fluency.

Effect size:
This study reports generally strong effect sizes, for the three of four significant outcome measures (Letter Word Identification: d=1.10; Word Attack: d=.99; Passage Comprehension: d=.94).

Long-term effects:
The study did not assess long-term effects of the program.

Limitations

  • It is not clear whether research assistants were blind to the group assignments.
  • No long term effects were assessed.
  • Wrong unit of analysis: Randomization was done at the classroom-level while the analysis was done using individual-level information.
  • This study used a small sample (38 students in 4 classrooms in 2 schools).
  • The use of a hybrid program (Linguistic Skills Training (LST) + PALS) does not allow the appraisal of isolated program effects of PALS.

Study 7

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Thirty 1st grade teachers in 5 schools in a southeastern medium-sized school district were recruited. Over the course of the study, two 1st grade teachers (7%) assigned to the PALS+ML group withdrew from the study (these teachers considered conducting PALS and the mini-lectures too demanding). From within each participating classroom, the highest-achieving (HA) child, one average achieving (AA) child, and the 3 lowest-achieving (LA) children were selected for assessment, based on teacher's judgment (reassessed by the investigators).

Study type/Randomization/Intervention:
This study used a randomized control trial design. Of the 30 teachers (who volunteered to participate), 10 were assigned to 1st grade PALS, 10 were assigned to 1st grade PALS + skills-focused mini-lessons (ML), and 10 were assigned to a control group. PALS was conducted with the entire class during three 35-minutes sessions per week for 14 weeks. As usual, teachers paired students based on reading strength (stronger + weaker reader). During PALS, pairs worked through two 15-minute routines which the authors termed "Sounds and Words" and "Story Sharing." Sounds and Words provided for systematic, explicit instruction in synthetic phonics (Letter-Sounds, Hearing Sounds, Sounding-Out, Passage Reading); Story Sharing used shared reading and rereading of authentic text (Pretend-Read, Read-Aloud, Retell).

In 8 classrooms, skills-focused mini-lessons (ML) were implemented in addition to PALS. These additional sessions were conducted with the 3 low achieving (LA) children in small groups 3 times per week for 15 to 20 minutes each session. The mini-lessons mirrored exactly the content of the Sounds and Words portion of 1st-Grade PALS but, unlike the children coaches, teachers provided instruction according to student needs. Each mini-lesson was conducted prior to the children conducting PALS. Teachers in the control group conducted reading instruction in their typical manner.

Assessment:
Students reading skills were assessed by means of a pretest and a posttest, administered by trained researchers. In addition, for continuous progress monitoring (CPM) the researchers administered probes of (1) oral reading fluency of connected 1st grade text and (2) phonological awareness segmenting skills each week of the study duration. It is not clear whether the individuals administering the tests were blind to children's group assignment.

Sample characteristics:
Teachers: The teachers were mostly Caucasian (about 80%) and African American (20%), had on average 15 years teaching experience, and around 20 students in their reading instruction classes. Students: Students were between 6 and 7 years of age, about 50% were male, and almost all had attended kindergarten. The majority of students were Caucasian with around 30% African Americans and small proportion (<5%) of other racial groups. Among the low achieving sub-group, about 70% qualified for special education due to some form of language/speech disability.

Measures:
Validity of measurements:
The validity of the used measures has been demonstrated in prior studies.

Primary outcomes:
Reading achievement was assessed using the following measures of the Woodcock Reading Mastery Test - Revised edition:

  • Word Identification
  • Word Attack
  • Basic Skills
  • Passage Comprehension

Analysis:
Multivariate analysis of variance (MANOVA) was performed to examine change from pre to posttest scores. In addition, univariate ANOVAs were performed for individual items if the MANOVA showed a significant main effect. The authors determined statistical significance using 1-tailed tests.

Intention-to-treat: It is unclear whether the study followed the intent to treat principle.

Outcomes - Posttest, Long-term

Implementation fidelity:
All participating teachers attended a one-day training session, where they received a detailed manual and learned how to conduct the procedures. Project staff provided support during the implementation as needed. In addition, classroom observations were conducted and checklists were used to quantify implementation fidelity. Overall, teachers conducted PALS with 92.59% and ML with 89.53% accuracy.

Baseline Equivalence:
No statistical difference in teachers' characteristics and teaching style across study groups was observed. Statistical tests indicated no significant differences between groups on characteristics such as students' age, gender, race, prior retention, or qualification for special education service. ANOVAs revealed no statistically significant differences between LA, AA, or HA participant groups on pretest measures.

Differential attrition:
Attrition was not reported and no analysis of differential attrition was performed.

Post-test:
Low Achiever
: For low achievers (LA) a significant main effect for treatment was observed using MANOVA (F=1.19, p<.05). Univariate analyses indicate that for the Word Attack, Word Identification, and Basic Skills subtests, both PALS and PALS+ML significantly improved reading skills compared to a control group. However, the results for PALS and PALS+ML did not differ significantly from each other. Nevertheless, effect sizes suggested that PALS+ML produced stronger improvements than PALS alone. No statistically significant difference was found for the measure of passage comprehension. Thus, out of 4 tests, 3 were significant in favor of the PALS intervention groups.

Average achievers (AA) and high achievers (HA): The MANOVA indicated significant main effects for treatment condition (p<.05). Univariate ANOVAs for average achievers (AA) yielded statistically significant differences, in favor of PALS, for all four outcome variables (Word Attack, Word Identification, Basic Skills, and Passage Comprehension). In contrast, among high achievers (HA) no statistically significant findings were detected for any measure of reading skills.

Effect size:
Medium to strong effect sizes were observed for significant variables among LA students (ES=.61 to 1.00) and AA students (ES=.83-1.66). No significant results were observed for HA students.

Long term follow-up:
Long-term program effects were not investigated.

Limitations

  • Statistical significance was determined using 1-tailed tests.
  • It is not clear whether research assistants who did the testing were blind to students' treatment conditions.
  • Attrition was not reported and no analysis of differential attrition was performed. The loss of two teachers in the PALS+ML condition suggests potential bias.
  • Wrong unit of analysis: Randomization was done at the classroom-level while the analysis was done using individual-level information.
  • Long-term program effects were not investigated.
  • It is unclear whether the study followed the intent-to-treat principle.

Study 8

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Thirty-six first-grade teachers in eight schools in a southeastern medium-sized school district were recruited to participate in the study. Of the 36 teachers, 12 were assigned to conduct PALS only, 12 were assigned to conduct PALS + Computer Assisted Instructions (CAI), and 12 were assigned to a control group. Six of the classrooms included both kindergarten and first-grade students and were distributed evenly across treatment groups (two teachers of multiage students in each condition). The remaining 30 classrooms had only first-grade students.

The results from a 1-min oral reading probe of all children in the 36 classrooms were used to rank-order children within each classroom. One high achieving (HA) child, an average-achieving (AA) child, and the four lowest achieving (LA) children were selected for evaluation. The study started out with 33 HA, 35 AA, and 130 LA students. Throughout the year, 1 HA, 2 AA, and 9 LA students moved out of the area and 3 LA students were removed from the study because of excessive absences (remaining sample N=183 or 92% of baseline sample). The study did not investigate differential attrition.

Study type/Randomization/Intervention:
This study uses a quasi-experimental design. Some of the teachers (N=6) who had participated in a prior study (Mathes and Babyak 2001), requested continued participation in the PALS condition, and were assigned randomly to either the PALS or PALS+CAI intervention. The researchers then identified control group matches (N=3) from within the same school. The remaining teachers (N=27) were recruited to participate in the study and randomly assigned to PALS (N=9), PALS+CAI (N=9), or control group (N=9).

The PALS intervention was implemented as usual for a total of 16 weeks. Students in the PALS+CAI condition received additional phonological awareness instruction using the software packages Daisy Quest and Daisy's Castle. CAI was implemented only with LA subjects for a total of 8 weeks (4 weeks prior and 4 weeks during PALS). Each week, children completed three 20- to 30-min CAI sessions. Teachers in the control group conducted reading instructions in their typical manner.

Assessment:
Trained research assistants conducted all assessments at pretest and posttest. However, it is not clear whether research assistants were blind to the group assignment.

Sample characteristics:
Teachers: Teachers were mostly Caucasians (about 85%) with a small number of African American teachers (about 15%). Teachers had around 17 years of teaching experience, of which about 8 years were spent teaching first grade. Students: On average, students were 6 years of age, all had attended kindergarten and between 5% and 27% had been prior retained in grade depending on the particular group. The majority of students were Caucasians, with a substantial fraction of African-Americans (around 30%). About 20% of the low achieving children had been referred to special education.

Measures:
Reading achievement was measured using the following tests:

  • Woodcock Reading Mastery Test - Revised (WRMT-R); Subtests included Word Identification, Word Attack, Passage Comprehension (alpha= .68-.99; concurrent validity ranges from .85 to .91)
  • Test of Early Reading - 2 (TERA-2); 46 items that measure a child's concepts of print (alpha=.70-.98)
  • The Test of Word Reading Efficiency (TOWRE); 2 items, administered only to LA children (alpha=.95-.96)
  • Comprehensive Test of Phonological Processes (CTOPP), 2 items (alpha=.87-.95).

Note: Even though alpha values are reported, most subtests were not combined to form a scale (except for TERA-2), but rather were included as multiple outcomes in a MANOVA.

Analysis:
Statistical methods/baseline control/correct unit of analysis:
The study used multivariate analysis of variance (MANOVA) and follow-up univariate analyzes of variance, controlling for baseline scores. However, researchers decided to determine statistical significance using a one-tailed test. Group assignment was conducted at the teacher (classroom) level while the statistical analysis was performed with individual level data.

Intention-to-treat: The study did not comply with the intent-to-treat principle. The researchers purposefully removed 3 LA students from the study "because of excessive absences."

Outcomes - Posttest, Long-term

Implementation fidelity:
To ensure that teachers did not differ systematically in their teaching approaches, information on teaching methodology and reading instructional time allocation was collected. No significant differences in reading approaches were observed across groups. Teachers attended an all-day workshop where they received a detailed manual and learned how to conduct the procedures. A research assistant provided assistance during the implementation process and evaluated implementation fidelity. Overall, teachers conducted PALS with 93.83% accuracy.

Baseline Equivalence:
ANOVAs revealed no statistically significant differences between study groups on most of the teacher's characteristics, except for age and total years teaching experience. Groups did not differ on student's characteristics such as gender, age, race, prior attendance of kindergarten, prior retention, and qualification for special education service. A statistically significant difference among LA participant groups was found at baseline on all four components of the WRMT-R test, with lower values for children in the PALS+CAI treatment group.

Differential attrition:
No test of differential attrition was performed.

Post-test:
LA students
: To summarize the findings, PALS+CAI did no better than PALS, but both PALS groups did better than the control group. More specifically, MANOVAs found significant main effects for treatment on all 3 test groups with more than one outcome measure (WRMT-R: F=2.42, p<.05, TOWRE: F=5.68, p<.001, CTOP: F=2.77, p<.05). Follow-up univariate ANOVAs showed that both the PALS and PALS+CAI group scored significantly higher (one-tailed test) than the control group on 8 out of 9 sub-tests (no significant difference was observed for the TOWRE subtest of Word Efficiency). However, only 4 out of the 9 sub-tests are significant at the .01 level and thus would be significant at .05 in a two tailed test. A statistical comparison between the effects for the PALS and PALS+CAI groups revealed no significant differences.

AA and HA students: Results from MANOVA indicated no significant main effect for treatment. Follow-up univariate ANOVAs indicated a statistically significant difference for AA students for the word Attack subtest of the WRMT-R (thus 1 out of 5 tests was significant). No other statistically significant findings were detected for any other measure for either AA or HA students.

Effect sizes:
For significant variables (8 out of 9 tests), PALS showed moderate effect sizes (ES=.43-.74) for LA students. For the one significant measure of the AA category a moderate effect size was reported (ES=.69).

Long-term effects:
The study did not investigate long-term effects.

Limitations

  • In dropping absent students, the study did not comply with the intent-to-treat principle.
  • A test for differential attrition was not performed.
  • The study did not investigate long term effects of PALS.
  • It is not clear whether research assistants were blind to the group assignment.
  • Statistical significance was determined using a one-tailed test.
  • Wrong unit of analysis: Group assignment was conducted at the teacher (classroom)-level while the statistical analysis was performed with individual-level data.
  • PALS was only effective among low achieving students but not for average- or high-achievers.
  • The teachers were not randomized, and some of those assigned to the treatment had already successfully used the program.

Study 9

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Twenty teachers in 6 schools in a southeastern U.S. urban school district were recruited to participate in the study, 10 to conduct PALS and 10 to serve as a control group. Within each of the 20 participating first-grade classrooms, the research team identified 5 students as participants based on performance on a standardized test (Curriculum-Based Measurement). The 3 lowest achieving (LA) students, one average-achieving (AA) student, and one high-achieving (HA) student were selected for study participation (total N=100). Four LA students (4%) moved during the course of the year. Despite the prevalence of attrition, no analysis of differential attrition was performed.

Study type/Randomization/Intervention:
The study used a quasi-experimental design. Four teachers had been involved in pilot testing the PALS program and requested continued participation. Control group matches for these 4 teachers were recruited from among teachers who had similar teaching profiles (e.g., years of experience, degree held, self-reported approach to teaching). The remaining 12 teachers were recruited to participate in the project, and then randomly assigned to either the PALS or control group.

PALS was conducted as usual with the entire class, three times per week for 35-minute sessions for 16 weeks. One difference from the original study was that within pairs of students the role assignment was fixed. Thus, stronger students acted always as the tutor while the weaker students always performed the role of the tutee. Teachers in the control condition conducted reading instruction in their normal fashion.

Assessment:
Students' reading skills were assessed pretest (beginning of school year) and posttest after 16 weeks of intervention. Assessment was conducted by trained research assistants. However, it is not clear whether research assistants were blind to students' group assignments.

Sample characteristics:
Teachers: Teachers were 85% Caucasian and 15% African American. The majority of teachers were in the age range of 30 to 39 years, had 11 years of teaching experience, and around 40% were holding a masters degree. Students: All classrooms were first-grade classrooms in which the average age of students was 6 years. Among low achieving students, 25% in the intervention and 4% in the control group showed a learning or language disability. Also, among low achievers 11% of children in the intervention group and 4% in the control group had been prior retained. In both the intervention and control group, 56% were Caucasian and 44% were African American.

Measures:
Reading achievement was assessed using three tests, which have been widely used and validated in prior studies:

  • Woodcock Reading Mastery Tests-Revised (WRMT-R); subtests include Word Identification, Word Attack, Passage Comprehension; reliability coefficients ranged from .68 to .99.
  • Test of Early Reading Ability-2 (TERA-2), 46 items that yield one score, reliability coefficients ranged from .70-.98. Note: TERA-2 was administered only to LA students.
  • Comprehensive Reading Assessment Battery - Revised (CRAB-R), used two tests, Reading Fluency and Comprehension, reliability of this measure was not assessed.

Analysis:
Because data on multiple dependent sub-tests was collected on the WRMT-R and the CRAB-R, multivariate analysis of variance (MANOVA) was performed initially. Since the authors examined change from pre- to post-treatment standard scores, they included baseline controls. A significant main effect on the MANOVA was followed-up by univariate ANOVAs for each sub-test separately. The analysis used individual-level information while the group assignment was done at the classroom-level.

Intention-to-treat: It is unclear whether the study complied with the intent-to-treat principle. No attempt was made to collect data on the 4 students who moved away.

Outcomes - Posttest, Long-term

Implementation fidelity:
Teachers attended an all-day workshop in which they were trained in the correct implementation of PALS and received a detailed manual explaining all aspects of the program. Through observations the researchers measured implementation fidelity. Data indicated that teachers conducted PALS with 87.67% accuracy, while student implementation was at 81.17% accuracy.

Baseline Equivalence:
ANOVAs revealed no statistically significant differences between groups on demographic characteristics of participating teachers. However, few statistically significant differences on students' demographics were observed across the intervention groups. More HA students in the PALS group were Caucasian and more LA students in the PALS group were special education students. In addition, ANOVAs indicated a statistically significant difference between PALS and control group for LA students on the CRAB-R Word Reading test results prior to intervention.

Differential attrition:
No test for differential attrition was performed.

Post-test:
MANOVAs showed a significant main effect for treatment on the WRMT-R tests (F=5.55, p<.05) but not for the CRAB-R tests. Follow-up univariate ANOVAs of WRMT-R change scores revealed a statistically significant difference for LA students between the intervention and control group in favor of PALS on two out of three subtests (Word Identification: F=6.84, p<.05; Word Attack: F=8.47, p<.01). For AA students, statistically significant differences were detected between groups on one of three subtests (Passage Comprehension: F=6.30, p<.05), while for HA students no statistically significant differences were observed.

The TERA-2 was analyzed using ANOVA since it yields only one score. The TERA-2 was given only to LA students. A significant effect was observed in which LA students who participated in PALS experienced statistically significantly higher improvements in reading skills compared to students in the control group (F=4.05, p<.05).

In summary, for LA students 3 out of 6 tests (50%) were significant, while only 1 out of 6 tests (16%) was significant for AA students and none was significant for HA students.

Effect size:
For significant variables among LA students a moderate effect size was reported (Word identification: ES=.70; Word attack: ES=.78), while for the one significant variable among AA students a strong effect was detected (Passage comprehension: ES=1.12).

Long-term effects:
No long term effects were investigated.

Limitations

  • It is unclear whether the study complied with the intent-to-treat principle.
  • A test for differential attrition was not performed, although attrition was low.
  • The study did not investigate long term effects of PALS.
  • It is not clear whether research assistants were blind to the group assignment.
  • Wrong unit of analysis: Group assignment was conducted at the teacher (classroom)-level while the statistical analysis was performed with individual-level data.
  • The teachers were not randomized, and some of those assigned to the treatment had already successfully used the program.

Study 10

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
Twenty-two first-grade teachers in six schools (selected based on size, mean reading skills, and student's SES) in a medium-sized southeastern school district were recruited to participate in the research. Seven conducted PALS, seven conducted teacher-directed small-group sounds and words lessons (TDI), and eight served as the control group. Only low achieving students, at risk of reading failure, were selected based on reading test scores. A selection criterion for participation was that students read below 8 words per minute. The goal was to select 5 low achieving students per class (theoretical N=110). However, several classrooms did not have five students who met the selection criterion and the researchers did not obtain parental consent for all qualifying students reducing the sample to N=100. Finally, across the year, two PALS, five TDI, and four contrast students moved out of participating classrooms, further reducing the sample to N=89. No test for differential attrition was performed.

Study type/Randomization/Intervention:
This study used a quasi-experimental design. A number of teachers (N=8) had conducted PALS in prior studies and had requested continued participation in the experimental condition. These teachers were randomly assigned to the PALS or TDI intervention group. The researchers then identified control group matches (N=4) from among teachers with similar teaching styles. The remaining teachers (N=10) were recruited to participate in the project and then randomly assigned to PALS, TDI, or the control group.

PALS was conducted as usual in 35-minute sessions, 3 times per week for 16 weeks. Students were paired so that in each group there was a stronger and a weaker reader. The tutor and tutee roles were reciprocal so that throughout the PALS session, each child performed both roles. In the small-group teacher-directed instruction (TDI) condition, small groups of four to five children were formed. In these groups, teachers provided additional training three times per week for 30 minutes each session. The TDI lessons began at the same time as the first-grade PALS intervention and mirrored exactly the content of PALS. Teachers in the control group conducted reading instruction in their typical manner.

Assessment:
Students' reading skills were assessed pretest (beginning of school year) and posttest after 16 weeks of intervention (end of school year). Assessment was conducted by trained research assistants. However, it is not clear whether research assistants were blind to students' group assignments.

Sample characteristics:
Teachers: Teachers were 85% Caucasian and 15% African American. Teachers had on average 17 years of teaching experience, and around 50% were holding a masters degree. Students: All classrooms were first-grade classrooms in which the average age of students was 6 years. About 50% of the students were Caucasian, 40% were African American, and 10% were categorized as "other." Since all children were low achievers, a substantial fraction (60%) received additional tutoring. Around 25% of the children qualified, or were in the process of qualifying, for special education.

Measures:
Reading achievement was assessed using three tests and related sub-tests as described in the following. Prior research has demonstrated adequate reliability and concurrent validity for each measure.

  • Woodcock Reading Mastery Tests-Revised (WRMT-R); sub-tests include Word Identification, Word Attack, Passage Comprehension
  • Test of Word Reading Efficiency (TOWRE); sub-tests include phonemic decoding efficiency test and sight word efficiency test
  • Comprehensive Test of Phonological Processes (CTOPP); sub-test used was the phoneme segmentation test

Analysis:
Because data on multiple dependent sub-tests were collected on the WRMT-R and the TOWRE, multivariate analysis of variance (MANOVA) was performed initially. Since the authors examined change from pre- to post-treatment scores, baseline controls are implicitly included. A significant main effect on the MANOVA was followed-up by univariate ANOVAs for each sub-test separately. The analysis used individual-level information while the group assignment was done at the classroom-level.

Intention-to-treat: It is unclear whether the study complied with the intent-to-treat principle. No attempt was made to collect data on the 11 students who moved away.

Outcomes - Posttest, Long-term

Implementation fidelity:
Teachers attended an all-day workshop in which they were trained in the correct implementation of their respective intervention and received a detailed manual explaining all aspects of the program. A staff member of the research team was assigned to each teacher and provided assistance if needed. Through observations and checklists, implementation fidelity was assessed. Data indicated that teachers conducted PALS with 89.22% and TDI with 87.43% accuracy.

Baseline Equivalence:
ANOVAs revealed statistically significant differences between groups on teacher's age and total years teaching experience. Teachers did not differ in self-reported reading approach or time allocated for certain reading activities. However, teachers in the intervention groups incorporated phonics instruction more frequently in their general reading arts classes. No significant difference was observed on student characteristics across the three study groups. In addition, no statistically significant difference was observed among participant groups on any outcome measure.

Differential attrition:
No test for differential attrition was performed.

Post-test:
Using MANOVA to examine changes from pre- to post-test performance indicated a main effect for treatment on WRMT-R (F=8.19, p<.001) and on the TOWRE (F=4.38, p<.05). Follow-up univariate ANOVAs revealed that both PALS and TDI groups were significantly different from the control group on the following measures: CTOPP segmentation, TOWRE nonword efficiency, and WRMT-R word attack. The TDI group was significantly different from the control group on three additional tests, the WRMT-R word identification and passage comprehension subtests and the TOWRE word efficiency subtest. Thus, for PALS 3 out of 6 tests (50%) were significant, while for TDI all 6 tests (100%) were significant. Even though TDI shows stronger effects than PALS, no statistically significant difference between these two interventions was observed.

Effect size:
Among low achieving students, PALS showed moderate to strong effect sizes for the three significant sub-tests (CTOPP-segmentation: ES=.87; TOWRE-nonword efficiency: ES=.57; WRMT-R-word attack: ES=.59). On most measures, stronger effects were observed for TDI (CTOPP-segmentation: ES=.50; TOWRE-nonword efficiency: ES=.73; WRMT-R-word attack: ES=.81).

Long-term effects:
The study did not investigate long-term effects.

Limitations

  • It is unclear whether the study complied with the intent-to-treat principle.
  • A test for differential attrition was not performed.
  • The study did not investigate long term effects of PALS.
  • It is not clear whether research assistants were blind to the group assignment.
  • Wrong unit of analysis: Group assignment was conducted at the teacher (classroom)-level while the statistical analysis was performed with individual-level data.
  • The teachers were not randomized, and some of those assigned to the treatment had already successfully used the program.

Study 11

Evaluation Methodology

Design:
Randomization/Sample size/Attrition:
This study took place in eight Metropolitan Nashville schools participating in a large-scale investigation of the First-Grade PALS reading program from October 2000 to April 2001. Four of the schools were high-poverty Title I schools, and four were middle-class Non-Title I schools. Thirty-three first-grade teachers who volunteered to participate were stratified by school type (Title I or Non-Title I) and assigned randomly within school to one of three conditions (Standard PALS: n=11; PALS + Fluency: n=11; control: n=11). However, the evaluation of nonresponders was only conducted with the Standard PALS and PALS + Fluency classrooms (n=22). A three-step process guided selection of the nonresponders:

  • Selecting students at risk for unresponsiveness: Students were rank ordered based on scores of a Rapid Letter Naming (RLN) test. From within each classroom, the 8 lowest-performing students and 4 average-performing students were selected. These lowest-performing students were considered at risk for unresponsiveness to PALS (n=176), while the average-performing students served as a comparison group (n=88).
  • Monitoring these students' progress: From October to December 2000, the at-risk and average-performing students' reading progress was monitored weekly.
  • Identifying nonresponders among the at-risk group: Nonresponders were identified in January 2001, after students participated in PALS for three 35-min sessions per week for 7 weeks. After 7 weeks, complete monitoring data were available for 166 at-risk and 87 average students. This reflects an attrition rate of 6% and 1% for at-risk and average students, respectively. Among the at-risk students, those scoring more than .50 standard deviations below the average readers within a class on the Curriculum-Based Measurements (CBM) were identified as nonresponders (n=66).

Within the Standard PALS and PALS + Fluency classes, the 66 nonresponders were stratified by low (n=28) and very low (n=38) status based on CBM levels. Then, 22 students were assigned randomly to PALS; 22 to Modified PALS, and 22 to tutoring. Because of attrition, at the end of the study, there were 21, 15, and 20 students in the PALS, Modified PALS, and Tutoring treatment, respectively for a total of 56 nonresponders (15% attrition).

Study type /Intervention:
This study used a randomized control trial design. In the large-scale investigation that provided the context for this study, Standard PALS was compared to PALS + Fluency, which was designed to promote reading fluency. Standard PALS and PALS + Fluency were implemented 3 times per week for approximately 35 min per session for which teachers paired higher with lower performing readers to conduct the PALS activities in pairs (reciprocal roles of coach and reader). Standard PALS activities included letter-sound recognition, decoding, sight word recognition, and reading short stories. PALS + Fluency included the same activities, with two modifications. The sight words were presented in phrases rather than in isolation, and the short stories were read in a repeated reading, "Speed Game" format. Modified PALS incorporated three important alterations: First, fewer sounds and words were introduced at one time; Second, the coach modeled the sounds and words for the reader; Third, greater emphasis was placed on phonological awareness and decoding skills. Tutoring took place three times per week for 35 min per session and substituted for PALS. Tutoring included similar activities as in PALS but was designed to more closely resemble a more individualized special education-like pull-out approach in which a specially trained research assistant provided the one-to-one tutoring.

Assessment:
A battery of measures was individually administered to all study participants prior to (October 2000) and immediately following (April 2001) the treatment period. Staff did not test students whom they had tutored, to avoid examiner bias.

Sample characteristics:
Socio-demographic characteristics of the study sample are not described.

Measures:
All measures had been used in prior published research. Research assistants administering the tests were trained until an interrater agreement of >90% was achieved on all measures.

Primary outcomes:
The measures used in this study included tests of rapid naming, phonological awareness, reading words, and spelling.

  • Rapid Naming: Two measures were employed, the Rapid Letter Naming (RLN) test and the Rapid Letter Sound (RLS) test.
  • Phonological Awareness: Two tests measured the number of phonemes expressed correctly in 1 min as well as the ability to blend phonemes into words.
  • Reading Words: The Word Identification (Word ID) and Word Attach subtests of the Woodcock Reading Mastery Test-Revised were used to measure word recognition and decoding skills (internal consistency >.80).
  • Spelling: The Wechsler Individual Achievement Test spelling subtest was administered (test-retest reliability coefficient =.94).

Monitoring Measures:

  • Chapter tests: A cumulative, untimed test that recorded the percentage of sounds and words read correctly in a PALS session.
  • Dolch Probes: This test measures the number of words read correctly in 1 min of a pool of 126 high-frequency words.
  • Nonword Fluency Probes: The test measures the total number of phonemes in nonwords pronounced correctly in 1 min (alternate form reliability=.83).

Analysis:
A three-factor nested design was used to analyze change scores, comparing pre- and posttreatment group differences. Treatment (PALS vs. Modified PALS vs. Tutoring) was nested within PALS program (Standard PALS vs. PALS + Fluency). Nonresponder status (low vs. very low) was nested within treatment. The posttreatment data were analyzed with 2 x 3 x 2 (PALS program x treatment x nonresponder status) analyses of covariance (ANCOVAs), controlling for December Dolch measurement levels.

Intention-to-treat: It is unclear whether the study followed the intent-to-treat principle. No explanation was provided for the attrition and no reported attempt was made to obtain measurements for those who dropped out during the study period.

Outcomes

Implementation fidelity:
All Standard PALS and PALS + Fluency teachers attended a full-day workshop in which the PALS procedures were described, demonstrated, and practiced. Also, research staff members attended a full-day workshop to learn the Modified PALS and Tutoring procedures. Each staff member was assigned tutoring and/or classroom support roles. Staff members provided weekly technical assistance to classroom teachers. Implementation fidelity was measured using a checklist. On average, PALS was implemented with 92% fidelity, Modified PALS with 86% fidelity, and Tutoring with 97% fidelity.

Baseline Equivalence:
Analyses of variance (ANOVAs) and Chi-square tests showed no statistically significant difference at baseline across the three study groups (PALS, modified PALS, tutoring) on any outcome measure or demographic characteristic (sex, ethnicity, English Language Learner status, Title I status, and special education status).

Differential attrition:
Attrition occurred (15%) but a test for differential attrition was not conducted.

Post-test:
No statistically significant between-group differences were found on any outcome measure. However, a power analysis confirmed that the lack of significant results can be attributed to the small number of cases in each group (PALS n=21, Modified PALS n=15, Tutoring n=20). Thus, the authors calculated effect sizes as another means to explore important between group differences. Small-to-moderate effects were found favoring (a) Tutoring over PALS on Word ID (ES=.43), Word Attack (ES=.38), blending (ES=.44), and comprehension (ES=.32); (b) Tutoring over Modified PALS on Word ID (ES=.44), blending (ES=.49), and spelling (ES=.31); and (c) Modified PALS over PALS on Rapid Letter Sound (ES=.40), Word Attack (ES=.33), and comprehension (ES=.34).

Finally, investigating the proportion of students who were identified as persistent nonresponders (scoring .50 standard deviations below average performers on CBM) at posttest was significantly (p<.05) different comparing Tutoring and PALS (50% vs. 81%) in favor of Tutoring.

Long-term effects:
The study did not conduct any long-term follow-up and therefore was not able to demonstrate sustained effects.

Limitations

  • Small sample size resulting in limited statistical power.
  • No significant between-group differences were found.
  • Attrition occurred (15%) but a test for differential attrition was not conducted.
  • It is unclear whether the study followed the intent-to-treat principle. No reported attempt was made to obtain measurements for those students who dropped out during the study period.
  • The study did not conduct any long-term follow-up and therefore was not able to demonstrate sustained effects.
  • Socio-demographic characteristics of the study sample were not described.
  • It is unclear whether those conducting the assessments were blind to students' group assignment.

Study 12

Evaluation Methodology
Three teachers from three high schools in a southeastern urban school district participated. These three teachers taught a total of 10 self-containing mathematics resource classes. These 10 classes were randomly assigned to an intervention and a control group and thus each teacher was teaching both intervention and control classes. The study population comprised 92 9th through 12th graders identified as having a math disability. The study used a hybrid intervention program combining PALS and curriculum based measurements (CBM) training. PALS occurred 2 days a week for approximately 30 minutes per session while CBM training occurred once a week for 50 minutes. The PALS/CBM treatment was implemented for 15 weeks. In the control group no PALS or CBM was used.

Student's progress was measured at the beginning (pretest) and end (posttest) of the academic year using The Math Operations Test-Revised (MOT-R), the Math Concepts and Applications Test (MCAT), and the mathematics portion of the Tennessee Comprehensive Achievement Test (TCAP).

Outcomes
Using ANOVAs, a significant difference was observed between groups, with the PALS/CBM group outperforming the control group on mathematical computation scores (MOT-R) (F=7.96, p<.05). However, no difference was found on the test for mathematical concepts/applications (MCAT) and the TCAP.

Limitations

  • Some substantial attrition occurred. The study began with 120 students, but 28 students (23%) dropped out of school during the study period. The study did not conduct a test for differential attrition.
  • It is not clear whether research assistants, who conducted the assessments, were blind to student's treatment conditions.
  • It is unclear whether the study complied with the intent-to-treat principle.
  • The study did not investigate long term effects of PALS.
  • Wrong unit of analysis: Group assignment was conducted at the classroom-level while the statistical analysis was performed with individual-level data.
  • The PALS/CBM intervention compared to a control group showed significant improvements on only one out of three measures of mathematical skills with a small effect size (ES=.40).
  • The outcomes may have been affected by the significantly higher proportion of minority students in the PALS/CBM intervention group relative to the control group.
  • With the same teachers leading both intervention and control groups, the potential exists for contamination.

Study 13

Evaluation Methodology
Participants were 20 kindergarten teachers in three Title 1 and two non-Title 1 schools in a southeastern metropolitan public school system. Teachers with their classrooms were randomly assigned to PALS or a control group. To estimate treatment effects, a subset (N=168) of kindergarteners (LA: N=21, AA: N=101, HA: N=31) was selected from each classroom based on performance on the mathematical section of a standardized test (SESAT). In addition, the authors separated the group of children (N=15) who had been referred for special education (DIS). No attrition seems to have occurred and thus the study complied with the intent-to-treat principle. Teachers in the control group followed the district's core curriculum while teachers in the intervention group amended the core curriculum by implementing PALS twice weekly for 15 weeks, each time for 20 minutes. To ensure fidelity, teachers received training in PALS and program implementation was monitored by research assistants.

Children completed pre- and posttests using the mathematics portion of the SESAT and posttested on the mathematics portion of the Primary 1 level of the Stanford Achievement Test. The study used ANOVAs controlling for baseline scores to estimate program effects.

Outcomes
Baseline equivalence was demonstrated by the authors. A three-way ANOVA on the pre- and posttest SESAT scores revealed a statistically significant time x condition interaction (F=4.78, p<.05), indicating that growth in mathematical skills for all PALS students (all student categories combined) exceeded that of the control group students, however, with a trivial effect size (ES=.24). Investigating the results for each student category separately, strongest effect sizes were observed for AA students (ES=.53), followed by LA students (ES=.46) and DIS students (ES=.41). A non significant, negative effect occurred for HA students (ES=-.20). However, no significant main effect for treatment was observed for the Primary Stanford posttest scores.

Limitations

  • It is not clear whether research assistants, who conducted the assessments, were blind to student's treatment conditions.
  • The study did not investigate long term effects of PALS.
  • Wrong unit of analysis: Group assignment was conducted at the classroom-level while the statistical analysis was performed with individual-level data.

Study 14

Evaluation Methodology

Participants were 20 first-grade teachers in a southeastern metropolitan public school system. Teachers with their respective classrooms were assigned randomly within each school to PALS or a control group. Students were classified as low achievers (LA: N=82), average achievers (AA: N=170), high achievers (HA: N=93), or as learning disabled (DIS: N=18). The authors analyzed all students who were present at pre- and posttest (327 [86%] of 380 students). No analysis of differential attrition was performed. The control group teachers followed the district's core curriculum while the intervention group teachers implemented PALS three times each week for 16 weeks, each time for 30 minutes. Children completed pre- and posttests on the mathematics portion of the Primary 1 level and Primary 2 level of the Stanford Achievement Test. The authors separated the 94 items of these tests into those aligned (72 items) and unaligned (22 items) with the PALS curriculum. The study used ANOVAs, controlling for baseline scores to assess treatment effects.

Outcomes
Implementation fidelity was assured through teachers training workshop and through observation and rating of the accuracy of implementation (mean accuracy of implementation was 96%). Statistical tests indicated baseline equivalence for the used outcome measures.

A three way ANOVA (treatment, student category, time) showed a significant treatment x time interaction (F=6.52, p=.011). On the aligned portions of the Stanford Achievement Test, the improvement of PALS students exceeded that of the control group (F=5.66, p<.018) with an effect size of .31. Effect sizes for the LA, AA, and HA students were .34, .33, .31, respectively. By contrast, on the unaligned portions of the Stanford Achievement Test, the improvement of PALS students did not differ significantly from the control group (F=.09, p=.760).

Similarly, for the separate analysis of treatment effects for the subgroup of students with disabilities, a significant difference between PALS and the control group in math skills improvement was observed for the aligned portion of the Stanford Achievement Test (ES=.55), but no significant differences were observed for the unaligned scores (ES=-.13).

Limitations

  • Even though attrition occurred, the study did not conduct a test for differential attrition.
  • It is not clear whether research assistants, who conducted the assessments, were blind to student's treatment conditions.
  • It is unclear whether the study complied with the intent-to-treat principle.
  • The study did not investigate long term effects of PALS.
  • Wrong unit of analysis: Group assignment was conducted at the classroom-level while the statistical analysis was performed with individual-level data.

Study 15

Evaluation Methodology

Design:
Recruitment/Sample size:
The study was conducted in nine schools in a southeastern, urban school district. From these schools, 40 general educators teaching grades 2-4 were recruited to participate in the study. To be eligible for participation, teachers had to include in their mainstream mathematics instruction at least one student with an identified learning disability. The program was administered to all students in the classroom, but only a selection of students was used for the assessments.

Study type/Randomization/Intervention:
The study employed a randomized controlled trial design. Stratified by grade level, teachers were randomly assigned to two treatments: teacher-mediated instruction with PALS (n=20) or a control group (i.e., teacher-directed instruction without PALS, n = 20). Each teacher identified three students, based on personal judgment, with different learning histories: (a) one student who was chronically low achieving in mathematics and had been classified as learning disabled (LD student) according to state regulations; (b) one student who was chronically low achieving but had never been referred for special education assessment (LA student); and (c) one student whose mathematics performance was near the middle of the class (AA student). As such, it can be assumed that the sample consisted of 120 students (40 teachers x 3 students), although this information is not explicitly stated. PALS occurred twice weekly for 25 weeks with each session lasting 25-30 minutes. Students were assigned to work in teams on a mathematic operation with one student functioning as tutor and the other being the tutee. Every 2 weeks, tutoring assignments changed to allow every student to function as tutor and tutee. Teachers in the control group implemented their usual teaching routine.

Assessment/Attrition:
Achievement tests were conducted immediately before and after the treatment in small groups administered by research assistants. It is unclear whether those conducting the measurements were blinded to students' group assignment.

The authors reported that "no students withdrew from the 25-week study," and thus attrition was of no concern.

Sample characteristics:
Data on student's age, gender, race, and socio-economic background was not provided. The demographic information displayed in Table 2 provides only limited information on teachers' characteristics. It appears that teachers were predominantly white females, responsible for classes of an average size of 24 students, had about 13 years teaching experience, and were mostly between 30 and 50 years of age.

Measures:
Validity of measurements:
All measures and scales have been used in prior published work and reliability and validity has been established.

Primary outcomes:

  • Math Operations Test-Revised: This test was used to assess the level of acquisition of math skills (alpha = .86). The scores equal the number of correct answers out of 8 problems.
  • Mathematics Concepts and Applications Test: This test was used as a measure of transfer skills (alpha = .90). The scores equal the number of correct answers out of 8 problems.

Secondary outcomes:

  • Teacher planning: Instructional planning was measured through (1) criteria based evaluation of instructional plan sheets and (2) through the Teacher Planning Scale (alpha = .64).

Analysis:
For the analysis of group differences in students' math skills, ANOVAs were conducted with values for pre- to posttest growth. Thus, the analyses implicitly included baseline values.

Intention-to-treat: The study followed the intent-to-treat principle.

Outcomes

Implementation fidelity:
Teachers were trained in the correct implementation of PALS during a one-day workshop. In addition, a research assistant (RA) was assigned to each teacher to serve as a consultant and provided help and feedback. RAs met with teachers once every 1-2 weeks for 5 to 10 minutes. RAs also observed teachers correct implementation of the program. The percentage of correctly implemented elements was 99.57 for weekly assessment, 97.53 for biweekly student feedback, and 98.30 for tutoring. As such, a high level of implementation fidelity was demonstrated.

Baseline Equivalence:
A test for baseline equivalence showed no significant differences on any characteristics for both students and teachers across groups with one exception: Among students with learning disability (LD), the IQ test revealed significantly lower levels of IQ within the PALS group compared to the control group.

Differential attrition:
Attrition did not occur and therefore no test for differential attrition was needed.

Post-test:
Primary outcomes:
For an ANOVA on the pre- to posttest growth, the treatment main effect (d=.40; p<.01) revealed that across types of learners (LD, LA, and AA students) and types of measures (mathematics acquisition and transfer skills), students in the PALS treatment outperformed those in the control group. This pattern held for each of the student types (LD, LA, and AA students) and for both measures (mathematics acquisition [d=.62] and transfer skills [d=.40]). Although not statistically significant, some discernible differences in the effect size emerged across the two variables for students with different learning histories. Comparing PALS to the control group, effect sizes for math skills acquisition were .30, .95, and .32 for LD, LA, and AA students, respectively. Effect sizes for transfer skills were .22, .07, and .34 for the same order of student types.

Secondary outcomes:
Although the two groups of teachers provided comparable amounts of mathematics instructions, PALS teachers differed from control group teachers in their teaching plans. PALS teachers reported teaching more operations skills (p<.05), devoting more time to one-to-one instruction (p<.01), less time to independent seatwork (p<.05), and more frequently used peer-mediated instruction (p<.01). Results for the Teacher Planning Scale indicated that PALS teachers scored higher on efficient classroom organization (p < .05) and responsiveness to individual needs (p < .05).

Long-term effects:
The study did not collect long-term, follow-up data and therefore was not able to demonstrate sustained effects.

Limitations

  • It is unclear whether those conducting the measurements were blinded to students' group assignment.
  • Data on student's age, gender, race, and socio-economic background was not provided.
  • Long-term effects were not investigated.
  • Selection of students for evaluation was based on subjective teacher judgment.
  • Small sample size.

Study 16

Evaluation Methodology

Design:
Recruitment/Sample size:
General educators (n=20) teaching grades 2 to 4 were recruited from four schools in a southeastern, urban school district. To be eligible for participation, teachers had to include in their mainstream mathematics instruction at least one student with an identified learning disability.

Study type/Randomization/Intervention:
The study employed a randomized controlled trial design. Stratified by grade level, teachers were randomly assigned to PALS with specialized adaptation (ADAPT; n=10) and PALS without adaptation (noADAPT; n=10). Teachers in the ADAPT group used routine and specialized teaching adaptation. Students' progress was monitored using curriculum-based measurement (CBM) and when students' progress was inadequate, support to implement adaptations for PALS was provided based on a decision rule. Teachers in the no-ADAPT group also received information about students' progress but without decision rules for implementing adaptations to PALS.

Although the program was implemented among all students, only a selection was analyzed. From each classroom, a student was selected that (a) was chronically low achieving in mathematics, as judged by the teacher, and (b) had been classified as learning disabled according to state regulations. Peer-mediated instruction occurred twice weekly with each session lasting 20 to 30 minutes. Every 2 weeks tutoring assignments changed so that all students acted as both tutors and tutees over the course of the semester. Teachers implemented routine adaptation for 25 school weeks, while specialized adaptation lasted 12 weeks for 3rd and 4th grade teachers and 6 weeks for 2nd grade teachers.

Assessment/Attrition:
Students' progress in mathematical skills was assessed using Curriculum Based Measurements (CBM) every week for the entire study period (25 weeks). Teachers administered the weekly assessments in whole-class format.

Although no information on attrition was provided, it appears that all students selected for monitoring were retained in the study.

Sample characteristics:
Teachers: Teachers in the sample taught classes with an average size of 25 students. About 35% of teachers had a Master's degree and 65% were holding a Bachelor's degree. All teachers were female and racially homogeneous (90% Caucasian, 10% African American).

Students: Students were on average 9.4 years of age and their learning disability had been identified about 2 years ago. Students were predominantly males (75%) and African American (60%) with an average IQ of 89 points.

Measures:
Primary outcome:

  • CBM slope: Achievement was assessed by estimating a regression line for the Curriculum Based Measurements (CBM) digits scores across time for each student. Each CBM assessment comprises 25 mathematical problems. At grades 2, 3, and 4, respectively, students had 1.5, 2, and 3 min to complete the assessment. Performance was scored as the total number of correct digits.

Analysis:
The authors conducted one-way ANOVAs and ANCOVAs with scores for the CBM slopes to investigate group differences. Using slope measures implicitly controls for baseline values.

Intention-to-treat: Due to a lack of information on attrition, it remains unclear whether the study followed the intent-to-treat principle, but with no attrition and analysis of all subjects, this would not be a problem.

Outcomes

Implementation fidelity:
A research assistant (RA) was assigned to each teacher and served as a consultant. All teachers participated in a full-day workshop that explained the program components and their implementation. The accuracy with which teachers implemented the treatment was assessed by RAs through direct observation. In the ADAPT condition, the percentage of correctly implemented elements was 100 for CBM, 98.35 for graph and skills profile feedback, 97.81 for peer-mediated instruction, and 82.46 for specialized adaptation. In the no-ADAPT condition, the percentages of correctly implemented elements, on the first three dimensions, respectively, were 99.15, 96.70, and 98.86. As such, a high level of implementation fidelity was guaranteed.

Baseline Equivalence:
One-way analyses of variance (ANOVAs) revealed no significant differences between groups on teacher characteristics (e.g., grade taught, class size, degree earned, race, number of students with learning disability, overall classroom achievement) and student characteristics (e.g., age, grade level functioning in math, IQ, years since identified for special education).

Differential attrition:
No information on attrition was provided and no test for differential attrition was performed. However, it appears that attrition did not occur.

Post-test:
The one-way ANOVA conducted on slope values (CBM measurement points across time) indicated no significant difference for students with learning disabilities in the PALS group (no-ADAPT) compared to the PALS with special adaptation group (ADAPT). Even if the models controlled for IQ, no statistical group difference was observed.

Long-term effects:
The study did not collect long-term, follow-up data and therefore was not able to demonstrate sustained effects.

Limitations

  • No significant difference on mathematical achievements among students with learning disabilities was observed, comparing PALS and PALS with special adaptation.
  • Although it appears that all subjects were analyzed, no information was provided on attrition, differential attrition, or intent to treat.
  • Long-term effects were not investigated.
  • Small sample size.

Study 17

Evaluation Methodology

Design:
Recruitment/Sample size:
The authors recruited 40 general education teachers of grades 2-4. The location and number of schools from which teachers were recruited was not disclosed. To be eligible to participate, teachers had to include students with identified learning disabilities in their mainstream mathematics instruction.

Study type/Randomization/Intervention:
This study employed a randomized controlled trial design. Stratified by grade level, teachers were assigned to (1) peer-mediated instruction (PMI, an earlier term for PALS) with training in how to offer and receive elaborated help (PMI-E), (2) PMI with training in elaborated help and in methods for providing conceptual mathematical explanations (PMI-E + Conceptual), and (3) a non-treatment control group.

PMI was implemented in two 35-minute sessions per week for 18 weeks. Both students in a dyad served the roles of tutor and tutee within each class session. Dyads were formed so that pairs constituted of one student who required assistance while the other one was able to provide help. In the elaborated help group (PMI-E) students were specially trained to improve the PMI learning experience through methods such as explaining how to find the answer instead of just giving it, using different ways to provide an explanation, and ways to ensure an explanation was understood. In the PMI-E + Conceptual group (PMI-EC), students received additional training in how to contextualize problem situations (e.g., real-life examples), and how to represent quantities with visual images or physical materials.

Although the program was implemented with all students, each teacher identified four students for whom treatment effects were assessed: (a) one student who was chronically low achieving in mathematics and had been classified as learning disabled (LD), (b) one student who was chronically low achieving (LA), (c) one student whose mathematics performance was near the middle of the class (AA), and (d) one student whose mathematics performance was near the top of the class (HA). To verify the appropriateness of teachers' nominations of students as LA, AA, and HA, the authors collected pretest measures for all students in a class. For eight (6.7%) of the 120 nominated students, class rank did not conform to nominated status and teachers were requested to substitute these students. The 40 LD students were identified through a formal assessment rather than a nomination process.

Assessment/Attrition:
Assessments were administered immediately before (pretest) and after treatment (posttest), in small groups by research assistants (RAs), who were trained in test administration.

No information on attrition was provided.

Sample characteristics:
Teachers: Teachers were between 30 and 50 years of age, predominantly female (98%), and 23% indicated minority as racial/ethnic group. About half of the teachers (55%) were holding Bachelor's degrees while the remaining 45% indicated having Master's degrees.

Students: Students were on average 9 years of age and 70% were male. Of the study population, 63% belonged to a racial ethnic minority group. Students attended grades 2, 3, and 4. No information on socioeconomic background was provided.

Measures:
Validity of measurements:
The employed scales have been used in prior published work. However, the authors did not report validity measures (e.g., Cronbach's alpha) for the employed scales.

Primary outcomes:

  • Mathematical operations: The operation subtest of the Comprehensive Mathematics Test was employed to test students' skills in addition, subtraction, multiplication, and division of whole numbers, decimals, and fractions. Performance was scored as number of correct problems.
  • Concepts/applications: The concepts and applications subtest of the Comprehensive Mathematics Test used story problems and graphic scenarios to investigate students' ability to apply their mathematical skills to tangible problems. Performance was scored as number of correct problems.

Analysis:
The authors performed ANOVAs to investigate group differences. Student type (LD, LA, AA, HA) was treated as within-classroom factor, which allowed using "classroom as the unit of analysis," to estimate the effect of the program. For the post hoc analyses, Fisher least significant difference (LSD) tests were employed to evaluate pairwise comparisons for the treatment main effect.

Intention-to-treat: Due to a lack of information on attrition, it remains unclear whether the study followed the intent-to-treat principle.

Outcomes

Implementation fidelity:
A research assistant (RA) was assigned to each teacher and served as a consultant. Teachers participated in a one-day workshop in which they learned about the concepts and correct implementation of the PMI procedures. RAs assessed the accuracy with which teachers implemented PMI. For PMI-Elaborated and PMI-Elaborated + Conceptual, respectively, the percentages of correctly implemented elements were 98 and 97.

Baseline Equivalence:
One-way ANOVAs and Chi-square tests revealed no significant differences among the three groups on teachers' age level, years teaching, class size, gender, race, or highest degree earned. Tests for students demonstrated statistical equivalence for age, gender, race, grade-level functioning, IQ, and years since identified for special education.

Differential attrition:
No information on attrition was provided and no test for differential attrition was conducted.

Post-test:
Investigating growth from pretest to posttest yielded a significant effect for treatment (p<.001) for both outcome measures, although growth was greater on the concepts/application than on the operations subtest. Post hoc tests showed that the growth of the Peer Mediated Instructions with Elaborated help training (PMI-E) + Conceptual group exceeded that of both the PMI-E (ES=.32; p<.05) and the control group (ES=.73; p<.001). The growth of the PMI-E group, in turn, exceeded that of the control group (ES=.42; p<.01).

An insignificant interaction between treatment and student type (p = .07) suggested that the program was similarly effective for students of varying learning histories. Despite this insignificant finding, the authors tentatively explored differences in program effectiveness for the various student types. They found that the PMI-E + Conceptual treatment was more effective than the control treatment for LA (ES=1.15; p<.001) and HA (ES=.63; p=.001) students but not for LD or AA students. Similarly, the PMI-E + Conceptual treatment was more effective than the PMI-E treatment for LA (ES=.54; p=.049) and HA (ES=.63; p=.031) students but not for LD or AA students.

Long-term effects:
The study did not collect long-term, follow-up data and therefore was not able to demonstrate sustained effects.

Limitations

  • No information on attrition was provided and no test for differential attrition was conducted.
  • Compliance with the intent-to-treat principle is unclear.
  • Although not a significant finding, no clear pattern of treatment effects emerged for various student types.
  • Long term effects were not investigated.
  • The authors did not report validity measures (e.g., Cronbach's alpha) for the employed scales.
  • No information on the socioeconomic background of sample children was provided.
  • Small sample size.

Study 18

Evaluation Methodology

Design:
Recruitment/Sample size:
Kindergarten teachers (n=20) were recruited from 3 Title 1 and 2 non-Title 1 schools in Nashville. To be included in the study, teachers had to agree to be assigned randomly to the K-PALS intervention or control group.

Study type/Randomization/Intervention:
The study employed a randomized controlled trial design. Kindergarten teachers (n=20) were randomly assigned to the K-PALS intervention group or the control group (information on group size was not provided). The control group employed the same basal math instructions but without using peer assisted learning strategies. In the intervention group, teachers implemented K-PALS with all students. As usual, only a subset of students per classroom was selected for analysis: (a) students with learning disabilities (LD, intervention group children: n=8; control group children: n=7); (b) low achievers (LA, intervention group children: n=8; control group children: n=7); (c) average achievers (AA, intervention group children: n=49; control group children: n=52); high achievers (HA, intervention group children: n=14; control group children: n=17). The program was implemented for 16 weeks with two K-PALS sessions, 20 minutes each, occurring every week.

In this study, students were paired by first ranking the children based on their mathematical competence and then pairing the highest-performing child with the lowest-performing student, the second highest-performing child with the next-to-the-lowest-performing child, and so forth. Every third week, however, teachers used an alternative strategy, in which each student was paired with the next highest-performing student. However, tutoring roles were reciprocal so that both students in each pair functioned as tutor and tutee within every session.

Assessment/Attrition:
Assessments were conducted prior to program implementation (pretest) and after the program was completed (posttest). It is unclear who conducted the assessments and whether these individuals were blind to students' group assignment.

No information on attrition was provided.

Sample characteristics:
No information on sample characteristics for teachers and students was provided.

Measures:
Validity of measurements:
The employed scale has been used in prior published work. However, reliability and validity measures (e.g., Cronbach's alpha) were not reported.

Primary outcomes: (aside from mentioning the names of the used scales no information was provided)

  • The mathematics portion of the Primary 1 level of the Stanford Achievement Test was employed to evaluate changes in children's mathematical skills.

Analysis:
The statistical methods employed for this analysis were not reported. However, based on other works by Fuchs et al. it seems likely that ANOVAs were employed as a commonly used methodology.

Intention-to-treat: Due to a lack of information on attrition, it is unclear whether the study followed the intent-to-treat principle.

Outcomes

Implementation fidelity:
The authors report that "with the exception of one teacher, K-PALS was conducted accurately: Teachers implemented lessons well, and children worked on K-PALS gameboards in the manner in which the activities had been designed." However, no measures to quantify implementation fidelity were reported.

Baseline Equivalence:
A test for baseline equivalence was not conducted.

Differential attrition:
No data on attrition was reported and a test for differential attrition was not performed.

Post-test:
The results from the mathematics standardized achievement test demonstrated a stronger improvement in mathematical skills among students in the K-PALS intervention group compared to the control group (significance levels were not reported). This difference was evident across all student types (e.g., LD, LA, AA, HA).

Long-term effects:
The authors mention that they also tested "as many children as [they] could find again in the subsequent fall." At that time, when children were beginning first grade, K-PALS students continued to outperform their counterparts of the control group.

Limitations

  • It is unclear whether the study followed the intent-to-treat principle.
  • The statistical methods employed for this analysis were not reported and it is unclear whether the correct unit of analysis was used.
  • A test for baseline equivalence was not conducted.
  • No data on attrition was reported and a test for differential attrition was not conducted.
  • No information on sample characteristics for teachers or students was provided.
  • It is unclear who conducted the assessments and whether these individuals were blind to students' group assignment.
  • Reliability and validity of the employed scale was not reported.
  • No measures to quantify implementation fidelity were reported.
  • Poor reporting of the methodology.
  • Small sample size.

Contact

Blueprints for Healthy Youth Development
University of Colorado Boulder
Institute of Behavioral Science
UCB 483, Boulder, CO 80309

Email: blueprints@colorado.edu

Sign up for Newsletter

If you are interested in staying connected with the work conducted by Blueprints, please share your email to receive quarterly updates.

Blueprints for Healthy Youth Development is
currently funded by Arnold Ventures (formerly the Laura and John Arnold Foundation) and historically has received funding from the Annie E. Casey Foundation and the Office of Juvenile Justice and Delinquency Prevention.