Print Page
Blueprints logo

Incredible Years – Teacher Classroom Management

A program that provides teachers of children ages 3-8 years with classroom management strategies (positive and proactive teaching techniques, positive teacher-student relationships, and supportive teacher-parent relationships) to manage difficult and inappropriate child behavior problems, while promoting social, emotional, and academic competence.

Fact Sheet

Program Outcomes

  • Conduct Problems
  • Emotional Regulation
  • Prosocial with Peers

Program Type

  • School - Environmental Strategies
  • Social Emotional Learning
  • Teacher Training

Program Setting

  • School

Continuum of Intervention

  • Selective Prevention
  • Universal Prevention

Age

  • Early Childhood (3-4) - Preschool
  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • All

Endorsements

Blueprints: Promising
Crime Solutions: Effective
OJJDP Model Programs: Effective
SAMHSA : 3.6-3.7

Program Information Contact

Malaysia Guzman, IY Solutions Consultant
Incredible Years, Inc.
1411 8th Avenue West
Seattle, WA 98119 USA
phone: 1-843-284-2216
malaysiag@incredibleyears.com
www.incredibleyears.com

Program Developer/Owner

Carolyn Webster-Stratton, Ph.D.
University of Washington


Brief Description of the Program

The Incredible Years is a series of programs that addresses multiple risk factors across settings related to the development of conduct disorders in children. In all three training programs (Parent, Teacher, Child), trained facilitators use videotaped scenes to encourage group discussion, problem-solving, and sharing of ideas. The parent and child components of the series are described in separate write-ups.

Incredible Years Training for Teachers. This series for teachers of children ages 3-8 years emphasizes effective classroom management skills such as: the effective use of teacher attention, praise and encouragement, use of incentives for difficult behavior problems, proactive teaching strategies, how to manage inappropriate classroom behaviors, the importance of building positive relationships with students, and how to teach empathy, social skills and problem-solving in the classroom. Teachers receive 4-6 days of training spread out over several months.

The Incredible Years Series is a comprehensive program for parents, teachers, and children with the goal of preventing, reducing, and treating behavioral and emotional problems in children ages two to eight. The core of the program is the BASIC parent training component which emphasizes parenting skills such as playing with children; helping children learn; using effective praise, incentives, and limit-setting; and handling misbehavior. Additional parent training components include an ADVANCE series which emphasizes parent interpersonal skills such as effective communication, anger management, problem-solving between adults, and ways to give and get support, and a SCHOOL series which focuses on parenting approaches designed to promote children's academic skills.

To facilitate generalization from home to the school environment, a training series for teachers providing effective classroom management skills was added to the Incredible Years Series. The last addition was the training series for children (Dina Dinosaur Curriculum), a "pull out" treatment program for small groups of children exhibiting conduct problems. This curriculum emphasizes emotional literacy, empathy and perspective taking, friendship development, anger management, interpersonal problem-solving, following school rules, and school success.

A brief description of the teacher program is provided below. The parent and child components of the series are described in separate write-ups.

Teacher Classroom Management Program

The teacher training series is a group-based program for teachers, school counselors, and psychologists. Participants receive 6 days (42 hours) of training spread throughout the academic year. The training targets teachers' use of effective classroom management strategies such as: the use of differential attention, academic persistence, social and emotional coaching, praise and encouragement, incentives to increase targeted positive behaviors, proactive teaching and positive discipline strategies to manage inappropriate classroom behaviors, and positive relationship-building skills. Additionally, teachers learn how to teach children empathy, social skills, emotional self-regulation and problem solving in the classroom. The program includes strategies for helping teachers to stay calm when dealing with difficult students, ways to build a support network with other teachers, and approaches to strengthen teachers' collaborative process and positive communication with parents (e.g., the importance of positive home phone calls, regular meetings with parents, home visits, and successful parent conferences). For indicated children (i.e., children with conduct disorders), teachers, parents, and group facilitators will jointly develop "transition plans" that detail classroom strategies that are successful with that individual child; and ways parents would like to be contacted by teachers. This information is passed on to the following year's teachers.

Additionally, teachers learn how to prevent peer rejection and bullying by helping the aggressive child learn appropriate problem-solving strategies and helping his/her peers respond appropriately to aggression. Physical aggression in unstructured settings (e.g., playground) is targeted for close monitoring, teaching and incentive programs.

Outcomes

Evaluations of the Incredible Years Teacher Classroom Management Program (IYTCM) show:

  • Significant decreases in conduct problems and other problem behavior among intervention children, including noncompliant and off task behavior, compared to control group children;
  • Significant improvements in behavior among high-risk intervention children, compared to control group children;
  • Significant decreases in disruptive behavior, when mental health consultants are used as support for trained teachers;
  • Significant reductions in conduct problems at home (with both mothers and fathers) and at school, when the IYTCM program is combined with other components of the Incredible Years Series.
  • Significant improvement in school readiness

Program Effects on Risk and Protective Factors:

  • Significant improvements in child self-regulation and cooperation skills;
  • Significant improvements in interpersonal skills, and reduction in stress and social impairments, among high risk children;
  • Increases in the use of positive classroom management strategies among program teachers and reductions in the use of negative classroom management strategies;
  • When used in combination with other components of the Incredible Years Series, there are significant reductions in negative parenting and increases in positive parenting, increases in teacher-parent (mother) bonding, and significant program effects on teacher classroom management skills.

Brief Evaluation Methodology

Although the original (BASIC) Incredible Years parent program has been tested and evaluated numerously for more than 20 years, researchers have only more recently begun to test the effectiveness of the teacher component alone on child and teacher behavior outcomes. The main study, a cluster randomized controlled trial (RCT) in the UK conducted by Hutchings et al. (2013) found significant program effects on both teacher classroom strategies and child problem behaviors. Other cluster RCT studies in Jamaica (Baker-Henningham et al., 2012) and Ireland (McGilloway et al., 2010) have also found significant favorable outcomes both for teachers and students. The teacher component has also been tested with some effectiveness in additional studies in Wales (Hutchings et al., 2007) and in Chicago, IL (Raver et al., 2008), including implementation with mental health professionals to support teachers in multiple Head Start centers in which parents were also encouraged to attend the parent training program (Williford & Shelton, 2008). Additional studies conducted in the US have also evaluated the additive effects of teacher-training in combination with parent training, child training or both (Webster-Stratton, Reid & Hammond, 2004; Reid, Webster-Stratton & Hammond, 2003; Webster-Stratton, Reid, & Hammond, 2001). Teacher outcomes among both 3-year-old and 4-year-old samples in Head Start centers in 10 states and four regions across the US were examined in a RCT conducted by Hsueh et al. (2014) and Morris et al. (2014). Webster-Stratton, Rei & Stoolmiller (2008) conducted an experimental study with 120 Seattle Head Start programs and 14 elementary schools to examine outcomes among preschool, kindergarten and first grade teachers. Using the same data set, Thompson et al. (2017) conducted an analysis on a sub-set of kindergarten teachers and their students entering kindergarten to examine teachers' perception of parental involvement in school. Herman and Reinke (2017) conducted the same analysis as Thompson et al. (2017) but used a different sample of kindergarten through third grade teachers.

Kirkhaug et al., (2016) and Fossum et al. (2017) used identical evaluation designs. In both of these studies, the program was evaluated using an unmatched quasi-experimental pretest-posttest design. Schools were recruited from rural and urban municipalities in Norway, and schools that had previously implemented the program's parent training portion were eligible to apply for participation in the intervention group (n=21 in Kirkhaug et al., 2016 and 46 in Fossum et al., 2017). Another 22 schools (in Kirkhaug et al., 2016) and 46 schools (in Fossum et al., 2017) were recruited to act as the waitlist control group provided they did not have any recent or ongoing evidence-based school behavior intervention programs. Participating children were selected by randomly selecting 7 children from each participating classroom in grades 1-3 and screening them for externalizing behaviors. For Kirkhaug et al. (2016), only those who scored in the clinical range were invited to participate (n=83) whereas in Fossum et al. (2017) these students were identified as "high-risk" and analyzed in moderation analyses. Participants were assessed at baseline and at a follow-up approximately 9 months later, when 80% (in Kirkhaug et al., 2016) and 74% (in Fossum et al., 2017) of students were retained. Primary outcome measures in both studies were externalizing behaviors, social skills, and student-teacher relationships.

Blueprints Certified Studies

Study 1

Hutchings, J., Martin-Forbes, P., Daley, D., & Williams, M. E. (2013). A randomized controlled trial of the impact of a teacher classroom management program on the classroom behavior of children with and without behavior problems. Journal of School Psychology, 51, 571-585.


Study 2

Baker-Henningham, H. Scott, S., Jones, K., & Walker, S. (2012). Reducing child conduct problems and promoting social skills in a middle-income country: Cluster randomised controlled trial. The British Journal of Psychiatry, 1-8. doi: 10.1192/bjp.bp.111.096834


Study 3

McGilloway, S., Hyland, L., Mhaille, G. N., Lodge, A., O'Neill, D., Kelly, P., . . . Donnelly, M. (2010). Positive classrooms, positive children: A randomised controlled trial to investigate the effectiveness of the Incredible Years Teacher Classroom Management programme in an Irish context (short-term outcomes). A Summary Report for Archways.


Risk and Protective Factors

Risk Factors

School: Poor academic performance

Protective Factors

Individual: Problem solving skills, Skills for social interaction*

Family: Parental involvement in education

School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school


* Risk/Protective Factor was significantly impacted by the program

See also: Incredible Years - Teacher Classroom Management Logic Model (PDF)

Race/Ethnicity/Gender Details

Gender Specific Findings
  • Male
  • Female
Race/Ethnicity/Gender Details
The multiple teacher-focused studies included diverse geographic populations, including Ireland, an inner-city population in Jamaica, Wales, the United Kingdom, and diverse populations in multiple Head Start centers in the United States with a majority of ethnic minority youth and parents.

Training and Technical Assistance

TEACHING PRYAMID™ Workshop
Teacher Classroom Management Group Leader Training


This workshop will help group leaders learn how to deliver the evidence-based Teacher Classroom Management Training Program to preschool and early school-age teachers. The Teaching Pyramid™ teaches how to strengthen teacher classroom management strategies, promote children's prosocial behavior and school readiness (reading and writing skills), and reduce classroom aggression and non-cooperation with peers and teachers. Additionally the curriculum focuses on ways teachers can effectively collaborate with parents to support their school involvement and promote consistency from home to school.

After the training, leaders can offer this 6-day curriculum to groups of teachers in their schools. It may be delivered in monthly or weekly meetings and takes 42 hours for teachers to complete the entire series. The curriculum uses Webster-Stratton's book Incredible Teachers: Nurturing Children's Social, Emotional and Academic Competence (Incredible Years Press) as the text for the teachers.

Group leaders should have a background in child development, social learning theory, adult group leadership skills and experience teaching children in the classroom. Education and accreditation as a teacher, psychologist, school counselor or completion of certification as an Incredible Years parent group leader or child group leader are requirements for certification in this program, in addition to attendance at this training workshop.

Teachers work in small groups to develop individual behavior plans for targeted students which they share with each other. Additionally the curriculum focuses on ways teachers can effectively collaborate with parents to support their school involvement and promote consistency from home to school in regard to their behavior and learning plans.

Benefits and Costs

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Program Costs

Start-Up Costs

Initial Training and Technical Assistance

Initial training and technical assistance costs typically include a five-day training for group leaders, delivered either online for $890 per leader for our prescheduled workshops or $7,875 for an agency hosted online training. In-person training at the program implementation site (which can be cost effective for groups of more than 15-25 leaders) costs $2,100 - $2,500 per day plus travel costs for trainers.

Curriculum and Materials

A set of program DVDs costs $1,725 plus shipping. Additional leader manuals cost $92 each.

Licensing

None.

Other Start-Up Costs

Equipment to play DVDs, toys for role plays, and video equipment to film sessions (if not already part of staff equipment).

Intervention Implementation Costs

Ongoing Curriculum and Materials

Teacher books cost $34 each, plus shipping. Program leaders should budget for handouts for the Teacher Group at approximately $10/teacher. Other materials such as puppets, feeling cards, and extra stickers may be needed if classrooms don't have them. Classroom sets may be purchased at $120/set.

Staffing

Qualifications: Group leaders may come from a variety of helping professions such as social work, psychology, and education. It is required that they have taken a course in child development and it is recommended that they have had training on social learning. The purveyor recommends that at least one of the two leaders running a group has a master's degree or higher if working with teachers in a high-risk environment. It is also recommended that one of the leaders be a teacher or former teacher and/or experienced teaching children.

Ratios: 2 group leaders lead a group with 14-16 participants.

Time to Deliver Intervention: Teacher workshops are held monthly for 6 months, for 7 or 8 hours per workshop. Program developers recommend budgeting 10 hours per workshop per group leader to account for preparation time, peer review of videos and weekly calls or classroom visits. This time will need to be expanded for teachers in high risk classrooms who may need additional classroom visits and/or make up appointments for missed workshops and for networking with other agencies.

Other Implementation Costs

Other implementation costs include snacks/meals provided at each workshop, estimated to cost $60/session; small rewards for teachers; and any space rental fees.

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

Ongoing consultation by accredited IY program coaches, mentors and trainers is recommended for an hourly rate ($365), with a suggested two hours per month, as well as a half day rate of $1,050.00 for online consultation with program leaders.

Fidelity Monitoring and Evaluation

The purveyor suggests video review and certification at a cost of $920 per program leader (video review, submission of lesson or session protocols, and additional paperwork are required for group leaders to become certified/accredited). Other video reviews can be arranged for $265/hour. The purveyor suggests that programs budget $700-$900 per leader for the first year or first 2-3 groups for video reviews and consultation. Once group leaders receive certification, they are eligible to receive training to become mentors, which permits them to provide authorized training workshops and coaching support to others.

Ongoing License Fees

None.

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

None.

Year One Cost Example

This example assumes that a community-based organization would offer the Incredible Years Teacher Classroom Management Program to two groups of 16 teachers, each with 2 program leaders, for 6 workshops for two cohorts per year (4 total leaders for 24 total workshop sessions, for 64 total teachers). Teacher substitutes and space costs not included.

Group leader initial training costs @ $890 x 4 $3,560.00
Set of program DVDs $1,725 x 2 $3,450.00
Group leader training manuals @ $92 x 2 $184.00
Ongoing consultation: 2 hours/month @ $365/hour x 6 months $4,380.00
Annual online half-day consultation with program leaders $1,050.00
Video review/certification @ $920/program leader x 4 $3,680.00
Food (lunch, snacks) for teacher sessions @ $60/session x 24 sessions $1,440.00
Handouts for parent sessions @ $10/teacher x 32 teachers x 2 $640.00
Teacher books @ $34/teacher x 64 total teachers/year (shipping cost varies) $2,176.00
Group leader time @ $25/hour x 4 leaders x 10 hours/month x 6 sessions x 2 $12,000.00
Total One Year Cost $32,560.00

With 64 teachers participating, the initial cost of the program is approximately $509/teacher in the first year, and $453 per teacher thereafter, assuming there is no need to train more group leaders.

Funding Strategies

Funding Overview

As a program that promotes positive parent, teacher, and child relationships in order to increase a child's success at school and at home, funding sources that promote positive mental/behavioral health, parenting education, and school readiness are all potential sources of support for the Incredible Years.

Funding Strategies

Improving the Use of Existing Public Funds

Early childhood education or elementary programs that already have a teacher professional development component could utilize the Incredible Years training and curriculum to structure and improve the effectiveness of teacher education. For example, some states and localities allocate resources to community school projects that offer regular professional development. Likewise, Head Start programs have a strong parent involvement component and could potentially utilize Incredible Years Basic and Advance programs with parents to complement the Teacher Classroom Management program.

Allocating State or Local General Funds

State and local mental/behavioral health funding sources are a key source of support for the Incredible Years program. State and local funds to support crime and delinquency prevention, as well as child welfare prevention funds, could also be considered.

Maximizing Federal Funds

Formula Funds:

  • Title I can potentially support curricula purchase, training, and teacher salaries. In order for Title I to be allocated, the Incredible Years would have to be viewed as contributing to overall academic achievement or promoting family engagement.
  • The Mental Health Services Block Grant (MHSBG) can fund a variety of mental health promotion and intervention activities and is a potential source of support for the Incredible Years.
  • The Child Care and Development Block Grant (CCDBG) is used by states to support child care subsidies, early childhood education contracts, and quality improvement efforts in early childhood education. CCDBG quality dollars could be used to train group leaders and purchase materials that could be implemented in early childhood education settings.
  • Title IV-B, Parts 1 & 2 provides fairly flexible funding to state child welfare agencies for child welfare services including prevention and family preservation activities.

Discretionary Grants: Federal discretional grants from the Substance Abuse and Mental Health Services Administration (SAMHSA) at the U.S. Department of Health and Human Services can be a source of funding.

Foundation Grants and Public-Private Partnerships

Foundations, especially those with a stated interest in teacher education, early childhood development, and the wellbeing of vulnerable children and families, can provide funding for initial training and program materials purchase. Foundations can also provide support for group leaders to receive certification, and become coaches and mentors who can provide ongoing training and support to others.

Generating New Revenue

Some programs charge teachers, schools or the district a small fee to cover or defray meeting costs. Parent Teacher Associations, business, and local civic associations can also serve as sponsors of fundraising campaigns to support the Incredible Years program. Many school districts already have some resources devoted to monitoring and supporting teachers in the classroom (e.g., some districts have instructional coaches, some schools even have instructional specialists in each building who devote at least part of their time to supporting teacher skill development in the classroom), but new revenue could go to support substitute teachers to increase involvement.

Data Sources

All information comes from the purveyor's website and from written responses submitted by the purveyor to the Annie E. Casey Foundation.

Evaluation Abstract

Program Developer/Owner

Carolyn Webster-Stratton, Ph.D.Professor EmeritusUniversity of WashingtonIncredible Years, Inc.1411 8th Avenue WestSeattle, WA 98119206-285-7565888-506-3562cwebsterstratton@comcast.net www.incredibleyears.com

Program Outcomes

  • Conduct Problems
  • Emotional Regulation
  • Prosocial with Peers

Program Specifics

Program Type

  • School - Environmental Strategies
  • Social Emotional Learning
  • Teacher Training

Program Setting

  • School

Continuum of Intervention

  • Selective Prevention
  • Universal Prevention

Program Goals

A program that provides teachers of children ages 3-8 years with classroom management strategies (positive and proactive teaching techniques, positive teacher-student relationships, and supportive teacher-parent relationships) to manage difficult and inappropriate child behavior problems, while promoting social, emotional, and academic competence.

Population Demographics

The IY teacher training program is targeted for teachers of children between the ages of three and eight and has separate training protocols for preschoolers (ages 3-5 years) and early school age children (ages 6-8 years). The program can be used in schools (e.g., Head Start, daycare, and kindergarten through grade 3).

Target Population

Age

  • Early Childhood (3-4) - Preschool
  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Gender Specific Findings

  • Male
  • Female

Race/Ethnicity

  • All

Race/Ethnicity/Gender Details

The multiple teacher-focused studies included diverse geographic populations, including Ireland, an inner-city population in Jamaica, Wales, the United Kingdom, and diverse populations in multiple Head Start centers in the United States with a majority of ethnic minority youth and parents.

Other Risk and Protective Factors

Risk: For students, targeted risk factors include poor social skills and academic underachievement. Teacher-related targeted risk factors include harsh and critical teaching approaches, a poorly managed classroom, poor teacher-student relationships, and failure to collaborate with parents.

Protective: The Incredible Years Teacher Classroom Management Program focuses upon teacher-related protective factors including strengthening children's emotional and social literacy, problem solving skills and peer relationships, which can also lead to increased social competence, emotional self-regulation, and stronger school bonding in children (protective factors linked to positive outcomes). Protective factors stressed in children include social competence in skills, such as how to play with other children, affective awareness, how to be friendly and talk to peers, self-control and anger management strategies, and how to problem-solve conflict situations.

Risk/Protective Factor Domain

  • Individual
  • School
  • Family

Risk/Protective Factors

Risk Factors

School: Poor academic performance

Protective Factors

Individual: Problem solving skills, Skills for social interaction*

Family: Parental involvement in education

School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school


*Risk/Protective Factor was significantly impacted by the program

Brief Description of the Program

The Incredible Years is a series of programs that addresses multiple risk factors across settings related to the development of conduct disorders in children. In all three training programs (Parent, Teacher, Child), trained facilitators use videotaped scenes to encourage group discussion, problem-solving, and sharing of ideas. The parent and child components of the series are described in separate write-ups.

Incredible Years Training for Teachers. This series for teachers of children ages 3-8 years emphasizes effective classroom management skills such as: the effective use of teacher attention, praise and encouragement, use of incentives for difficult behavior problems, proactive teaching strategies, how to manage inappropriate classroom behaviors, the importance of building positive relationships with students, and how to teach empathy, social skills and problem-solving in the classroom. Teachers receive 4-6 days of training spread out over several months.

Description of the Program

The Incredible Years Series is a comprehensive program for parents, teachers, and children with the goal of preventing, reducing, and treating behavioral and emotional problems in children ages two to eight. The core of the program is the BASIC parent training component which emphasizes parenting skills such as playing with children; helping children learn; using effective praise, incentives, and limit-setting; and handling misbehavior. Additional parent training components include an ADVANCE series which emphasizes parent interpersonal skills such as effective communication, anger management, problem-solving between adults, and ways to give and get support, and a SCHOOL series which focuses on parenting approaches designed to promote children's academic skills.

To facilitate generalization from home to the school environment, a training series for teachers providing effective classroom management skills was added to the Incredible Years Series. The last addition was the training series for children (Dina Dinosaur Curriculum), a "pull out" treatment program for small groups of children exhibiting conduct problems. This curriculum emphasizes emotional literacy, empathy and perspective taking, friendship development, anger management, interpersonal problem-solving, following school rules, and school success.

A brief description of the teacher program is provided below. The parent and child components of the series are described in separate write-ups.

Teacher Classroom Management Program

The teacher training series is a group-based program for teachers, school counselors, and psychologists. Participants receive 6 days (42 hours) of training spread throughout the academic year. The training targets teachers' use of effective classroom management strategies such as: the use of differential attention, academic persistence, social and emotional coaching, praise and encouragement, incentives to increase targeted positive behaviors, proactive teaching and positive discipline strategies to manage inappropriate classroom behaviors, and positive relationship-building skills. Additionally, teachers learn how to teach children empathy, social skills, emotional self-regulation and problem solving in the classroom. The program includes strategies for helping teachers to stay calm when dealing with difficult students, ways to build a support network with other teachers, and approaches to strengthen teachers' collaborative process and positive communication with parents (e.g., the importance of positive home phone calls, regular meetings with parents, home visits, and successful parent conferences). For indicated children (i.e., children with conduct disorders), teachers, parents, and group facilitators will jointly develop "transition plans" that detail classroom strategies that are successful with that individual child; and ways parents would like to be contacted by teachers. This information is passed on to the following year's teachers.

Additionally, teachers learn how to prevent peer rejection and bullying by helping the aggressive child learn appropriate problem-solving strategies and helping his/her peers respond appropriately to aggression. Physical aggression in unstructured settings (e.g., playground) is targeted for close monitoring, teaching and incentive programs.

Theoretical Rationale

Theoretical Rationale/Conceptual Framework for the Incredible Years Teacher Training Series

When children with behavior problems enter school, negative academic and social experiences make key contributions to the further development of conduct problems. Aggressive, disruptive children quickly become socially excluded. This leads to fewer opportunities to interact socially and to learn appropriate friendship skills. Over time, peers become mistrustful and respond to aggressive children in ways that increase the likelihood of reactive aggression. Evidence suggests that peer rejection eventually leads to these children's association with deviant peers. Once children have formed deviant peer groups, the risk for drug abuse and antisocial behavior is even higher.

Furthermore, Rutter and colleagues found that teacher behaviors and school characteristics such as low emphasis of teachers on academic work, low rates of praise, little emphasis on individual responsibility, and high student-teacher ratio were related to classroom aggressive behaviors, delinquency, and poor academic performance. High-risk children are often clustered in classrooms with a high density of other high-risk students, thus presenting the teacher with additional management challenges. Rejecting and nonsupportive responses from teachers further exacerbate the problems of aggressive children. Such children often develop poor relationships with teachers and receive less support, nurturing, and teaching and more criticism in the classroom. The lack of teacher support and exclusion from the classroom not only exacerbates these children's social problems, but also their academic difficulties and contributes to the likelihood of school dropout. Finally, research has recently shown that poorly managed classrooms have higher levels of classroom aggression and rejection that, in turn, influences the continuing escalation of individual child behavior problems. A spiraling pattern of child negative behavior and teacher reactivity can ultimately lead to parent demoralization, withdrawal and a lack of connection and consistency between the socialization activities of the school and home.

A preventive model needs to promote healthy bonds or "supportive networks" between teachers and parents, and children and teachers. Strong family-school networks benefit children due to parents' increased expectations, interest in, and support for their child's social and academic performance, and create a consistent socialization process across home and school settings. The negative cycle described above can be prevented when teachers develop clear classroom rules about bullying, prevent social isolation, and offer a curriculum which includes training students in emotional literacy, social skills, and conflict management. Considerable research has demonstrated that effective classroom management can reduce disruptive behavior and enhance social and academic achievement. Well-trained teachers can help aggressive, disruptive, and uncooperative children to develop the appropriate social behavior that is a prerequisite for their success in school.

Theoretical Orientation

  • Cognitive Behavioral
  • Behavioral
  • Attachment - Bonding
  • Social Learning

Brief Evaluation Methodology

Although the original (BASIC) Incredible Years parent program has been tested and evaluated numerously for more than 20 years, researchers have only more recently begun to test the effectiveness of the teacher component alone on child and teacher behavior outcomes. The main study, a cluster randomized controlled trial (RCT) in the UK conducted by Hutchings et al. (2013) found significant program effects on both teacher classroom strategies and child problem behaviors. Other cluster RCT studies in Jamaica (Baker-Henningham et al., 2012) and Ireland (McGilloway et al., 2010) have also found significant favorable outcomes both for teachers and students. The teacher component has also been tested with some effectiveness in additional studies in Wales (Hutchings et al., 2007) and in Chicago, IL (Raver et al., 2008), including implementation with mental health professionals to support teachers in multiple Head Start centers in which parents were also encouraged to attend the parent training program (Williford & Shelton, 2008). Additional studies conducted in the US have also evaluated the additive effects of teacher-training in combination with parent training, child training or both (Webster-Stratton, Reid & Hammond, 2004; Reid, Webster-Stratton & Hammond, 2003; Webster-Stratton, Reid, & Hammond, 2001). Teacher outcomes among both 3-year-old and 4-year-old samples in Head Start centers in 10 states and four regions across the US were examined in a RCT conducted by Hsueh et al. (2014) and Morris et al. (2014). Webster-Stratton, Rei & Stoolmiller (2008) conducted an experimental study with 120 Seattle Head Start programs and 14 elementary schools to examine outcomes among preschool, kindergarten and first grade teachers. Using the same data set, Thompson et al. (2017) conducted an analysis on a sub-set of kindergarten teachers and their students entering kindergarten to examine teachers' perception of parental involvement in school. Herman and Reinke (2017) conducted the same analysis as Thompson et al. (2017) but used a different sample of kindergarten through third grade teachers.

Kirkhaug et al., (2016) and Fossum et al. (2017) used identical evaluation designs. In both of these studies, the program was evaluated using an unmatched quasi-experimental pretest-posttest design. Schools were recruited from rural and urban municipalities in Norway, and schools that had previously implemented the program's parent training portion were eligible to apply for participation in the intervention group (n=21 in Kirkhaug et al., 2016 and 46 in Fossum et al., 2017). Another 22 schools (in Kirkhaug et al., 2016) and 46 schools (in Fossum et al., 2017) were recruited to act as the waitlist control group provided they did not have any recent or ongoing evidence-based school behavior intervention programs. Participating children were selected by randomly selecting 7 children from each participating classroom in grades 1-3 and screening them for externalizing behaviors. For Kirkhaug et al. (2016), only those who scored in the clinical range were invited to participate (n=83) whereas in Fossum et al. (2017) these students were identified as "high-risk" and analyzed in moderation analyses. Participants were assessed at baseline and at a follow-up approximately 9 months later, when 80% (in Kirkhaug et al., 2016) and 74% (in Fossum et al., 2017) of students were retained. Primary outcome measures in both studies were externalizing behaviors, social skills, and student-teacher relationships.

Outcomes (Brief, over all studies)

Evaluations of the teacher component alone show some benefit in improving classroom behavior problems, particularly among high-risk children, including reductions in conduct problems and improvements in social skills and problem-solving. There are also improvements in classroom management strategies among teachers, including increases in positive teaching techniques, and reductions in negative teaching techniques.

The teacher training (TT) program in combination with either the child training (CT) or parent training (PT) program, or both, resulted in a reduction in conduct problems at home (with both mothers and fathers) and at school (Webster-Stratton et al., 2004). The combination programs also reduced mother negative parenting. Father negative parenting was reduced and mother positive parenting increased, except in the CT+TT condition. Teacher negativity decreased in three of the combined conditions. Unfortunately, this study did not assess the effect of the teacher program alone.

Kirkhaug et al. (2016) assessed the effect of the teacher program, alone, and found that compared to the control group, participants in the intervention group had improved posttest academic performance and student-teacher conflicts, both teacher-reported measures.

Outcomes

Evaluations of the Incredible Years Teacher Classroom Management Program (IYTCM) show:

  • Significant decreases in conduct problems and other problem behavior among intervention children, including noncompliant and off task behavior, compared to control group children;
  • Significant improvements in behavior among high-risk intervention children, compared to control group children;
  • Significant decreases in disruptive behavior, when mental health consultants are used as support for trained teachers;
  • Significant reductions in conduct problems at home (with both mothers and fathers) and at school, when the IYTCM program is combined with other components of the Incredible Years Series.
  • Significant improvement in school readiness

Program Effects on Risk and Protective Factors:

  • Significant improvements in child self-regulation and cooperation skills;
  • Significant improvements in interpersonal skills, and reduction in stress and social impairments, among high risk children;
  • Increases in the use of positive classroom management strategies among program teachers and reductions in the use of negative classroom management strategies;
  • When used in combination with other components of the Incredible Years Series, there are significant reductions in negative parenting and increases in positive parenting, increases in teacher-parent (mother) bonding, and significant program effects on teacher classroom management skills.

Effect Size

Effect sizes were calculated for many of the studies. In the main study (Hutchings et al., 2013), classroom-level effect sizes on the significant outcome was large (.53 for off-task behavior). Effect sizes on significant program effects on teaching strategies and index child behavior were moderate (.36 for teacher negative strategies; .42 for child negatives to teacher and .48 for child off-task behavior). In the Jamaica study (Baker-Henningham et al., 2012), moderate to large effect sizes were found for all significant outcomes: observed child conduct problems (ES=.42) and observed friendship skills (ES=.74), teacher-reported behavior difficulties (ES=.47) and teacher-reported social skills (ES=.59), and parent-reported child behavior difficulties (ES=.22). In the Ireland study (McGilloway et al., 2010), effect sizes on significant outcomes also ranged from small to large. Small effect sizes were found for the following teacher observational measures: indirect and direct commands; large effect sizes (range from .21 to .77) were found on positive and negative strategies, praise, and time to comply with instructions and questions (no opportunity). Large effect sizes (range from .43 to .56) were also found on 3 teacher strategies scores (total positive strategies - frequency and usefulness, and inappropriate strategies - frequency). On teacher-reported child outcome measures for all index children, small effect sizes were found on 2 measures (peer problems and pro-social); moderate effect sizes were found on 2 measures (hyperactivity on both the SDQ and Conners measure); large effect sizes were found on 4 measures (emotional symptoms, conduct problems, and SDQ total and impact). As noted below, only the emotional symptoms measure had a statistically significant difference between groups. On the same measures for the high risk group only, effect sizes were small for hyperactivity (.05), moderate on 3 measures (conduct problems, pro-social, and Conners hyperactivity) and large on 4 measures (emotional, peer problems, and SDQ total and impact; range from .15 to .24).

Effect sizes across the remaining studies were small to large across a variety of outcomes, including teaching strategies, child behaviors, and teachers' perceptions of parental involvement in school.

Generalizability

The Incredible Years Teacher Classroom Management program has been tested in multiple locations with diverse populations across Europe (Norway, Ireland, and the UK), as well as one location in the Caribbean (Jamaica) with high-risk inner-city children. In the United States, the program has been demonstrated as effective for families living in poverty, such as children living in Head Start centers.

Potential Limitations

Many of the limitations related to the studies evaluating the Incredible Years Teacher Classroom Management Program as a standalone intervention overlap. Most of the evaluations used small sample sizes, which limits the statistical power of the results. Many of the studies also did not provide a fidelity measurement, so there is no way to know how well the teachers implemented the skills learned through training. Three studies (Webster-Stratton et al., 2001, Hsueh et al., 2014, and Morris et al., 2014) provided one-year follow-up data (one of these studies also included a parent training component), so it is difficult to determine whether any of the significant outcomes found post-intervention were sustained. However, evaluations of the teacher component alone are fairly recent, and it is possible that additional long-term follow-up studies have not been completed at this time. Not all of the studies have multiple reporting sources, such as independent observers to corroborate teacher-reported outcomes. Some of the studies targeted a very specific population (i.e., Head Start children), which limits the generalizability of the results. Three of the studies (Baker-Henningham et al., 2012, Raver et al., 2008 and Williford & Shelton, 2008) made adaptations to the original program model. In the Jamaica study (Baker-Henningham et al., 2012), the material was modified to make it culturally appropriate for the target population. Two of the studies (Raver et al., 2008 and Williford and Shelton, 2008) used mental health consultants to provide support to the intervention teachers. A few of the studies also did not use certified trainers to provide the teacher training (e. g. Willford and Shelton, 2008). Specific limitations for each study are listed below.

Hutchings et al., 2013: Several limitations were noted by the authors. First, the sample size was small, using only 12 classrooms and 107 index children. Second, the study targeted only children between the ages of 3 and 7, limiting the generalizability of results to the broader range of students. Third, about 10% of the parents of the highest TD-scoring children declined participation in the child-level measures. However, those children were part of the classroom-level measures. Fourth, there was no fidelity measurement on implementation, only training. Fifth, the TSDQ Total Difficulties variable was dichotomized for the analyses, potentially reducing the power within the analyses. There was also no follow-up conducted on this study to determine maintenance of program effects.

Baker-Henningham et al., 2012: This evaluation was limited in scope, only assessing children with elevated levels of conduct problems at school, so results are unknown for the population of students exhibiting low to moderate levels of behavior problems. There is no follow-up data to determine whether the effects of the program were sustained long-term, particularly when the children transitioned out to primary school.

McGilloway et al., 2010: There was a 7% dropout rate by follow-up. No mention was made in the summary regarding the potential differences between the participants who dropped out of the study versus those who completed the program. This study also suffered from a low level of statistical power, in that it used a small sample of schools and teachers. The only significant child outcomes were reported by teachers who received the intervention; child observations showed no significant differences. Another limitation of the this study was the fact that at the time of the post-intervention assessment, the teachers were still receiving training, representing the possibility that they were not able to fully implement the extent of their training in such a short period of time.

Hutchings et al., 2007: This study (Study 2) used a small sample size, which limits the power of the analysis. Also, there was a mix of trained and untrained teachers in all of the schools, which may have led to some contamination, with some untrained teachers picking up some of the classroom management strategies from trained teachers. This possibility was not controlled for in the analysis. There was also no baseline measurement of teaching strategies from the teachers, creating the possibility that differences between teachers was a result of sampling bias, although the large effect sizes likely precludes this possibility.

Raver et al., 2008: This study targeted a very specific population (Head Start funded programs in high poverty neighborhoods). The outcomes were measured at posttest, and follow-up would be needed to determine the lasting effects of the implementation. Importantly, the addition of the Mental Health Consultants (MHC) was an adaptation to the standard Incredible Years program, which may have contributed to the positive results. A study examining the differences between implementing the intervention by itself and using an MHC in coordination with the treatment would be necessary to tease out the differences in the impact of the program. Also, teachers were not trained by instructors certified in the program, and fidelity measures were not collected to determine how closely the teachers folllowed the program curriculum. The average amount of time spent by teachers in training (18 hours) was less than half the recommended dosage (42 hours).

Williford & Shelton, 2008: Assignment of Head Start programs was not random, which limits the validity of the results. Additionally, caregivers were encouraged and not required to attend the parent training component, and an analysis was not conducted to determine the differential impact of the program among caregivers who participated in parent training and those who did not. Since parent training was provided to 35% of the parents, this is not an evaluation of teacher training alone. Teachers were not trained by instructors certified in the program, and fidelity measures were not collected to determine how closely the teachers followed the program curriculum. Another limitation of this study is that it relied solely on teacher and caregiver reports, and did not include independent observations of child behavior. Finally, high attrition rates at follow-up limit the generalizability of the results, and attrition was differential with higher losses among children with higher levels of ADHD and ODD behaviors.

Davenport & Tansey, 2009: Nonsignificant results on the Teacher Strategies Questionnaire may have been due to the high number of non-responses to various items, which may have weakened the statistical power of results. Another limitation in this study is the lack of control group and the small sample size, which limits the generalizability of the findings. The SDQs were analyzed as if they had been derived from an unrelated group. There may have been teacher factors influencing the group score. The small n precluded analysis of such factor effects.

Webster-Stratton et al., 2003:

  • Few additive effects when compared with treatments delivered in single or dual risk domains.
  • Follow-up assessments at one-year and two-year do not include the control group, which makes it difficult to determine sustained effects of the program.
  • Sample is fairly homogeneous (primarily white, two-parent, middle-income families), and who were motivated to bring their children to a clinic for treatment, which limits the generalizability of the study.
  • Sample size in each treatment condition is small, thus limiting the statistical power.
  • Attrition is low, but may differ by level of child's behavior problems.

Webster-Stratton, Reid, & Hammond, 2001:

  • The study was only able to recruit 50% of the eligible families enrolled in Head Start which limits the generalizability of the study.
  • The experimental and control groups differed on several demographic and outcome variables at baseline.
  • The study suffered from differential attrition.
  • The study randomized classrooms but analyzed individuals.

Hsueh et al., 2014; Morris, P., Mattera et al., 2014:

  • Among the 3-year-old sample, all measures are teacher-reported
  • Some evidence of differential attrition, but possibly similar across conditions
  • Weak evidence of long-term effects in tests for 4-year-olds

Webster-Stratton et al., 2008:

  • Sample sizes not always clear
  • Demographic measures equivalent at baseline, but outcome measures not reported
  • Differential attrition not discussed
  • Weak posttest effects for child conduct

Thompson et al., 2017. - same data as Webster-Stratton et al., 2008 so the same limitations. In addition:

  • No measure of a behavioral outcome included
  • Although schools in Webster-Stratton et al. (2008) were randomized, it is not clear that the subsample was randomly assigned
  • Teachers both delivered the program and rated parental involvement
  • Analysis at teacher level, but randomization appears to be at school level
  • No report of baseline equivalence for demographic measures and one significant difference for outcome measures
  • Small effects for teacher perception of parental involvement in school (protective factor)

Reinke, W. M., Herman, K. C., Dong, N., 2016:

  • Teachers both delivered the program and rated student behavior
  • No tests for differential attrition
  • Missing information on baseline equivalence
  • Effects only for teacher-rated behavioral outcome measures

Herman & Reinke, 2017:

  • No measure of a behavioral outcome included
  • Teachers both delivered the program and rated parental involvement
  • Reliability of outcome measures were only provided for the control group
  • Tested baseline equivalence for outcome measures but not for socio-demographic variables
  • No report of differential attrition
  • Sample is from one urban area and may not be generalizable

Kirkhaug et al., 2016:

  • QED with no matching
  • Teachers implement the program and rate all measures
  • Differential attrition of treatment group
  • No impacts on independently measured behavioral outcomes

Fossum et al., 2017

  • QED with no matching
  • Teachers implement the program and rate all measures
  • Not clear whether the analysis controlled for clustering or pretests
  • Some evidence of baseline differences
  • Some evidence of differential attrition

Endorsements

Blueprints: Promising
Crime Solutions: Effective
OJJDP Model Programs: Effective
SAMHSA : 3.6-3.7

Peer Implementation Sites

Contact Person: Judy Ohm or Angie Clair
Organization Name: Wilder Foundation, Parent Education Center
Address: 451 Lexington Pkwy. North, St. Paul, MN 55104
Phone: 651-280-2606
Email: Judy.Ohm@wilder.org
Organization URL: Wilder.org

Program Information Contact

Malaysia Guzman, IY Solutions Consultant
Incredible Years, Inc.
1411 8th Avenue West
Seattle, WA 98119 USA
phone: 1-843-284-2216
malaysiag@incredibleyears.com
www.incredibleyears.com

References

Study 1

Certified Hutchings, J., Martin-Forbes, P., Daley, D., & Williams, M. E. (2013). A randomized controlled trial of the impact of a teacher classroom management program on the classroom behavior of children with and without behavior problems. Journal of School Psychology, 51, 571-585.

Study 2

Certified Baker-Henningham, H. Scott, S., Jones, K., & Walker, S. (2012). Reducing child conduct problems and promoting social skills in a middle-income country: Cluster randomised controlled trial. The British Journal of Psychiatry, 1-8. doi: 10.1192/bjp.bp.111.096834

Study 3

Certified McGilloway, S., Hyland, L., Mhaille, G. N., Lodge, A., O'Neill, D., Kelly, P., . . . Donnelly, M. (2010). Positive classrooms, positive children: A randomised controlled trial to investigate the effectiveness of the Incredible Years Teacher Classroom Management programme in an Irish context (short-term outcomes). A Summary Report for Archways.

Study 4

Hutchings, J., Daley, D., Jones, K., Martin, P., Bywater, T., & Gwyn, R. (2007). Early results from developing and researching the Webster-Stratton Incredible Years Teacher Classroom Management Training Programme in North West Wales. Journal of Children's Services, 2, 15-26.

Study 5

Raver, C. C., Jones, S. M., Li-Grining, C. P., Metzger, M., Smallwood, K., & Sardin, L. (2008). Improving preschool classroom processes: Preliminary findings from a randomized trial implemented in Head Start settings. Early Childhood Research Quarterly, 23, 10-26.

Study 6

Williford, A. P., & Shelton, T. L. (2008). Using mental health consultation to decrease disruptive behaviors in preschoolers: Adapting an empirically-supported intervention. Journal of Child Psychology and Psychiatry, 49, 191-200.

Study 7

Davenport, J., & Tansey, A. (2009). Outcomes of Incredible Years classroom management training with multiple schools. Manuscript submitted for publication.

Study 8

Reid, M . J., Webster-Stratton, C., & Hammond, M. (2003). Follow-up of children who received the Incredible Years intervention for oppositional defiant disorder: Maintenance and prediction of 2-year outcomes. Behavior Therapy, 34,471-491.

Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child and Adolescent Psychology, 33(1), 105-124.

Study 9

Webster-Stratton, C., Reid, M. J., & Hammond, M. (2001). Preventing conduct problems, promoting social competence: A parent and teacher training partnership in Head Start. Journal of Clinical Child Psychology, 30(3), 283-302.

Study 10

Hsueh, J., Lowenstein, A. E., Morris, P., Mattera, S. K., & Bangser, M. (2014). Impacts of social-emotional curricula on three-year-olds: Exploratory findings from the Head Start CARES demonstration. Report from the Office of Planning, Research, and Evaluation, U.S. Department of Health and Human Services.

Mattera, S., Lloyd, C. M., Fishman, M., & Bangser, M. (2013). A first look at the Head Start CARES demonstration: Large-scale implementation of programs to improve children's social-emotional competence. OPRE Report 2013-47. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Morris, P., Mattera, S. K., Castells, N., Bangser, M., Bierman, K., & Raver, C. (2014). Impact findings from the Head Start CARES demonstration: National evaluation of three approaches to improving preschoolers' social and emotional competence. Report from the Office of Planning, Research, and Evaluation, U.S. Department of Health and Human Services.

Study 11

Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the Incredible Years Teacher and Child Training Programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49,471-488.

Thompson, A. M., Herman, K. C., Stormont, M. A., Reinke, W. M., & Webster-Stratton, C. (2017). Impact of Incredible Years® on teacher perceptions of parental involvement: A latent transition analysis. Journal of School Psychology, 62, 51-65.

Study 12

Reinke, W. M., Herman, K. C., & Dong, N. (2016). The Incredible Years Teacher Classroom Management Program: Outcomes from a group randomized trial. Unpublished manuscript. http://incredibleyears.com/wp-content/uploads/Reinke-IY-TCM-Program-Outcomes.pdf

Study 13

Herman, K. C., & Reinke, W. M. (2017). Improving teacher perceptions of parent involvement patterns: Findings from a group randomized trial. School Psychology Quarterly, 32(1), 89.

Study 14

Kirkhaug, B., Drugli, M. B., Handegard, B. H., Lyderson, S., Asheim, M., & Fossum, S. (2016). Does the Incredible Years Teacher Classroom Management Training programme have positive effects for young children exhibiting severe externalizing problems in school?: A quasi-experimental pre-post study. Boston Medical Center Psychiatry, 16(1), 362.

Study 15

Fossum, S., Handegård, B.H. & Britt Drugli, M. J. (2017). The Incredible Years teacher classroom management programme in kindergartens: Effects of a universal preventative effort. Journal of Child and Family Studies, 26(8), 2215-2223.

Study 1

Evaluation Methodology

Design: A single-blind stratified cluster RCT design was used. An independent researcher paired classrooms according to school size, classroom size, and locality after pre-intervention measures were collected. Schools were randomly assigned to condition (intervention or control) and the paired school was automatically assigned to the control group in each case. Participants were teachers and their students from 12 classes (6 intervention and 6 control) from 11 primary schools in rural and urban areas of North West Wales (county of Gwynedd). Schools were recruited by a Gwynedd Education Service official. Class sizes ranged from 14 to 30 students, and students were between three and seven years old. Data was collected from eight full-time teachers and four job-share teachers (teachers who shared teaching duties with another teacher). Teachers completed the Teacher Strengths and Difficulties Questionnaire (TSDQ) to rate their students' behavioral, emotional, and social difficulties, using four subscales (hyperactivity problems, peer-problems, emotional problems, and conduct problems) to produce a Total Difficulties (TD) score. The 107 students targeted and recruited for assessment (index children) were chosen based on their TSDQ scores: ideally, those having the three highest, three lowest, and three mid-point scores, for a total of 9 children per class (one class only had 8 recruited students). In actuality, 80 children (75%) had scores in the low scoring range on the TSDQ, while 27 children (25%) had scores within the high scoring range. For data analysis, the target children were divided into two groups according to TSDQ guidelines, where there were 18 children designated to the high-scoring group, meaning their scores rated above the point of clinical concern.

Teachers in the intervention condition received one full day training session per month for five months, conducted by two trained leaders - a trained program leader and a certified program mentor. Nearly all (97%) of the teachers attended all five training sessions. Teachers in the control condition received no in-service or professional development opportunities during the study.

Sample Characteristics: Teacher characteristics: all teachers were female, with a mean age of 34 years and an average of nine years of teaching experience. Student characteristics: The mean percentage of students who were eligible for free school meals was 13% (slightly below the national average of 17%). Of the 107 participating children, 58 were boys and 49 were girls, with a mean age of 57.51 months. All children were Caucasian. All parents had to sign a consent for their children to be observed, and where consents were not given, the next highest scoring child was chosen. All participants received an incentive for participating in the study.

Measures: Outcome measures were collected at baseline and post-intervention. The Teacher-Pupil Observation Tool (T-POT) was used to measure the frequency counts of teacher and child behaviors through direct observation. Eight composite categories were used: teacher positives, teacher negatives, compliance, non-compliance, negative behaviors towards the teacher, prosocial behavior, deviance, and off-task behavior. This measure has good inter-rater reliability (78%). The T-POT measure was collected by three postgraduates with experience using similar observation measures. Observations were conducted during structured lessons, each child was observed for 15 minutes. At the same time, observers were coding teacher interactions with the class as a whole and class responses to the teacher and behavior generally. Coders were blind to the conditions they were observing.

Analysis: Independent t-tests were used to determine baseline equivalence. Analyses were intent-to-treat, using full maximum estimation likelihood for missing data. Analysis on outcome measures used hierarchical linear modeling to account for the clustered nature of the data, with post-intervention T-POT scores as dependent variables and intervention status as a classroom level random factor. Effect sizes were also calculated.

Outcomes

Fidelity Monitoring: The trainers for this study were certified to train the Teacher Classroom Management program. They also completed fidelity checklists (Teacher Workshop Checklist) following each training session. Participating teachers completed a satisfaction questionnaire at the end of implementation and teacher workshop evaluations at the end of each training workshop.

Baseline Equivalence and Differential Attrition: Observational data was available on all children at baseline and all but one student (from the control group) at post-intervention. All teachers were retained for the duration of the study. There were no significant baseline differences in child, teacher, or school demographics, nor were there differences in the number of children in the low and high TSDQ groups between conditions at pre-intervention.

Posttest:
Classroom-level findings: On teacher and classroom behavior observational measures, only one of eight measures was significant: classroom off-task behavior. Effect size (ES) for this outcome was large (.53).

Child-level findings: There were three significant program effects on teacher and index children's behavior, with moderate effect sizes: reductions on teacher negatives (ES = .36), child negatives to the teacher (ES = .42) and child off-task behavior (ES = .48). The intervention also had a significant effect on child compliance to commands (ES = .37). Further analysis on this measure showed a significant reduction in teacher commands to intervention group children (ES = .48), whereas teachers in the control group showed an increase in total number of commands. Also, children in the intervention group showed an increase from 68% to 81% in compliance to commands, with no change among control group children. When comparing the low and high Total Difficulties (TD)-scoring children, at post-intervention, there was a significant interaction effect on two measures: child negatives to teachers and child off-task behavior. The intervention was more beneficial to children who initially scored high on the TSDQ Total Difficulties, whereas children who scored low followed the same pattern from pre- to posttest as children in the control group (increases in child negatives to teacher and no change in child off-task behavior).

Brief Bulleted Outcomes

Classroom-level Outcomes:

  • Significant decreases in intervention classroom off-task behavior, compared to control room classrooms.
  • Large effect sizes on significant outcome: .53 for off-task behavior.

Child-level Outcomes:

  • Significant program effects on teacher negative strategies, compared to control teachers (effect size = .36).
  • Significant reductions in child negatives to teacher and child off-task behavior among intervention group index children, compared to control group index children (effect sizes = .42 and .48, respectively).
  • Significant reductions on teacher commands to intervention group children (ES = .48), compared to control group teachers, and increases among intervention children in compliance to commands (from 68% to 81%), compared to children in the control group, who showed no change.
  • Significant program interaction effect on child negatives to teacher and child off-task behavior among intervention children in the high Total-Difficulties-scoring group, compared to intervention children in the low Total-Difficulties-scoring group and control group children.

Limitations: Several limitations were noted by the authors. First, the sample size was small, using only 12 classrooms and 107 index children. Second, the study targeted only children between the ages of 3 and 7, limiting the generalizability of results to the broader range of students. Third, about 10% of the parents of the highest TD-scoring children declined participation in the child-level measures. However, those children were part of the classroom-level measures. Fourth, there was no fidelity measurement on implementation, only training. Fifth, the TSDQ Total Difficulties variable was dichotomized for the analyses, potentially reducing the power within the analyses. There was also no follow-up conducted on this study to determine maintenance of program effects.

Study 2

Evaluation Methodology

Design: Twenty four community pre-schools in inner-city areas of Kingston, Jamaica were recruited to participate in the study, which took place from October, 2009 until June, 2010. Criteria for schools to be included in the study were: there were at least three to four classes of children with at least 20 students per class, schools needed to be situated in a specified geographical area and all teachers needed to consent to participate. Of the 50 schools that were initially approached, 24 met the criteria and were recruited to participate. Teachers completed an interviewer-administered screening questionnaire for each child (10 items measuring conduct disorder on the ICD-10 Classification of Mental and Behavioral Disorders), to determine which children were at high risk for subsequent externalizing problems. The three children from each class with the highest scores were enrolled to participate in the evaluation. Children were excluded from the study if they had attended school less than 70% of the time, had a developmental disability, lived in an institution, or were a sibling of an enrolled child. Based on this criteria, 24 children were excluded, and replaced with the next highest-scoring children. In total, 225 children from 73 classrooms were included in the analysis (113 intervention, 112 control).

Randomization occurred at the school level to either the intervention or control group. Each school contained one class per age group of 3-, 4-, and 5-year old children (one school had an additional class).

All intervention school teachers and principals received eight days of training for the Incredible Years Teacher Training Program, and training included adapting the program to be culturally appropriate. Adaptations included a) using culturally appropriate videos (Jamaican classrooms), b) training occurred over 8 days instead of 5-6 days, c) adding exercises and activities and emphasizing certain skills-building modules that were designed based on the experiences and concerns of the teachers, d) placing more emphasis on certain aspects of the intervention based on teacher feedback, and e) making handouts culturally appropriate. Adaptations were intended to increase teachers' self-confidence and their understanding and application of the skills introduced in the program and to enhance their ability to generalize these skills in their classes. Beyond training and to ensure fidelity, teachers received assistance with modeling, coaching, and other support and feedback each month for four months. Education materials were also provided to both intervention and control teachers on teaching strategies. Control group teachers also attended the regular in-service workshops provided by the Ministry of Education.

Sample Characteristics: The majority of the children in the study were male (59% in intervention, 63% in control), and the average age was 4.2 years. Most children lived with their mother, although less than half had a father living at home. Less than half of caregivers had completed high school. Most of the teachers in the study were female.

Measures: Data was collected at baseline and post-intervention. Researchers who collected the data were blind to condition. The primary outcome measure - directly observed in-class child behavior, was measured by observing each child for 5 minutes on a rotational basis (with 3 children per class) for a total of 15 minutes per day per child over 4 days and across different times of the school day, which resulted in 1 hour of observation. Event recording was used to count aggressive/destructive behaviors and friendship skills (in frequencies per hour). The behaviors were chosen based on the Dyadic Parent-Child Interaction Coding System and Multi-Option Observation System for Experimental Studies behavior categories. Observers rated the frequency of conduct problems, activity level, on-task behavior and follows rules/expectations of the classroom using seven-point scales.

Secondary outcome measures on child behavior were collected from teachers and parents. Teacher-reported measures included the Sutter-Eyberg Student Behavior Inventory (SESBI) to measure child conduct problems; the Conner's Global Index to measure hyperactivity and attention difficulties; the Strengths and Difficulties Questionnaire (SDQ) to measure behavior difficulties and prosocial skills, and the Preschool and Kindergarten Behavior Scales (PKBS): Social Skills Scale to measure social skills. Parent-reported measures included the Eyberg Child Behavior Inventory (ECBI) and the SDQ.

Also measured was child attendance (taken from school records) and parents' attitude to school, which was measured through a 10-item questionnaire. This measure was only collected at post-intervention.

Analysis: Analysis was conducted using multilevel multiple regression models on an intent-to-treat basis.

Outcomes

Baseline Equivalence and Differential Attrition: There were no significant differences between groups on child, family, classroom or school characteristics at baseline. At post-intervention, data from 15 children lost at follow-up was missing. The authors reported no significant differences between those who dropped out of the study and those who remained, but no statistics were presented.

Posttest: Significant outcomes were found on observed child behavior, including reductions in conduct problems and increased friendship skills. The intervention also had a significant effect on teacher-reported child behavior difficulties and social skills and parent-reported child difficulties. There were significant differences between groups on the proportion of children being rated in the clinical range for conduct problems, with fewer children in the intervention group being rated by teachers (but not parents) as in the clinical range at post-test, compared to control group children, and less likely to have pervasive conduct problems. Official records showed better child attendance for intervention children.

Brief Bulleted Outcomes

  • Significant positive program effects on conduct problems and child behavior difficulties.
  • Significant improvements in friendship and social skills among intervention group children, compared to control group children.
  • Significant reduction in numbers of intervention group children rated as clinical on conduct problems by both teachers and parents, compared to control group children.

Limitations: This evaluation was limited in scope, only assessing children with elevated levels of conduct problems at school, so results are unknown for the population of students exhibiting low to moderate levels of behavior problems. There is no follow-up data to determine whether the effects of the program were sustained long-term, particularly when the children transitioned out to primary school.

Study 3

Evaluation Methodology

Design: Eleven schools in or around the Limerick, Ireland area were recruited to participate in the study. Two teachers from each of the schools (n=22) agreed to participate. The teachers were randomly assigned to either the treatment or control condition on a 1:1 ratio for each school. At baseline, each teacher completed the Teacher Strengths and Difficulties Questionnaire (T-SDQ) for each child in their class (total n=445). Class sizes ranged from 11 to 29 students. From these results, a cross-section of index children (12 children per classroom) was determined for inclusion in the analysis. Based on the 'total difficulties' measure of the T-SDQ, the index children represented the four highest scoring, four lowest scoring, and the four middle-scoring children from each class.

Children were excluded from the study if they had a formal diagnosis of a developmental disorder, if parental consent forms were not returned, or if they were absent from school on more than one occasion during baseline assessment, yielding a baseline population of 234 index children. Seventeen children (7%) were lost to follow-up, so analyses for the study only included the remaining 217 (107 control group and 110 intervention) index children.

Sample: The majority (n=7) of the participating schools were designated as 'disadvantaged,' meaning they received a greater level of support in terms of student-teacher ratios, special grants, and extra support for students. Participant teachers in the study had been assigned either to a Junior or Senior Infant class. Most teachers (n=15) were between the ages of 25 and 34 years, had spent an average of 9 years teaching, with an average of almost 5 of those years teaching infant classes, and all but one were female. Of the total population (n=445) of children (ages 4-7) taught by the study teachers, teachers indicated that about 1/4 showed significant levels of social, emotional, and/or behavioral difficulties at baseline, and 42% were rated by teachers to be above the recommended cut-off indicating the presence of hyperactivity. The index population of children included 102 boys and 115 girls.

Measures: The Teacher Strengths and Difficulties Questionnaire (T-SDQ) and the Conners Abbreviated Teacher Rating Scale were used to measure the nature and severity of child emotional and behavioral problems, which included: emotional symptoms; conduct disorder; peer-relationship problems; and hyperactivity in the classroom. In addition, observational measures were captured to evaluate classroom management and child behavior outcomes using the Teacher-Pupil Observational Tool (T-POT). This tool measured teacher-pupil interactions in the classroom, and frequency counts of teacher behaviors and index child positive and negative behavior. Reliability of 73% was achieved and where possible, the research team was blind to the condition.

A self-reported teacher measure, the Teacher Strategies Questionnaire, was used to collect data relating to teachers' confidence in managing challenging behavior in the classroom, as well as their use of positive strategies, negative classroom management strategies, and strategies to improve home-school links. The questionnaire also captured teacher perceptions of the utility of the classroom management strategies. Intervention teachers also completed a Teacher Satisfaction Questionnaire at follow-up to assess their perceptions of the intervention.

Analysis: Changes in teachers' classroom strategies from baseline to follow-up were assessed by computing mean differences in teacher scores. Independent sample t-tests were then used to examine significance. ANCOVA was used to assess teachers' self-reported use of classroom management strategies. Analyses of child outcomes were conducted at the classroom level. Regression modeling was used to examine pre- and post-test differences in child behaviors, adjusting for clustering effects. Effect sizes were calculated for both teacher and child outcomes. Sub-analyses on child outcomes for the index group divided the children into two groups: 1) a 'high risk' group (children who scored a 12 or higher on the SDQ 'total difficulties' measure; n=63), and 2) a 'low risk' group (children who scored 11 or lower on the SDQ 'total difficulties' measure; n=154). Analysis of child outcome measures was conducted using robust regression analysis, controlling for clustering.

Outcomes

Posttest: At post-intervention, there was a significant reduction in observed incidence of teacher negatives in the intervention group, with intervention teachers using significantly fewer harsh and critical statements, as well as fewer negative commands in their interactions with students, when compared to control group teachers. There was a significant increase in giving more time to students to comply with instructions and questions among intervention teachers compared to control teachers. Measures approaching significance included use of positive classroom management strategies and use of direct commands among intervention condition teachers, compared to control condition teachers.

On self-reported use of classroom management strategies at post-intervention, there were significant between-group differences favoring the intervention group, indicating that teachers using the Incredible Years Teacher Classroom Management Program found the classroom management strategies both easy to implement and useful. The intervention group of teachers also reported significant decreases in frequency of use of inappropriate strategies for managing misbehavior, compared to control group teachers, who actually reported some increases in frequency and perceived usefulness of these strategies. Intervention group teachers were also generally satisfied with the IY program, indicating that the program was appropriate for managing classroom behavior, and 64% indicated they would recommend the program to other teachers. All of the intervention teachers also indicated that they felt more confident in their ability to manage classroom behavior problems.

There were a few child outcomes as reported by teachers to note. Intervention group children showed significant improvements in emotional symptoms (self-regulation and cooperation skills), compared to control group children. There were intervention group decreases in total difficulties score on the Strengths and Difficulties Questionnaire, and decreases in conduct problems approached significance, compared to scores from the control group. Hyperactivity, peer problems, and prosocial behavior did not differ.

In the sub-analysis of index children according to levels of behavioral need, there was a significant impact of the program on the high risk group of intervention children on total behavioral difficulties, peer problems (improved interpersonal skills), and distress and social impairment, compared to control group high risk children, as reported by teachers.

Outcomes from classroom observations of the index children were not statistically significant, but showed moderate effect sizes (described above), for intervention group children on measures of negative responses to teacher-initiated behaviors and non-compliant behavior.

Brief Bulleted Outcomes

Teacher outcomes:

  • Intervention group teachers showed significant post-intervention reductions in the use of negative teaching strategies (harsh and critical statements, negative commands), compared to control group teachers.
  • Intervention group teachers significantly increased their use of positive teaching strategies (giving more time to comply with instructions and questions), compared to control group teachers.
  • Intervention group teachers reported significant reductions in the use of inappropriate strategies for managing classroom misbehavior, compared to control group teachers.

Child outcomes:

  • Significant improvements in emotional symptoms (self-regulation and cooperation skills) among intervention children, compared to control group children, as reported by teachers.
  • Significant teacher-reported positive program impact on high risk intervention children on behavioral difficulties, peer problems (interpersonal skills), distress, and social impairment, compared to their control group counterparts.

Limitations: There was a 7% dropout rate by follow-up. No mention was made in the summary regarding the potential differences between the participants who dropped out of the study versus those who completed the program. This study also suffered from a low level of statistical power, in that it used a small sample of schools and teachers. The only significant child outcomes were reported by teachers who received the intervention; child observations showed no significant differences. Another limitation of the this study was the fact that at the time of the post-intervention assessment, the teachers were still receiving training, representing the possibility that they were not able to fully implement the extent of their training in such a short period of time.

Study 4

Evaluation Methodology

Design: For the first study, two Teacher Classroom Management (TCM) courses were conducted in 2003-04 with a total of 23 teachers (11 in one and 12 in the other). Teachers were nominated by their employers (two Authorities in North West Wales) to participate in the training, and attended either because they were interested, or their employer identified a need. Courses ran for one day per month for five months and teachers received feedback on classroom assignments in between course classes. The majority of teachers (65%) attended all training sessions.

The second study used 21 classrooms in three schools in Gwynedd. Schools were recruited to participate in the study. Observers were blind to condition, and each school had classrooms where teachers had and had not completed TCM training. Teacher and student behavior was coded continuously and simultaneously during 30-minute sessions. Each teacher was observed for one 30-minute session.

Sample Characteristics: All but one of the teachers was female and had an average age of 37 years, and teachers had been teaching an average of 17 years.

Measures: In the first study, data collection measures included the Teacher Satisfaction Questionnaire (TSQ), which measured four areas: usefulness of the program, confidence in using the ideas, difficulty/ease of putting the program into practice, and use of strategies for improving home-school links. A total satisfaction score was also calculated. TSQ data was collected from 20 teachers. Also collected was qualitative data, conducted via interview with 21 (91%) of the teachers once the course was completed and teachers had implemented the program in their classrooms. Two independent coders coded the transcripts with high inter-rater reliability (k = .82).

The observational component of the second study took place with 21 teachers from three primary schools in Gwynedd, where 10 teachers were trained in the TCM program and 11 were not. Measures were collected using the Teacher-Pupil Observation Tool (T-POT), which measured 12 teacher behaviors and 17 child behaviors. Scoring consisted of frequency counts of each behavior. The 29 items were then reduced to six global teacher and four child categories: Teacher measures - teacher positive, teacher negative, teacher praise, teacher indirect command, teacher direct command, and no opportunity given for compliance; Child measures - child positive behavior, child negative behavior, child compliance, and child non-compliance. Three coders were used to complete the observations. Observer scores were comprised of frequency counts of each behavior. Inter-rater reliability on the scoring was 70%.

Analysis: Mean scores for the first study were calculated with the TSQ data, ratings were on a scale of 1-5. In the second study, a Mann-Whitney U test was performed to determine differences between groups. Effect sizes were also calculated.

Outcomes

Posttest: In the first study, teachers indicated a high level of overall satisfaction with the program for all scales examined: usefulness of the program (4.5), confidence in delivering the program (4.6), ease of putting the program into practice (4.8), use of strategies to improve home-school links (3.5) and total satisfaction (4.2). The qualitative interview also revealed, overall, high satisfaction with the program, with teachers reporting benefits both to themselves as teachers (increased range of effective strategies and greater use of existing positive strategies) and their students (increased attention, more considerate of others and increased confidence). Teachers also indicated making use of the teaching strategies taught, with few barriers to implementation reported, and that the training helped them become better equipped to deal with difficult child behaviors. Most (91%) teachers indicated they would recommend the program to others.

Observational outcomes from study 2 indicate significant program effects on four of ten observed measures (2 teacher and 2 child). Specifically, teachers trained in the TCM program had significantly higher rates of using direct commands with children, and gave children significantly more time to comply with commands, compared to teachers who were not trained. For children, students in classrooms where the teacher was trained in TCM showed significantly less non-compliant behavior and significantly more positive behaviors, compared to students of untrained teachers. The effect sizes on the significant measures were large (direct commands = .94, no opportunity = .79, non-compliance = 1.13, child positives = .99).

Brief Bulleted Outcomes

  • Significant program effects for trained teachers on increasing use of direct commands with children, and giving children ample time to comply with commands, compared to untrained teachers (effect sizes = .94 and .79, respectively).
  • Significant differences among children in intervention classrooms on non-compliant behavior and positive behaviors, compared to children in non-intervention classrooms (effect sizes = 1.13, .99, respectively).

Limitations: This study (Study 2) used a small sample size, which limits the power of the analysis. Also, there was a mix of trained and untrained teachers in all of the schools, which may have led to some contamination, with some untrained teachers picking up some of the classroom management strategies from trained teachers. This possibility was not controlled for in the analysis. There was also no baseline measurement of teaching strategies from the teachers, creating the possibility that differences between teachers was a result of sampling bias, although the large effect sizes likely precludes this possibility.

Study 5

Evaluation Methodology

Design: This study was part of the Chicago School Readiness Project (CSRP), which provided teachers in Head Start classrooms in Chicago with weekly coaching support as part of their workforce development and adult training. Head Start sites that offered full day programming (2 or more classrooms) were recruited from seven urban neighborhoods in Chicago that had high rates of poverty. There were a total of 18 participating sites, which included 35 classrooms. Sites were randomized using a pair-wise matching procedure. The Incredible Years Teacher Classroom Management intervention was implemented with two cohorts of teachers and their classroom students, with cohort 1 implementation occurring from fall to spring 2004-05 and cohort 2 implementation occurring from fall to spring 2005-06. There were 94 participating teachers, and 65 (69%) consented to complete the teacher surveys. Four of the teachers either moved or quit during the school year, yielding a final dataset of 90 teachers. Intervention teachers were asked to participate in five six-hour long training sessions. Each teacher spent an average of 18 hours in training.

Each classroom receiving CSRP intervention services was assigned a weekly Mental Health Consultant (MHC) who attended all 5 teacher training sessions with the teaching staff and were available in the classrooms one morning a week (for an average of 4.54 hours per week). These consultants, or coaches, provided encouragement and feedback to teachers on their use of the classroom management strategies taught through the training sessions. Coaches were also available to provide stress management for the teachers. The MHCs held a master's in Social Work. Through CSRP, targeted children also received one-on-one child-focused mental health consultations. Control group classrooms were assigned a Teacher's Aide for the same amount of time as the MHCs (averaging 5.18 hours per week).

At baseline, there were 543 children attending Head Start and participating in the study. By the spring, there were 509 children participating (some children left while others entered the program and were added to the study).

Sample Characteristics: Demographic measures were only available for the 69% of teachers who completed surveys. Most teachers (97%) were female and the average age of teachers was 40. The majority of teachers were African-American (70%), followed by Latina (20%) and White (10%). Three quarters of the teachers held a post-secondary degree.

Participating children were those children for whom parental consent was obtained (M=91%). Classrooms consisted primarily of ethnic minority children (67% African American and 26% Latino). Classrooms averaged 15 to 16 children. Additional demographic information on the children was not provided.

Measures: Measures were collected at baseline and post-intervention. Data was collected using classroom observations. Twelve trained observers who were blind to condition collected classroom-level data using the Classroom Assessment Scoring System (CLASS) and the Early Childhood Environment Rating Scale, revised edition (ECERS-R). CLASS Observations were conducted one day a month in three sessions per day - breakfast, "circle time/free time" and lunch. The resultant data utilized the mean scores across the three observation sessions. The CLASS measure tested the impact of the intervention on classroom quality, using four scales: positive climate, negative climate, teacher sensitivity, and behavior management. Three quarters of the observations were double-coded by two observers to gauge inter-rater reliability, which ranged from .66 to .82. The ECERS-R was used to measure early childhood classroom quality. Items were scored on a scale from 1 to 7. ECERS-R data was also double-coded (43% of the time) to assess inter-rater reliability (.87).

Analysis: Analyses were intent-to-treat. Cohort membership was treated as a covariate in the analysis. Post-hoc repeated measures ANCOVA were conducted to yield covariate-adjusted estimates of means and standard deviations for treatment and control groups for the four dependent variables. Classrooms were nested within treatment and control groups, and Hierarchical Linear Modeling (HLM) was used to present class- and site-level outcomes. Effect sizes were also calculated.

Outcomes

Baseline Equivalence and Differential Attrition: There were no baseline differences between groups of teachers on any of the demographic or educational variables. There were also no baseline differences across sites on any of the teacher, child, or site measures assessed. Teacher attrition was low (5%) and child attrition reached 16%. Attrition analyses indicate that there were no statistically significant differences between the children who remained in the program compared to those who exited early. There were also no significant differences between the children who enrolled at baseline compared to those who enrolled later in the year, with the exception that those who were enrolled later were somewhat younger. There were also no significant differences between groups on remaining in the program compared to those exiting early, and only one significant difference between groups on those who began at baseline and those who were enrolled later, in that there were significantly more girls than boys who entered the control classrooms, compared to equal numbers of girls and boys entering the treatment classrooms late.

Posttest: Two of four classroom quality measures were significant: positive climate and negative climate, where intervention classrooms showed significant improvement compared to the control classrooms. Effect sizes were large (.89 for positive climate and .64 for negative climate).

Brief Bulleted Outcomes

  • Significant increases in positive classroom climate for intervention classes, compared to control classes (effect size = .89).
  • Significant decreases in negative classroom climate for intervention classes, compared to control classes (effect size = .64).

Limitations: This study targeted a very specific population (Head Start funded programs in high poverty neighborhoods). The outcomes were measured at posttest, and follow-up would be needed to determine the lasting effects of the implementation. Importantly, the addition of the Mental Health Consultants (MHC) was an adaptation to the standard Incredible Years program, which may have contributed to the positive results. A study examining the differences between implementing the intervention by itself and using an MHC in coordination with the treatment would be necessary to tease out the differences in the impact of the program. Also, teachers were not trained by instructors certified in the program, and fidelity measures were not collected to determine how closely the teachers folllowed the program curriculum. The average amount of time spent by teachers in training (18 hours) was less than half the recommended dosage (42 hours).

Study 6

Evaluation Methodology

Design: Head Start centers were assigned to either the intervention or control condition. Assignment was not completely random, but based on staffing and transportation needs. Participants were screened and recruited through Head Start registrations over a two year period. Eligible families included 96 caregivers and their 103 preschool-aged children. Eligibility requirements were not described in the article. Fifty-nine children were assigned to the intervention group and 37 children were in the comparison group. All participants received monetary incentives for completing the assessments, and comparison group families received resources for mental health services.

Nine teachers participated in the study (6 intervention and 3 control). Intervention lasted a minimum of four months. Intervention teachers received one introductory training session on the intervention, and then participated in weekly individualized consultations that focused on classroom management strategies, increasing the use of effective and proactive discipline, and strengthening positive teacher-child relationships. Adaptations to the program model included the use of mental health consultants, who were two advanced graduate level students in clinical psychology. Teachers received support based on their particular classroom needs, and therefore, not all teachers received consultation on all areas of the Teacher Training Series.

Intervention families were all encouraged to participate in a 10-session Incredible Years Parent Training program, and 35% (n=21) chose to participate (i.e., attended at least 50% of the sessions). There were no group differences in terms of the severity of caregiver or teacher reports of child behaviors. However, significantly more caregivers who attended the sessions were older, married, and experienced less parental distress.

Sample Characteristics: Most of the participants were African-American (86-92%) and mean age of the children was 4.5 years. The majority of children were boys (68-72%) and lived in single parent homes (76-84%). The Total Hollingshead score was 25.8 for the intervention group and 24.6 for the comparison group.

Measures: Measures were collected at baseline, post-intervention, and one year post treatment. Attrition was somewhat high. Sixty-three (66%) participants had a complete data profile (both caregiver and teacher reports) for the baseline and posttest time points, with the remainder having only caregiver (10%) or teacher (19%) assessments at posttest or neither (5%). One year follow-up data was collected from 63% of caregivers.

Specific child behaviors (ADHD and Oppositional Defiant Disorder/ODD) were measured using the ADHD-IV Rating Scale, which was completed by both caregivers and teachers. The ADHD scale was amended to include additional symptoms of ODD and Conduct Disorder (CD). Broad externalizing behaviors were measured using the Behavior Assessment System for Children (BASC), also completed by both caregivers and teachers. Caregivers completed 3 additional measures: the Parenting Stress Index-Short Form on 4 scales of stress: child; caregiver-child relationship, caregiver, and total; the Parenting Scale, which contained three scales: laxness, over-reactivity, and verbosity; and the Child Behavior Management Questionnaire, which measured caregiver knowledge of behavioral principles as they apply to parenting. Teachers completed the Teacher Strategies Questionnaire (TSQ), which measured effective teaching strategies, both on difficulty and usefulness, on two scales: positive strategies and limit setting. All measurement tools were reported to have high reliability ratings.

Analysis: Hierarchical Linear Modeling was used to determine change over time on measures. For teacher measures, a two-level random intercept model was used to estimate the hierarchical/organizational structure of the data where children were nested within classrooms. For caregiver measures, a series of three-level random intercept and slope hierarchical models were used. Three waves of data for each of the three time points were nested within children and children were nested within classrooms. Analyses were intent-to-treat.

To measure clinical improvement in children's disruptive behavior, an examination was made of the differences in the number of children who evidenced at least 1 SD improvement (at baseline) in their score for at least one measure of disruptive behavior for the intervention versus the comparison group.

Outcomes

Baseline Equivalence and Differential Attrition: At baseline, there were no significant differences between the intervention and comparison groups on any of the demographic variables, nor on the severity of child behavior problems or eligibility requirements. There were also no differences between intervention and comparison group classrooms, in terms of the quality of classroom environment, teacher qualifications/experience, or teacher-child ratios. Attrition analyses revealed no significant differences between those who completed the post-intervention assessment and those who did not, with one exception. Children without teacher post data were rated by teachers as having higher initial levels of ADHD and ODD behaviors than children with teacher post data. These children were also rated by their caregivers as having higher initial levels of ADHD and externalizing behaviors overall. For one-year follow-up data, there were no differences in the rates of non-completion by group or in baseline statistics between participants with follow-up data and those without.

Posttest: Outcomes on child disruptive behavior were mixed, in that teacher reports showed significant differences favoring the intervention group, while parent reports did not. Teacher measures also indicated significant program benefits on use of effective teacher strategies, compared to control group teachers. Among caregiver-only measures, two of five measures were significant at posttest: caregivers in the intervention group reported significantly decreased verbosity and increased knowledge of behaviorally-based child management strategies (behavior management), compared to caregiver reports in the comparison group.

Clinical improvement in child disruptive behaviors showed that caregivers in the intervention group reported 64% of children improved at least 1 SD on one measure of disruptive behavior, compared to 33% of the children in the comparison group. Teachers in the intervention group reported that 55% of the children improved at least 1 SD on at least one measure compared to 30% of children in the comparison group.

One-year Follow-up: Child behavior management differences among caregivers remained significant at the one-year follow-up, but program effects were lost on parenting skills (verbosity).

Brief Bulleted Outcomes

  • Significant decreases in intervention children's disruptive behavior, compared to children in the comparison group (teacher reports only).
  • Significant program effects on parenting behaviors: verbosity and child behavior management, as reported by intervention caregivers, compared to comparison group caregivers.
  • Significant clinical improvements in child disruptive behaviors, reported by both teachers and caregivers, with at least half of intervention children improving at least 1 SD, compared to only one-third of children in comparison group improving.
  • Program effects at one year on parent-reported child behavior management remained significant.

Limitations: Assignment of Head Start programs was not random, which limits the validity of the results. Additionally, caregivers were encouraged and not required to attend the parent training component, and an analysis was not conducted to determine the differential impact of the program among caregivers who participated in parent training and those who did not. Since parent training was provided to 35% of the parents, this is not an evaluation of teacher training alone. Teachers were not trained by instructors certified in the program, and fidelity measures were not collected to determine how closely the teachers followed the program curriculum. Another limitation of this study is that it relied solely on teacher and caregiver reports, and did not include independent observations of child behavior. Finally, high attrition rates at follow-up limit the generalizability of the results, and attrition was differential with higher losses among children with higher levels of ADHD and ODD behaviors.

Study 7

Evaluation Methodology

Design: Participant teachers were solicited via advertisement to complete an Incredible Years Teacher Training Course. Fifteen teachers from the Education Center in the Greater Dublin Area were recruited on a first come basis to complete the course from February to June 2007 (two consecutive afternoons a month for five months). The schools represented by these teachers were both suburban and rural.

Measures: Measures were collected at baseline, post-intervention, and at one-year follow-up (teacher measures only). Teachers completed the Incredible Years Teacher Strategies Questionnaire (TSQ) to measure the frequency of use and the usefulness of five teacher strategies: praise and effectiveness, proactive strategies, limit-setting, total positive approaches, and inappropriate strategies. Frequency of use of strategies that involved working with parents was also measured. Teachers also completed the Teacher Sense of Efficacy Scale to measure change in teacher perceived self-efficacy on four measures: student engagement, instructional strategies, classroom management, and total efficacy. Child behaviors were measured using the Strengths and Difficulties Questionnaire (SDQ) on six measures: emotional symptoms, conduct problems, hyperactivity, peer problems, total difficulties, and prosocial behavior. Impact of Difficulties (on self, peers and class) was also measured. The SDQ was completed per teacher on a number of children upon whom they chose to focus their teacher management skills. A total of 51 SDQs were completed (between 1 and 4 per teacher).

Qualitative measures were also completed by teachers to assess the quality of the course, using the Incredible Years Evaluation form.

Analysis: One-tailed tests were conducted on all measures. All quantitative measures were analyzed using the Wilcoxon Signed Ranks Test. Frequencies were used to analyze the qualitative measures.

Outcomes

Posttest: There were significant improvements on four of six child measures (SDQ): hyperactivity, peer problems, prosocial behaviors, and total difficulties. Effect sizes on these measures were medium to large (.46, .47, .52, and .62, respectively). Improvements on the remaining two measures (emotional difficulties and conduct problems) approached significance. There was also a significant improvement over time on the Impact of Difficulties measure in the proportion of children perceived as having no or minor difficulties as opposed to definite or severe difficulties. On teacher measures, three of four measures of teacher efficacy showed significant improvement, both at posttest and one-year follow-up: student engagement, classroom management, total teacher efficacy. There were no significant outcomes on any of the Teacher Strategies Questionnaire frequency measures, and only one of five perceived usefulness measures was significant: limit-setting.

Qualitative outcomes indicate that teachers were satisfied with the course and the program and felt that the strategies used were useful.

Brief Bulleted Outcomes

  • Significant program effects on four of six child outcome measures: reductions in hyperactivity, peer problems, and total difficulties, and increases in prosocial behaviors.
  • Moderate to large effect sizes on significant outcomes: hyperactivity = .46; peer problems = .47; prosocial behaviors = .52; total difficulties = .62.

Limitations: Nonsignificant results on the Teacher Strategies Questionnaire may have been due to the high number of non-responses to various items, which may have weakened the statistical power of results. Another limitation in this study is the lack of control group and the small sample size, which limits the generalizability of the findings. The SDQs were analyzed as if they had been derived from an unrelated group. There may have been teacher factors influencing the group score. The small n precluded analysis of such factor effects.

Study 8

Evaluation Methodology

Design: Participants were recruited from families requesting treatment for their child's conduct problems at the University of Washington Parenting Clinic. One-third of the families were self-referred, the others were referred by professionals (teachers, physicians) in the community. Families entered the study in three 50-55 family cohorts in the fall of 1995, 1996 and 1997. Families of 159 4- to 8-year-old children with oppositional defiant disorder were randomly assigned to one of six conditions: parent training (PT; n=31), child training (CT; n=30), parent training plus teacher training (PT+TT; n=24), child training plus teacher training (CT+TT; n=23), parent/child/teacher training (PT+CT+TT; n=25), and a waitlist control group (n=26). The waitlist control group received no treatment and had no contact with the research team or the team's therapists during the 8-9 month waitlist period. After the post-intervention assessment, control group families were offered the parent training program. Baseline assessments were conducted in early Fall, post-intervention assessments were conducted approximately six months later in the Spring, for all six conditions in each of the three cohorts. Follow-up assessments were conducted one year later in the Spring for five of the six conditions, because the control group was treated after post-intervention assessments.

Of the entire sample that completed baseline assessments, four families dropped out before beginning treatment and refused to participate in post-assessments. The rest of the sample completed post-assessments regardless of how many sessions attended. There was no significant difference in drop-out rate by treatment condition. No teachers dropped out of the TT condition.

Interventions: In the Child Training intervention, children were offered weekly two-hour sessions for 18 to 19 weeks for about six months with two therapists and six to seven children (Dinosaur School). The Parent Training intervention involved 10 to 12 parents meeting with 2 therapists for 2-hour sessions over the course of 22 to 24 weeks. Teacher Training involved 4 full days (32 hours) of group training sequenced throughout the school year.

Sample Characteristics: The children in the study were 90% boys, 79% white, with an average age of 5 years and 11 months. 74% of the parents were married.

Measures: Composite measures were calculated by adding multiple instruments in five areas:

Positive and Negative Parenting: Two parent-report measures (Parenting Practices Interview and Dyadic Parent-Child Interactive Coding System-Revised) and two observational measures (Coder Impressions Inventory and Parent Daily Discipline Inventory) were used to assess parenting styles and skills. The Parenting Practices Interview (PPI) has internal consistency alpha coefficients of .71 for harsh discipline and .66 for supportive parenting. The Dyadic Parent-Child Interactive Coding System-Revised (DPICS-R), which looks at positive parenting and critical statements, has a Cronbach's alpha of .78. The Coder Impressions Inventory-Parent (CII-P) is completed following a half-hour parent-child observation where parents are measured on 12 harsh-critical items (Cronbach's alpha = .89; interclass consistency coefficient = .54), 13 nurturing-supportive items (Cronbach's alpha = .88; interclass consistency coefficient = .67) and a rating on a 5-point scale of the observers perception of the degree to which a family needs help (interclass consistency coefficient = .64).

Child Conduct Problems at Home: Composite scores were calculated using one parent-report variable (the Eyberg Child Behavior Inventory - ECBI; Cronbach's alpha = .92) and four in-home observations including the DPICS-R coding system and the Coder Impressions Inventory for Children (CII-C).

Child Conduct Problems at School: The composite scores included two teacher report variables (scales from Teacher Assessment of School Behavior - TASB - Cronbach's alpha ranges between .62 to .91 ; and pertinent scales from the Teacher Rating of Perceived Competence Scale for Young Children - PCSC - reliability ranges between .70 to .90 for the subscales) and two summary scores from independent observations of classrooms.

Child Social Competence with Peers: The composite score includes two teacher-report variables (relevant scales from TASB and PCSC), one classroom observation (30-minute observations calculating conduct problems; Cronbach's alpha = .71) and one laboratory observation of the child with a peer (observation focused on Inappropriate Play scale from DPIS; internal consistency = .88).

Negative Classroom Management: A composite score of negative classroom management and atmosphere was computed via direct observation for each teacher using five variables: total teacher criticism; observation of classroom atmosphere (CAM, Cronbach's alpha ranging from .94 to .95) measured on a 10-item questionnaire; and three items from the Coder Impression Inventory - Teacher which rates harsh techniques, nurturing techniques and percentage of time teacher is inappropriate.

Analysis: The procedures were consistent with an intent-to-treat analysis. The four families who dropped out before treatment began were followed up, but they refused to complete post-assessments. Treatment effects for each measure were analyzed using six-group analysis of covariance with pretest scores as covariates for corresponding posttest scores. Planned comparisons contrasting each treatment condition with the control condition were conducted and then the addition of TT to CT and to PT was tested against CT alone and PT alone. Lastly, PT+CT+TT condition was tested against two factor conditions (CT+TT and PT+TT). Missing data were handled at two levels: an individual summary score was only computed if at least 60% of the items that made up the scale were present. Composite scores were also only computed if at least 60% of the summary scores in the composite were present. Cases were excluded from analysis on that composite if the composite score was missing at one of the time points. This resulted in sample sizes that fluctuated differently for different composite scores and treatment groups. The CT only condition had the most fluctuation at follow up for the father's positive parenting (n dropped from 30 to 22 at follow up).

Outcomes

Implementation Fidelity: Therapists conducting the parent or child group co-led their first group with a supervisor, followed a treatment manual for each session and documented fidelity to the program through a weekly protocol checklist of standards to be covered in each session. Group sessions were videotaped for feedback and analysis at weekly supervision meetings. Therapists received ongoing supervision, feedback and training throughout the study. Supervisors also randomly selected videotapes for integrity checks and the study reports that the analysis of checklists indicates high treatment integrity. Teacher training sessions were manualized, videotaped and the tapes were reviewed to ensure that training procedures did not vary across cohorts of teachers.

Attendance at the children and parent groups ranged between 90% and 100% for at least 15 sessions among the CT, CT+TT, PT, PT+TT and CT+PT+TT groups. All teachers attended the four days of training and all attended at least two meetings to work on individualized behavior plans for the child.

Baseline Equivalence: The study used ANOVA and chi-square analyses to compare baseline equivalence across all six conditions. No significant differences were found at baseline among all groups on demographic or family background variables. Further, there were no significant differences between conditions at baseline on any composite scores.

Differential Attrition: From the entire sample that completed baseline assessments, only four families dropped out prior to the beginning of the study and refused to participate in post-assessments. No information was provided about how these four families differed at baseline from those that continued in the study. At the two-year follow up, 9% of the families in the treatment groups (n=12 families) dropped out. Significantly more families in the CT condition dropped out. The children in these families who dropped out at two-year follow-up had fewer behavior problems at post-assessment.

Post-test (Webster-Stratton, Reid, and Hammond, 2004):
 Following the six-month intervention, all treatment conditions resulted in significantly fewer conduct problems, both at home and school, for children compared to controls. However, when compared with each other, there was little significant difference between treatment groups.

The following results at posttest specifically related to the teacher program in combination with either the child program or the parent program were found:

  • reduction in conduct problems at home (with both mothers and fathers) and at school;
  • teachers less negative;
  • less negative parenting by mothers.

One-year post intervention (Maintenance effects, no control group):
Because the control group was treated after post-assessments, these follow up results were only focused on whether children and parents in the treatment conditions improved or maintained effects in the following eight areas: mother negative parenting; father negative parenting; mother positive parenting; father positive parenting; child conduct at home per father; child conduct at home per mother; child conduct at school; and, child social competence. Mixed design (Time x Condition) ANOVAs were computed for each composite score from post-assessment to follow up. Seven of the eight areas were not significant at one-year follow up. Only child conduct at school was significant at one-year follow up (F=3.45; p<.01). However, seven of the eight measures were maintained to the 1-year follow-up. School behavior of children in the PT+CT+TT deteriorated from post-assessment to follow-up.

Two-years post intervention (Reid et al., 2003):
At two-year follow-up, parents were asked if their children received additional services or treatment after completing the program, and what changes in medications their children received. Results indicated no significant differences for any of these variables among treatment conditions. On ratings of conduct problems at home, comparison among treatment groups showed that children in the PT+TT group had significantly better outcomes that in the PT alone condition (chi-sq=5.27; p<.02). On ratings of conduct at school, the number of children showing significant improvement according to teacher reports maintained or improved for all five treatment groups. However, there were no significant differences between treatment groups.

Limitations: The limitations of this study can be summarized as follows:

  • Few additive effects when compared with treatments delivered in single or dual risk domains.
  • Follow-up assessments at one-year and two-year do not include the control group, which makes it difficult to determine sustained effects of the program.
  • Sample is fairly homogeneous (primarily white, two-parent, middle-income families), and who were motivated to bring their children to a clinic for treatment, which limits the generalizability of the study.
  • Sample size in each treatment condition is small, thus limiting the statistical power.
  • Attrition is low, but may differ by level of child's behavior problems.

Study 9

Evaluation Methodology

Design: Using a randomized control design, 14 Head Start centers (36 classrooms) were randomly assigned to two groups:

  • An experimental condition in which parents, teachers, and family service workers participated in the prevention programs (BASIC + ADVANCE + SCHOOL + TEACHER) (n=191, 23 classrooms from 9 centers)
  • A CONTROL condition in which parents, teachers, and family service workers participated in the regular center-based Head Start program (n=81, 13 classrooms from 5 centers).

Centers were chosen based on their willingness to participate in the study and to be randomly assigned to either intervention or control. Parents were recruited into the experimental condition by Head Start family service workers and teachers during summer and fall orientation. Originally, 328 families indicated interest in participating in the study (225 experimental and 103 control). Thirty-four families (15%) from the experimental centers and 22 families (21%) from the control centers did not complete spring posts assessments. This left 272 families (191 experimental and 81 control).

The experimental condition included four components: the standard BASIC parent training program, an abbreviated version of the ADVANCE and academic skills training (SCHOOL) programs, and a teacher training program (TEACHER). The first three components were held for parents of children attending Head Start and consisted of 16 weeks of two-hour group training sessions led by Head Start family service workers. The twelve-week standard BASIC program was offered to parents in the Head Start year, and a four-week abbreviated version of ADVANCE and SCHOOL was offered in the kindergarten year. Whereas the ADVANCE component targeted parent problem-solving skills and strengthening their relationships with partners, the SCHOOL program promoted parents' increased communication with teachers; techniques for encouraging children's reading, academic and problem-solving skills; arranging child "play dates;" and coaching positive peer play skills. The final component, the TEACHER training program, focused on improving teachers' classroom management and discipline skills, building relationships with students and parents, and helping instructors promote students' social and emotional competence. All Head Start teachers and aides in the intervention group received six monthly workshops sequenced over the first year of the intervention. Those in the control centers received the usual Head Start services, which included parent education in stress management, nutrition, self-care, and dental care.

Sample Characteristics: This study included a sample of 272 Head Start mothers, 272 four-year old children and 61 teachers. As in Study 5, many of the families in this study faced multiple risk factors for child conduct problems, including low education, low income, parent substance abuse, depression, and spouse or child abuse. The sample of children was 54% male with an average age of 55 months. 63% of the families represented ethnic minority groups, including predominantly African American, Hispanic, and Asian American ethnicities, and 52% of the families were single-parent households.

Measures: Assessments were conducted at baseline, post intervention, one year, and two years post intervention (through grade one). They included parent reports of children's behavior at home (using the CBCL and ECBI); teacher reports of children's conduct problems at school (using the Social Competence and Behavior Evaluation, SCBE, to measure social competence, emotional expression and adjustment difficulties; and the teacher ADHD checklist to assess the presence of attention deficit hyperactivity disorder); and blinded observations at home and at school (assessing children's noncompliance and deviance at home, and conduct problems and the amount of time engaged in activities at school). Construct scores combining observational and report data were calculated for negative (harsh discipline, critical parenting) and positive (monitoring, praise, consistent discipline, positive affect) parenting styles; parent-teacher bonding (primarily assessing increased communication between parents and teachers and parent involvement in school activities); child conduct problems at home (noncompliant, aggressive, and disruptive behaviors) and at school (including children's aggression, noncompliance, ADHD symptoms, engagement and social competence); and observer reports of teacher classroom management style (including teachers' positive techniques and harsh discipline, and classroom atmosphere).

Analysis: Analyses of covariance on the post-treatment construct scores using the corresponding pretest construct score as a covariate, was conducted. When these ANCOVAs showed significant group effect for the construct, then adjusted posttreatment means were examined to determine if the experimental group improved significantly more than the control group.

OUTCOMES:

Fidelity Monitoring: Monthly supervision was provided for parent group leaders coupled with close monitoring, direct observations and detailed training sessions and training manuals. 100% of the group leaders reported discussing all videotaped vignettes assigned and assigning all homework sessions described. For teachers, workshop checklists, standardized handouts, and videotaping of all teacher training was used to assure the integrity of the intervention.

Baseline Equivalence and Attrition: Analysis compared baseline equivalence of the experimental and control groups of the sample who completed both pre and post assessments (n=272). Analyses revealed several differences between groups at baseline, with the experimental group reporting significantly more risk factors than control group members. Specifically, the intervention group contained fewer boys, more minority members, higher rates of mother depression and stressful life events, and lower annual incomes. There were also significant differences in two of the six construct domains. Intervention mothers had higher negative parenting scores and lower positive parenting and bonding scores. Because of the differences on the two construct scores, analyses of covariance on the post-treatment construct scores using the corresponding pretest construct score as a covariate, was used.

Attrition analyses done at the one-year follow-up indicated no significant differences in dropouts between groups in terms of demographic and baseline risk factors. Differences between drops and nondrops on risk, demographic, and construct scores were examined separately for intervention and control groups. There were no differences on any variable in the intervention group, and only one difference between drops and nondrops in the control group. More families from the control group who reported children with behavior problems dropped out between pre- and post-assessments.

Posttest: At post assessment, intervention mothers had significantly lower scores on the negative parenting construct and higher scores on the positive parenting construct compared to control mothers. Parent-teacher bonding was marginally significant in the full sample and significantly higher in a subsample of intervention mothers who attended six or more intervention sessions than for control mothers. In terms of children's behaviors, intervention children showed significant improvements on the conduct problems school construct (including lower reports of hyperactivity and antisocial behaviors and more social competence), marginally significant results on the home construct, and children of mothers who attended six or more sessions showed significant improvement on the conduct problems at home construct, compared with control children. In addition, children who were in the "highest risk" category (based on observations of high rates of non-compliant and aggressive behavior) at baseline showed greater clinically significant reductions in aggressive and noncompliant behaviors than high-risk children in the control group. Teachers' behavior also improved, with instructors in the intervention group having significantly higher scores on the classroom management construct than control teachers.

Long-term: When baseline scores were compared to the one-year follow-up measures, some intervention effects remained. Only four of the original six construct scores were examined (negative and positive parenting, parent-teacher bonding, and child behavior problems at home). Observer reports of teachers' classroom management and children's behavior at school were not available. There were marginally significant effects in the predicted direction for the Positive Parenting and the Child Conduct Problems at Home constructs. Parents who attended more than nine sessions continued to have lower scores on the negative parenting construct and higher scores on the positive parenting construct. Clinically significant findings on children's conduct problems found at post assessment continued to be present one year later. This difference was significant with the whole sample, as well as with the children of parents classified as "attenders". Parent-teacher bonding was significant at the one-year follow-up, but in the reverse direction, with intervention mothers showing less bonding with teachers than control mothers. Analyses of the two-year follow-up results are currently being conducted.

Brief Bulleted Outcomes:

  • Significant program effects on reducing negative and increasing positive parenting, compared to parents in the control condition.
  • Significant program effects on parent-teacher bonding for mothers who attended more than six intervention sessions, compared to mothers in the control condition.
  • Significant and marginally significant program effects on child school conduct problems and home behaviors, respectively, compared to control group children.
  • Greater program effects for children who attended more intervention sessions or where categorized as "higher risk" for conduct problems at home and aggressive and noncompliant behaviors, respectively.
  • Significant program effects on teacher classroom management skills, compared to control group teachers.
  • Some sustainment of program effects at one-year follow-up: parenting (both negative and positive) and child conduct problems.

Limitations:

  • The study was only able to recruit 50% of the eligible families enrolled in Head Start which limits the generalizability of the study.
  • The experimental and control groups differed on several demographic and outcome variables at baseline.
  • The study suffered from differential attrition.
  • The study randomized classrooms but analyzed individuals.

Study 10

Evaluation Methodology

Design:

Recruitment: Head Start centers that are part of a larger Head Start Cares program volunteered to participate in the program. To select a diverse sample, the study included centers in 10 states and 4 regions of the country. The study separately examined 3- and 4-year old children in 4-year old and mixed 3 and 4-year old classrooms. Among 4-year olds, 79% of parents consented at baseline to have their children participate, and 90% consented by the posttest.

Assignment: After being blocked on race/ethnic composition and part-day/full-day schedules, centers were randomly assigned (N = 104) to one of four conditions: Incredible Years Teacher Training, a control, and two other interventions. Analysis was then conducted with the separate interventions and, for 3-year olds, with a treatment group that combined all three interventions. The control group included 220 3-year-old children and the Incredible Years treatment group included 246 3-year-old children. For 4-year-old children, the control group consisted of 621 children and the Incredible Years treatment group consisted of 702 children.

Attrition: For the study of 3-year olds, the child pretest occurred in the fall and the posttest in the spring. The study reported approximately 12% attrition in the Incredible Years treatment group and 19% attrition in the control group among 3-year-olds. However, Table B6 (page 69) lists 211 control children and 214 Incredible Years children at baseline, while the analysis results in Table 8 (page 41) list the full sample of 220 control children and 246 Incredible Years children.

For the study of 4-year olds, the child pretest in the fall and posttest in the spring were supplemented by a 1-year follow-up in the spring of kindergarten. The design allowed for children who did not have consent at baseline to be added for the posttest and follow-up when parents gave consent late. Figure 3.1 thus shows 2,128 children who consented, were selected, and were in the center at baseline, and it shows 868 who were not consented or selected at baseline but were added later. The total equals 2,996. At posttest, the number remaining in the centers was 2,670 (89%). The study thus reported 11% attrition (p. 249). The study tracked 2,599 (87%) into kindergarten.

Note that the pretest occurred from September to December, possibly months after the program began.

Sample: Children in the 3-year-old sample were an average of 3.42 years old and roughly half female (50.44%). Children in the 4-year-old sample were an average of 4.38 years old and roughly half female (48.08%). No information is provided on race/ethnicity of the 3-year-old child sample. The sample of Head Start centers suggests socioeconomically disadvantaged participants.

Among 4-year-olds, the sample was 43% Hispanic, 33% Black, and 16% White. The sample shows evidence of being disadvantaged; 59% of participating children's families were eligible for the Supplemental Nutrition Assistance Program (SNAP) and the average income was $1,800 per month.

Measures: All measures of child outcomes reported for 3-year olds came from teacher reports. The study used 1) the Behavior Problems Index (BPI), which measures the frequency, range, and type of children's behavior problems, and had scales for total problems, externalizing, hyperactivity, and internalizing; 2) the Cooper-Farran Behavior Rating Scales for work related skills and interpersonal skills, 3) Social Skills Rating System, which measures children's' ability to regulate their own behavior, assert themselves to solve conflicts with peers, and ability to cooperate, 4) Student-Teacher Relationship Scale, which measures the teacher's perception of the quality of their relationship with individual children. Finally, the study examined pre-academic skills, using the Academic Rating Scale of early language and literacy, mathematical thinking, and general knowledge skills.

The study of 4-year olds supplemented preschool teacher reports with assessments done by "independent observers" and by kindergarten teachers who presumably did not know of group assignment. For child behavior outcomes, the study used Head-to-Toes, pencil tap, and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure executive function and used the Behavior Problems Index (BPI) to measure behavior regulation, and learning behaviors. The study used the facial emotions identification task, emotions situation task, challenging situations task, the social skills rating system (SSRS), and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure interpersonal skills. For pre-academic skills, the study used the Woodcock-Johnson III Letter-Word Identification test of pre-literacy and literacy skills and applied problems (math), the expressive one-word picture vocabulary test (EOWPVT) of vocabulary, and the Academic Rating Scale (ARS) for early language and literacy, math, and general knowledge skills.

To measure teacher outcomes among both 3-year-old and 4-year-old samples, the study used the Adapted Teaching Style Rating Scale (Adapted TSRS) to measure classroom management, social-emotional instruction, and scaffolding, in addition to the Classroom Assessment Scoring System (CLASS), which measured emotional support, classroom organization, instructional support, and literacy focus.

Analysis: The two studies analyzed two-level models for children within classroom and classrooms within centers. The models included fixed-effects dummy variables for the blocks in which centers were randomized, random effects for centers, and controls for baseline outcomes.

Intent-to-Treat: All participants with available data were included at each assessment point, but the study did not follow children who left the centers.

Outcomes

Implementation Fidelity: The study did not discuss implementation fidelity for 3-year-olds. For 4-year-olds, the study stated that implementation fidelity for Incredible Years exceeded "satisfactory." Mattera et al. (2013) reported that the intervention provided teachers with materials, training, coaching, monitoring and technical assistance. Coaches spent 90-minutes in each classroom observing and then meeting with teachers an average of 3 times per month. Monthly fidelity logs were completed by coaches where teachers received a fidelity rating on a 1 (low) to 5 (high) scale. Teachers on average received a rating of 3.69 from coaches on the fidelity scale. The study also reported that fidelity improved over time.

Baseline Equivalence: For the 3-year olds, tests for child age, gender, and outcome measures at baseline in Table B6 found no significant differences between the Incredible Years treatment group and the control group. However, the table includes only a subset of the cases. Tests for teacher and classroom characteristics revealed only two significant differences in race/ethnicity of teachers.

For the 4-year olds, only one significant child difference (p < .05) emerged from about 30 tests (Table B4). The Ns for the tests (541 for Incredible Years and 512) for the control group are again smaller than the full sample of randomized subjects. Tests for teacher and classroom differences (Tables B2 and B3) showed a few marginal differences and significantly lower scores in emotion coaching (p<.05).

For both 3- and 4-year olds, the occurrence of pretests after the program start may have affected the baseline equivalence tests.

Differential Attrition: The study of 3-year old children found lower rates of attrition among the Incredible Years treatment group than in the control group (12% versus 19%, p < .10) and reported (page 95) that higher-risk children were more likely to dropout. Specifically, children who left the centers had significantly higher levels of total behavior problems and externalizing and internalizing behavior problems than did those who stayed. An additional check examined whether differential attrition produced condition differences at baseline among those who remained at posttest. To quote page 97 for 3-year-olds, "Among children who did not leave the sample, there were no statistically significant differences between the Incredible Years group and the control group." The results suggest that attrition did not produce an unbalanced sample.

For 4-year olds, tests showed that children who left the centers were more likely to be male, be rated as having lower levels of social behavior, learning behavior, and pre-academic skills, and perform poorly on executive function and cognitive tests than those who stayed (Table Q.1.). Among Incredible Years subjects, attrition was 11% (significantly different at p < .10 than the 15% for control group). Table B5 tests for baseline differences among the analytic sample and therefore reflects the influence of attrition. It shows one significant difference (of 30 plus tests) on baseline social problem solving.

Posttest: For 3-year old children, the study found significant posttest differences for the Incredible Years classrooms as compared to the control group in 5 of the 12 teacher-reported social-emotional outcomes (social skills rating scale, assertion, cooperation, self-control, and closeness of the student-teacher relationship) and all 3 teacher-reported pre-academic skill outcomes (general knowledge, language, mathematical thinking).

Among 4-year-olds, the study reported no significant intervention effects for directly assessed pre-academic skills, but significant intervention effects for all 3 teacher-reported pre-academic skills: general knowledge, language and literacy, and mathematical thinking. The study also found a significant intervention effect for the teacher-reported work-related skills measure (p<.05), facial emotions and social problem-solving (p<.05), and teacher-reported social behaviors (p<.05).

For teacher and classroom measures, the study did not find any statistically significant posttest differences for 3-year-olds (p < .05) on 17 measures of teacher practice or 14 measures of classroom climate between the Incredible Years treatment group and the control group. Of 14 teacher-reported measures of teacher behavior for 4-year-olds, teachers in the Incredible Years treatment group reported significantly better scores for 3 of 6 measures of classroom management and 1 of 6 measures of social-emotional instruction. Of 11 teacher-reported measures of classroom climate among four-year-olds, teachers in the Incredibly Years treatment group reported improved on 1 of 4 measures of emotional support and 1 of 3 measures of classroom organization.

Mediation Analysis: The study did not conduct formal mediator analysis.

Long-Term: The study did not include a long-term follow-up for 3-year-olds and no significant effects were found at the one-year follow-up for 4-year-olds on measures of behavior and academics. However parents did report significantly greater receipt of special education services, which the authors interpret in positive terms as identifying child problems early.

Limitations:

  • Among the 3-year-old sample, all measures are teacher-reported
  • Some evidence of differential attrition, but possibly similar across conditions
  • Weak evidence of long-term effects in tests for 4-year-olds

Study 11

Evaluation Methodology

Design:

Recruitment: All children in 120 Seattle area Head Start programs and 14 elementary schools were invited to participate in the program over 4 years. Schools were selected based on a larger proportion of students receiving free or reduced price lunch. In total, 86% of the parents of children in the selected Head Start programs and 77% of the parents of students in the elementary schools (kindergarten and first grade) consented to participate.

Assignment: The study matched pairs of Head Start centers and schools based on size, geographic location, and demographics. The matched pairs were then randomly assigned to either the intervention or a waitlist control group. The sample size seems to vary between measures and the study is unclear about how many teachers were assigned to the treatment or intervention groups. For classroom and teacher measures, there were 120 classrooms with 153 teachers. For the observer measure of student conduct, there were 1,786 students nested in 160 classrooms and 119 teachers.

Attrition: A posttest assessment at the end of the school year followed a pretest assessment in the fall. For the teacher and classroom measures, the study reported that missing data reduced the number of classrooms from 120 to 115 (96%) and the number of teachers from 153 to 139 (91%). For the student measures, missing data reduced the number of students from 1,786 to 1,746 (98%).

Sample: Students in the study were, on average, 63.7 months of age and were 50% male. The sample was 18% Hispanic, 18% black, 20% Asian, 27% white, 8% African, and 9% other. Thirty one percent of the sample did not speak English as a first language.

Measures: The study measured social and emotional competencies, conduct problems, teacher competencies, teacher efforts to involve parents, and classroom environment through teacher reports and independent observations of teachers and students. The study also administered the Wally Problem-Solving test to a high-risk subsample. The measures showed good reliability.

Observers were blind to condition. To assess teachers, observers utilized the Multiple Option Observation System for Experimental Studies to examine positive reinforcement, critical statements, and amount of interaction with students. In addition, they used the Teacher Coder Impressions Inventory to evaluate teachers' styles and classroom management. The Teacher Coder Impressions Inventory rated teachers on harsh/critical, inconsistent/permissive, warm/affectionate, social/emotional, and effective discipline.

To assess child behavior, coders used the Multiple Option Observation System for Experimental Studies to assess total conduct problems, time disengaged or off-task, time in solitary place, child positive with teacher, child positive with peers, and time in peer involvement. They also used the School Readiness and Conduct Problems: Coder Observation of Adaptation-Revised to evaluate child school readiness based on emotional self-regulation, social skills, and conduct problems.

To assess classroom atmosphere, observers used the Classroom Atmosphere Measure to examine overall level of students' cooperation and problem solving, interest in subject matter, focus, responsiveness, on-task behavior, and classroom support.

The study piloted the Wally Problem Solving and Feelings Tests during the latter half of the study among students with more significant conduct problems, as reported by teachers, parents, or school counselors.

Teachers rated parent involvement using the INVOLVE-T questionnaire, which evaluates the extent to which parents participate in school activities, seem comfortable with the teacher and school environment, value education, support the teacher, and assist children with their homework.

Analysis: The study used pre-post ANCOVA multilevel models that allowed for both classroom- and teacher-level variation in intercepts. For the teacher outcome analyses, a three-level model nested repeated measures within teachers and teachers within classrooms. For the student outcome analyses, the four levels included repeated measures, students, classrooms, and teachers. The random effects could differ in each model, depending on tests for significance and convergence of the estimates.

Intent-to-Treat: The sources of missing data and the procedure used to deal with the missing data are unclear. Nearly all students were included (98%) in the student analyses and most teachers were included (90%) in the teacher analyses.

Outcomes

Implementation Fidelity: The study found that on average, 27 of the 30 required lessons were completed in the intervention classrooms and an average of 30 recommended vignettes and 25 small group activities were completed in the intervention classrooms.

Baseline Equivalence: The study reported no significant differences between conditions for schools, students, or teachers on demographic variables, but it did not discuss outcome variables at baseline.

Differential Attrition: The study did not discuss differential attrition, though missing data for students was low.

Posttest: For the observer rating of teacher styles, 4 of 5 constructs had significant main effects. As compared to control teachers, intervention teachers became less harsh/critical and inconsistent/permissive, more warm/affection, and placed more emphasis on social/emotional teaching. There was no significant main effect for effective discipline, however there was significant improvement among Head Start teachers only.

Observer ratings of teacher classroom management showed a significant effect for 1 of 3 measures. Teachers in the intervention condition became significantly less critical after the intervention as compared to control teachers. Neither teacher involvement nor teacher praise showed a significant effect.

Observer ratings of school readiness found a greater improvement in school readiness for intervention students as compared to control students. None of the 6 child constructs for conduct problems and disengagement had significant effects. However, 2 of the outcomes were significant for students with high baseline conduct problems.

Observers found a greater improvement in the classroom atmosphere in the intervention as compared to the control groups.

A subsample of children with conduct problems reported improvement in positive problem solving strategies and positive feelings they could identify. However, these findings are preliminary given a small sample size.

The teacher report of parent involvement found a significant effect for parent-teacher bonding, but no effect on three other measures of parent involvement.

Mediation Analysis: The study did not conduct formal mediator analysis.

Long-Term: The study did not include a long-term follow-up.

Study 11, continued
Thompson, et al. (2017)

Evaluation Methodology

Design:

Recruitment: Using the same data set as Webster-Stratton, Rei and Stoolmiller (2008) in which matched pairs of schools/Head Start centers were randomly assigned to condition, this study conducted an analysis on a sub-set of kindergarten teachers and their students entering kindergarten. Given the information provided in the text, it is not clear whether the school/center-level randomization is lost by examining just this subset of teachers.

Assignment: This study included 42 kindergarten teachers and 805 students entering kindergarten, with 19 teachers and 504 students in the treatment group and 23 teachers and 301 students in the control group.

Attrition: According to the Webster-Stratton, Rei & Stoolmiller (2008) study, teacher retention was 96%, but this study did not report attrition specifically for the subset of kindergarten teachers and their students.

Sample: A majority of teacher participants were female (97%) and white (75%), an additional 22% were African American. A majority of students were male (51%) and were 5.6 years old (on average). Nearly all students qualified for subsidized lunch, 39% of students' families identified as white, 20% as Asian, 13% as African American, and 17% as Hispanic and other.

Measures: The study created latent profile models using the Teacher-Parental Involvement Questionnaire (INVOLVE-T). In this questionnaire, teachers rated their perception of the extent to which parents participate in school activities, seem comfortable with the teacher and school environment, value education, support the teacher, and assist children with their homework. Three subscales were analyzed:

  • Bonding, which gauged teacher perceptions of whether a teacher felt parents appeared comfortable and had a good relationship with a parent (sample α = .76).
  • Parental involvement in education, which examined teacher perceptions of whether parents were involved in school activities or if they were supportive of educational goals (sample α = .91).
  • Parental involvement with teacher, which assessed teacher perceptions of whether parents called, attended conferences or visited the school (sample α = .84).

Analysis: Latent class profile models for each of the three outcomes (bonding, parental involvement in education, and parental involvement with teacher) were estimated using FIML, and included a cluster to adjust for intraclass correlations.

Intent-to-Treat: The study used a Full Information Maximum Likelihood (FIML) procedure to account for missing data and the analysis examined outcomes for all cases with available data.

Outcomes

Implementation Fidelity: No significant departures from protocol or principles were noted. On average, teachers received 3.73 days of training and only four teachers attended less than four days of training. Checklists completed by researchers indicated that, on average, teachers and students were exposed to 87% of all planned intervention elements.

Baseline Equivalence: The treatment and control classrooms did not differ at baseline on two out of three indicators used to model the profiles. At baseline, teachers reported similar ratings of general parental involvement and bonding with parents, but teachers in the control group reported having better relationships with parents compared to treatment teachers.

Differential Attrition: The study reported that there was no relationship between missingness and observed student or teacher characteristics or treatment assignment. However, the study reported attrition for teachers overall, not for the kindergarten subsample, and did not provide the details of the attrition analysis.

Posttest: The study reported that there was a significant, but small, association between assignment and transition pattern (Cramer's V =.143). Teachers in the treatment group were more likely to transition to a more adaptive view of parental involvement (as defined by improved perceptions of bonding and involvement of parents at home and school) at follow-up compared to teachers in the control condition. Though the study refers to several tables and figures reporting results, these tables were not included in the report. Findings were therefore based on what was included in the authors' text.

Mediation Analysis: The study did not conduct a formal mediator analysis.

Long-Term: The study did not include a long-term follow-up.

Study 12

Evaluation Methodology

Design:

Recruitment: This study recruited nine urban schools in a Midwestern part of the United States that included a majority African American students. All the schools were implementing a Positive Behavior Supports and Interventions program with high fidelity, which may not be typical of other schools. Of all 109 eligible teachers, 96% enrolled and were randomized, and 84% of 2,168 eligible students consented to participate. Teachers were recruited and randomized across three sequential cohorts.

Assignment: The enrolled teacher participants were randomly assigned to either the intervention group or the control condition within each of the schools according to a blocked cluster randomized design. There were 53 teachers and 900 students allocated to the IY program group and 52 students and 917 students assigned to the control group. The three sets of two-day workshops were delivered in October, December, and February for the intervention group, while the control group classrooms continued their usual curriculum.

Attrition: Data were collected at three time points: beginning of the school year, immediately prior to intervention (October), and post-intervention (April and May). The CONSORT diagram showed 98% of the teachers and 92% of the students (1680 of 1817 randomized and consented students) completed the follow-up and were used in the analysis. Among those who completed the follow-up, the maximum rate of missing data in the final samples used for analyses was 1.8%.

Sample: The teacher participants were mostly female (97%) and white (75%), with 11.1 years of teaching experience on average. The student sample was 52% male, mostly African American (76%), and 61% qualified for free or reduced lunch.

Measures: Teachers (who helped deliver the program) filled out the 54-item Teacher Observation of Classroom Adaptation-Checklist to measure four child behaviors: concentration problems, disruptive behavior, prosocial behavior, and emotional dysregulation. Cronbach's alpha for these measures ranged from 0.77 to 0.96.

Teachers also filled out the Revised Social Competence Scale-Teacher version. Cronbach's alphas ranged from 0.93 to 0.96 for the social competence subscale and from 0.92 to 0.93 for the academic subscale.

Standardized tests of student academic performance in reading and math were obtained using the Woodcock-Johnson III Normative Update Tests of Achievement, and the test-retest reliability for this measure ranged from 0.80 to 0.95.

Analysis: Three-level hierarchical linear models were used to analyze the main effects of treatment on student behavior and academic performance. With students nested within teachers and teachers nested within schools, the models adjusted for random assignment of classrooms and associated clustering. Student-level covariates, including the pretest outcomes, were controlled at the first level.

Intent-to-Treat: The study lost 8% of the students who did not complete the posttest but otherwise used all subjects with data and imputed any missing items.

Outcomes

Implementation Fidelity: Almost all teachers attending all six workshops (attendance rates ranged from 94 -100%), and if teachers had to miss a workshop due to illness or other reasons they met with the coach to review the missed material.

Baseline Equivalence: The authors stated that the conditions were equivalent at baseline but did not report effect sizes or significance tests.

Differential Attrition: There were no tests for differential attrition of those lost to follow-up. A brief mention that the largest differential missing rates between intervention and control groups were 0.7 at posttest appears to refer to missing data among those completing the posttest.

Posttest: There were significant main effects of the program on teacher-rated prosocial behavior (d = 0.13, p = 0.03), emotional dysregulation (d = -0.14, p = 0.00), and overall social competence (d = 0.13, p = 0.02). The program failed to significantly improve the measures of academic outcomes, although the teacher-rated measure was marginally significant (p = .07). It had no effects on the two independent measures of reading and math achievement.

There was also a moderating effect of child pretest levels of social competence on posttest social competence reports (b = -0.09, p = 0.04), and a moderating effect of pretest academic competence scores on increases in academic competence over the course of the program, compared to controls (b = -0.08, p = 0.00).

Long-Term: Not examined.

Study 13

This study analyzed changes in teacher-reported perceptions of parental involvement in their children's education, but does not analyze a Blueprints behavioral outcome.

Evaluation Methodology

Design:

Recruitment: Teachers were recruited across three sequential, annual cohorts from nine urban schools serving primarily African American students. No more information was provided on how recruitment occurred. Teacher and parent participants provided written consent and students provided written assent.

Assignment: An experimental design was used in which teachers were randomly assigned to either the intervention or a wait-list control condition. Initially, 109 teachers and 2,168 students were deemed eligible for the study, but 4% of eligible teachers (n = 4) and 16% of eligible students (n = 350) declined to participate prior to randomization. A total of 105 teachers and 1,818 of their students were therefore randomized to condition. The intervention group included 53 teachers and 901 students, while the control group included 52 teachers and 917 students. Control group teachers continued their business-as-usual teaching and professional development opportunities.

Attrition: Baseline data for several measures not included in this write up (as they are not relevant to the experimental study) were collected in October and follow up data were collected in April of the same academic year. The study reported no teachers were lost to follow-up, while 63 students in the intervention condition (7%) and 57 students in the control condition (6%) moved out of the study schools. At follow-up, the sample therefore included 105 teachers and 1,698 students.

Sample: All students were in grades kindergarten through third grade. The majority of teachers were female (97%). Most (75%) were white, and an additional 22% were African American. Slightly over half the student sample was male (52%), and 76% of students were African American while 22% were white (with 2% identified as "other"). Half the student sample qualified for free or reduced price lunch.

Measures: The study measured teacher perceptions of parental involvement in their children's education using the Parent Involvement Measure-Teacher, which was completed for each student. Two subscales, Parent-Teacher Contact and Parent comfort, were included in the analysis examining differences in outcomes between treatment and control teachers. For the control group sample, test-retest reliability was .57 for the Parent-Teacher Contact subscale and .68 for the Parent Comfort subtest. Reliability was not reported for the treatment group or for the overall combined sample. Several other measures, including standardized achievement tests, classroom observations, and teacher ratings of parent and student behaviors, were included in the report but were not in the analysis examining differences between treatment and control groups. These measures were therefore excluded from this write up.

Analysis: The study used latent transition analyses, accounting for students clustered within classrooms, to examine changes in teacher perceived parental involvement from baseline to follow-up. Covariates included student socio-demographic characteristics (i.e., sex, race, and lunch status).

Intent-to-Treat: The study only excluded students missing all data. FIML procedures were used to account for missing data.

Outcomes

Implementation Fidelity: Attendance at workshops ranged from 94-100% for each of the six workshops, and coaches followed up with absent teachers to review the missed material. Teachers reported high satisfaction with the training workshops, and on average teachers received 6 hours of additional coaching between and after workshop sessions. Observations conducted in late April and May of the same academic year revealed that treatment teachers used higher rates of proactive teacher strategies compared to control teachers, though it is not clear who conducted these observations.

Baseline Equivalence: The study reported that baseline equivalence was demonstrated for outcome measures but no tests of baseline equivalence were reported for socio-demographic variables.

Differential Attrition: The study did not report analysis of differential attrition.

Posttest: Program outcomes included teacher perceptions of parent involvement patterns and the progression from a less adaptive to a more adaptive pattern. The study found that teachers in the control condition reported 7% of parents in the lower two adaptive classes at baseline transitioned to a more adaptive class at follow-up, whereas teachers in the treatment condition reported 39% of parents did the same. Of parents rated to have transitioned to the highest adaptive class at follow-up, 80% were in the treatment group. A chi-square test of transition types (those transitioning to higher classes, those staying the same, and those transitioning to lower classes) found that the differences between conditions were significant in favor of the treatment group. No effect sizes were reported.

Mediation Analysis: The study did not conduct a formal mediator analysis.

Long-Term: The study did not include a long-term follow-up.

Study 14

Evaluation Methodology

Design:

Recruitment: Schools that had previously implemented the program's parent training intervention were eligible to apply for participation, resulting in 21 intervention schools. Another 22 schools were recruited from rural and urban municipalities to act as waitlist control schools. No participating schools had implemented any other evidence-based school behavior intervention program for at least a year prior to baseline. Potentially participating children were identified by randomly selecting 7 children from each classroom (grades 1, 2, 3) for screening (n=1343). Only children who scored in the clinical range for externalizing behaviors were included in the study (n=83).

Assignment: Schools were non-randomly selected to serve in intervention (n=45 students) or waitlist control conditions (n=38 students).

Attrition: Total student attrition at posttest was 21%, due to school dropout, teacher dropout, and family/student dropout. The number of schools lost was not described.

Sample:

The student sample was split between grades 1 (33.7%), 2 (22.9%), and 3 (43.4%) and was predominantly male (78.3%). No other student demographic characteristics were described, but all children met diagnostic criteria for externalizing behaviors.

Measures:

Participants were assessed at baseline and posttest (approximately 9 months after baseline). All measures were rated by teachers, who also delivered the intervention.

Externalizing problems were measured with the Sutter-Eyberg Student Behavior Inventory-Revised, which consists of 7-point Likert items that evaluate frequency and severity of common behavior problems. These problems include temper tantrums, outing, acting defiant, difficulty staying on task, trouble paying attention, and failing to complete tasks or projects. Total scores are summed and range from 38 to 266, though a score above 144 is considered in the clinical range (α=.97).

Internalizing problems and academic performance were assessed using the Teacher Report Form of the Achenbach System of Empirically Based Assessment family of instruments. The internalizing instrument contains subscales of Anxiety/Depressed (α=.79), Withdrawn (α=.72), and Somatic Complaints (α=.53). Academic performance was assessed by informally comparing individual student scores to class averages across six different academic subjects.

Social skills were assessed using the Social Skills Rating System, which measures the occurrence and importance of specific social skills, academic competence, and behavioral problems. The instrument covers a wide range of behaviors, including making friends easily, temper control, getting along with others, and following directions. Internal consistency was high, α=.94.

Student-teacher relationship was assessed using the Student-Teacher Relationship Scale, which is comprised of two subscales, closeness and conflict. The closeness subscale (α=.81) measures the degree of affection and open communication between student and teacher, while the conflict subscale (α=.84) measures the extent to which that relationship is characterized by antagonistic, disharmonious interactions.

Analysis:

The effects of the intervention were evaluated using multilevel mixed-effects linear models, with measurements nested within individuals nested within schools.

Intent-to-Treat: Although most attrition occurred in the intervention schools, missing data were treated as missing at random within the individual, and all available data were used in the analysis.

Outcomes

Implementation Fidelity:

Fidelity was not assessed. The authors note that teachers "received either verbal or written feedback/guidance from the group leaders on classroom based practice of new skills," but do not mention the frequency of classroom observations nor any rating system.

Baseline Equivalence:

There were no significant baseline differences between groups on any demographic characteristics or outcome measures.

Differential Attrition:

There was a total attrition rate of 21%, with most of this (93%) occurring in the intervention group. No formal tests were used, but the authors imply that the difference in rates was significant.

Posttest:

At posttest, compared to the control group, participants in the intervention group showed significant improvement in teacher-reported student-teacher conflicts and academic performance. There were no significant group differences on any other outcomes.

Study 15

Evaluation Methodology

Design:

Recruitment:

The Incredible Years Norway program invited Norwegian municipalities where the Incredible Years Parenting Training program was already implemented to participate in the trial. A total of 3,161 students and 92 teachers were recruited for the study, 46 teachers (n=1,527 students) in the treatment group and 46 teachers (n=1,634 students) in the control group. Teachers in the intervention group received the program free of charge and a small economic compensation for the time spent in completing questionnaires. The comparison group consisted of teachers recruited from both rural and urban municipalities in which none of the Incredible Years programs were available. Interested kindergarten teachers applied to Incredible Years Norway to participate in the comparison group. They received economic compensation for time spent on administering assessments and were offered training in the Incredible Years Teacher Classroom Management program one year later. To be included in the study, none of the kindergarten staff could have attended an evidence-based kindergarten behavioral intervention or program in the course of the previous year. Parents who did not speak or understand Norwegian were excluded (though not explicitly stated, it is assumed this means the children of parents who did not speak or understand Norwegian were excluded), and if the kindergarten received the program, all children in the classroom indirectly received the intervention (whether or not they participated in the study).

Assignment:

Authors conducted a quasi-experimental study with no matching. A total of 86 teachers were assigned to a condition and completed a pretest assessment, though it was not clear why this number dropped from the 92 teachers originally recruited for the study. In each department (which is assumed to be a classroom, but was not clearly stated), the authors randomized seven children to participate in the trial. That is, 7 children from each classroom in the study were randomly selected by a statistician "blind to characteristics of the classroom" (p. 2218) to be assessed. For example, in a group of 21 children, a random number list from 1-21 was generated electronically, and the kindergarten teachers matched the 7 numbers with the children's alphabetical order. The authors claimed this was done to "reduce data dependency" (p. 2218). This resulted in 1,218 students assigned to condition for the purpose of the study, 581 children in treatment and 637 in control.

Attrition:

At the pretest, there were 1,049 students within 86 kindergarten classes that completed an assessment (for a student-level attrition rate of 14% and a teacher-level attrition rate of 7%). At the posttest, there were 90 teachers (though it was not clear how this number increased from 86 at the baseline assessment) and 908 students who completed an assessment, for a student-level attrition rate of 25%. Attrition rates at the teacher level could not be calculated given the numbers presented in the consort diagram (see Figure 1, page 2217).

Sample:

Demographic information at the student level was only reported by condition. Students' average age for treatment and condition was 4.3 years and 4.4 years (respectively). In addition, 50% of the treatment group and 48% of the control group were girls, while 12% of the treatment group and 9% of the control group were deemed high-risk.

Measures:

Participants were assessed at baseline and posttest after program implementation. All measures were rated by teachers, who also delivered the intervention. Measures included:

  • Sutter-Eyberg Student Behavior Inventory-Revised: 38 items that measure the frequency (alpha = .94) and intensity (alpha = .97) of various types of problem behaviors.
  • Child Teacher Report Form Internalization Scale: Four subscales which rate children behaviors on 99 items. Subscales include: (1) emotional reactive, (2) anxious depressed, (3) somatic complaints and (4) withdrawn. Though reliability and validity for this measure was not specifically reported, the authors stated that: "Test-retest reliability and validity on the C-TRF have been found to be high" (p. 2219).

Risk and protective factors were assessed using one measure:

  • Social Competence and Behavior Evaluation for Teachers: A 40-item tool assessing social competence that summarizes the positive characteristics of a child. Test-retest reliabilities range from .74 to .87.

Analysis:

Linear mixed models were utilized to assess differences between groups in outcomes from pretest to posttest. In the analysis of the entire group, the data were hierarchically organized with observations (level 1) nested within children (level 2), and children were nested within teachers (level 3). Though the authors state the data were nested, they did not specifically report whether the analysis controlled for clustering. In addition, moderation analyses were conducted on a sub-group of children deemed "high-risk," as measured by those who scored equal to or above 138 on a baseline measure of the Sutter-Eyeberg Student Behavior Inventory Total Intensity score (n = 106).

Intent-to-Treat:

ITT was not explicitly discussed, nor was how missing data were handled, but it appeared that ITT protocols were followed.

Outcomes

Implementation Fidelity:

Not reported.

Baseline Equivalence:

Baseline equivalence was met for demographic and baseline outcome measures for all students who completed the pretest (n=1,049) with one exception - children in the treatment group were slightly younger than children in the control group. Baseline equivalence was not tested on the assigned sample.

Differential Attrition:

Children who attrited differed significantly from those with complete data on the behavior intensity measure at baseline. In terms of attrition by condition, none of the baseline child measures differed significantly between the intervention and comparison groups in the analytic sample, however tests involving demographic characteristics were not conducted.

Posttest:

At posttest, as compared to control, students in the treatment group showed greater improvement in intensity, problem and internalizing behaviors. In addition, compared to control, students in the treatment group showed greater social competence (a risk and protective factor).

Moderation analysis found that high-risk students in the treatment group showed improved social competence as compared to high-risk students in the control group.

Long-Term:

Not conducted.