Sign Up For Newsletter

Blueprints For Healthy Youth Development logo

Wyman’s Teen Outreach Program®

A nine month program that engages high school students in a minimum of 20 hours of community service learning annually with weekly meetings. The goal is to reduce rates of teen pregnancy, course failure, and academic suspension.

Program Outcomes

  • Academic Performance
  • Sexual Risk Behaviors
  • Teen Pregnancy

Program Type

  • Civic Responsibility/Education
  • Leadership and Youth Development
  • School - Individual Strategies
  • Skills Training
  • Social Emotional Learning

Program Setting

  • School
  • Community
  • Residential Facility

Continuum of Intervention

  • Universal Prevention

Age

  • Late Adolescence (15-18) - High School
  • Early Adolescence (12-14) - Middle School

Gender

  • Both

Race/Ethnicity

  • All

Endorsements

Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
SAMHSA : 2.2-2.3

Program Information Contact

Christina Donald
Wyman Center
600 Kiwanis Drive
St. Louis, Missouri 63025
Tel: 314-717-2071
Website 1: www.wymancenter.org
Website 2: www.teenoutreachprogram.com

Program Developer/Owner

Original Developer: Brenda Hostetler
Current Owner/Purveyor: Wyman Center


Brief Description of the Program

Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. As a result, teens are better able to navigate challenges during the teenage years - a time when decisions matter.

TOP is designed to meet the developmental needs of middle (6th-8th grades) and high school (9th-12th grades) teens in a variety of settings, including in school, after-school, through community organizations or in systems and institutional settings. Curriculum topics include: emotion management, problem-solving, decision-making, goal-setting, health and wellness, healthy decision making, self-understanding, social identity, empathy, communication, relationships and community.

Outcomes

Two studies assessed the effect of the Teen Outreach Program® on the three different problem behaviors (Allen et al, 1997; Allen and Philliber, 2001):

  • The risk of pregnancy among Teen Outreach participants was 41% and 53% as large as the comparison group. The effect was larger among those who were already parents than among non-parents.
  • The risk of course failure was 42% and 60% as large as the comparison group, respectively. The program was only successful for females. Also, minorities and those with prior academic suspensions benefitted more from the program than did whites and those without suspensions.
  • The risk of academic suspension was 39% and 52% as large as the comparison group, respectively.

Studies with No Effects

  • Walsh-Buhi et al., (2016) found that, at follow-up, compared to the control group, participants in the intervention group showed no significant improvements in risky sexual behaviors overall.
  • Robinson et al. (2016) reported no significant effects on initiation of sexual intercourse or use of birth control.
  • Bull et al. (2016) found no effects overall but in comparison to the control condition, the intervention condition showed significantly fewer pregnancies for Hispanic participants only.

Brief Evaluation Methodology

The efficacy of the Teen Outreach Program has been typically studied using quasi-experimental designs. In these designs, students at each studied site were divided into Teen Outreach program groups and comparison groups. Youth in each group filled out confidential questionnaires within the first few weeks of the program/academic year and the last few weeks of the program/academic year. Data collected included sociodemographic characteristics, whether they had ever caused a pregnancy or been pregnant, whether they had failed any courses in the prior school year, and whether they had been suspended academically in the prior school year. The post-test changed the reference time period to the program/academic year itself. To determine the efficacy of the Teen Outreach program, the post-test data were compared across the two groups, controlling for past behaviors and individual characteristics.

Walsh-Buhi et al., (2016) tested the program using a cluster randomized trial of 28 public schools in 12 nonmetropolitan Florida counties. The schools were paired on structural and demographic characteristics and randomized into the intervention or control groups. A total of 7,667 students across two cohorts participated in the program. Primary outcomes regarding sexual activity included ever having had sex, recent sex, recent risky sex, and intention to have risky sex. Participants were assessed at baseline in the fall of the school year and at follow-up the next spring, approximately 9 months after baseline, with a 62% retention rate.

Robinson et al. (2016) evaluated the program in a community setting. The researchers randomly assigned 4769 youth from Louisiana and 966 youth from Rochester to the intervention and control conditions. The youth were recruited through convenience sampling methods. Measures of sexual initiation and use of birth control were collected through self-reports at baseline and immediately after the intervention period. In Louisiana the control condition received no programming, and in Rochester the control condition received a work-readiness intervention.

Bull et al. (2016) examined subjects from eight Boys & Girls clubs in Denver over 4 years. The control condition consisted of TOP programming alone, and the intervention condition consisted of TOP programming plus the text message supplement. Participants completed self-report measures at baseline and immediately at program completion.

Study 1

Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach program. Journal of Community Psychology, 29(6), 637-655.


Study 2

Allen, J., Philliber, S., Herrling, S., & Kuperminc, G. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729-742.


Risk Factors

School: Low school commitment and attachment

Protective Factors

Individual: Academic self-efficacy, Problem solving skills, Prosocial behavior*, Prosocial involvement*, Skills for social interaction

School: Opportunities for prosocial involvement in education*

Neighborhood/Community: Opportunities for prosocial involvement*, Rewards for prosocial involvement


* Risk/Protective Factor was significantly impacted by the program

See also: Wyman's Teen Outreach Program® Logic Model (PDF)

Race/Ethnicity/Gender Details
Results provided by gender (male/female) and race (black/white/Hispanic/other). Bull et al. (2016) found a program effect only for Hispanic participants.

Teen Outreach Program® Facilitator Training is designed for those who will directly deliver TOP® to teens. The primary goal of this training is to prepare facilitators to begin their TOP® club(s) and to know where they can access additional information and support. Those supervising the implementation of TOP® may also benefit from attending so they are clear on the messages delivered in training.

TOP® Facilitator Training Content includes:

  • An orientation to TOP® including program history, core goals and principles, key outcomes, the TOP® fidelity model and an overview of how and why TOP® gets results
  • An introduction to the TOP® Changing Scenes Curriculum© and the modeling of two lessons
  • A review of Community Service Learning principles and practice building a service learning project with a group
  • Coaching and practice on facilitating sensitive subject matter
  • Values neutral facilitation
  • A review of the experiential learning cycle and multiple intelligence theory with practice integrating both into lessons and service learning
  • An overview of operational, sequencing and routine considerations for planning a TOP® club
  • Teams of two to three training participants will prepare and practice facilitating a TOP® lesson
  • Consistent modeling, by trainers, of techniques for group engagement, processing learning, and integrating the three core components of TOP®

The TOP® Facilitator Training does not include:

  • Adolescent development
  • Group or classroom management
  • Positive discipline strategies

It is highly recommended that providers select facilitators with experience and proven skills working with teens. Wyman operates under the assumption that providers will hold their own staff orientations and trainings specific to the population they serve and the settings in which their programs operate.

Training Certification Process

The TOP® Training of Trainers is designed for those who will deliver the TOP® Facilitator Training in their network. The primary goal of this 5-day training is to prepare trainers to be able to deliver the content and approach of the TOP® Facilitator Training and to share where additional information and support may be found. Those supervising TOP® trainers may also benefit from attending so they are clear on the messages delivered in the training.

TOP® Training of Trainers content includes:

  • A complete modeling of the TOP® Facilitator Training (first 2.5 days). This is intended to demonstrate a clear set of expectations, training lessons, and messages for the training they will deliver. See the TOP® Facilitator Training overview for a list of content covered in this section.
  • A review of the schedule and content of the TOP® Facilitator Training with self-evaluation of those areas where participants feel confident and where additional training or support may be needed.
  • An exercise to examine leadership styles of participants and the impact of those styles on their approach to being a trainer.
  • A review of the logic models for TOP® and for the TOP® Facilitator Training.
  • An overview of the roles and responsibilities for each level of the National Network.
  • An introduction to TOPnetOnline - the website designed to support networking with Wyman, partners, providers and facilitators, to provide access to helpful information and resources, and to collect data from teens and facilitators that demonstrate fidelity and informs technical assistance and support.
  • Group discussion on training techniques, adult learning, training do's and don'ts, and group management strategies for training.
  • Teams of two to three training participants will prepare and practice leading a TOP® Facilitator Training Lesson.
  • A discussion of next steps including the training test, which is emailed to participants the week following training.
  • New trainers are provided with all training lesson plans, background information for self-study, access to TOPnet Online materials, and two coaching calls - one before their first Facilitator Training and one afterward.

TOP® Training of Trainers content does not include the following and these can be addressed with individual partner technical assistance:

  • Specific partner contract and MOU discussions
  • Research and evaluation strategies for partners
  • Group planning time for teams from partner organizations

It is highly recommended that partners select trainers with experience and proven skills in facilitation and training adults. Wyman operates under the assumption that partners will hold their own staff orientations and trainings specific to planning and organizing their network of TOP® trainers, providers and clubs.

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Start-Up Costs

Initial Training and Technical Assistance

The purveyor requires each program to become a Certified Replication Partner. This allows the program to train group facilitators at the local level. The cost to become a Replication Partner is $26,000 and includes training for one trainer, labor for the initial certification site visit, 10 sets of curricula, the first year license, eight hours of technical assistance in Year 1, and access and set-up on TOPnet Online. The certified trainer may then train multiple local group facilitators.

Curriculum and Materials

Additional curricula are available only to Certified Replication Partners and their provider organizations. Individual sets are $500 plus shipping.

Materials Available in Other Language: TOP Teen Pre and Post Survey is available in Spanish, but program materials have not been translated.

Licensing

Included in the initial start-up fee for a Certified Replication Partner.

Other Start-Up Costs

None.

Intervention Implementation Costs

Ongoing Curriculum and Materials

Estimated at $2,000 per group/club for replacement materials and art supplies.

Staffing

Facilitators lead groups on a part-time basis. Teachers are sometimes used as facilitators but special qualifications are not required as long as facilitators are trained in the program. Facilitators lead groups of 25 or fewer participants. A co-leader is recommended, bringing the ideal leader to teen ratio to 1:12. Facilitators lead an average of 3-4 groups at a time, spending a total of 3-4 hours a week in planning, implementation, communication and logistics. Groups run concurrently with the academic year.

Other Implementation Costs

It is recommended that $2,000 per group/club should be budgeted for costs of transportation of participants to and from community service learning experiences.

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

No additional training from the purveyor is required. Replacement trainers can be trained and supported for $6,000 each. Eight hours of Technical Assistance is included in the Fidelity Monitoring cost described below. Additional T.A. is available at $75 per hour.

Fidelity Monitoring and Evaluation

The cost for monitoring, including site visits every two years, T.A., and access to the online data system is $6,000 per year.

Ongoing License Fees

Included in the Fidelity Monitoring cost described above.

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

Space is usually donated by schools or other interested organizations. If space is not available, there may be a cost for accessing suitable space for group meetings.

Year One Cost Example

This example describes a sponsoring organization supporting 10 facilitators each with 3-4 groups of 25 youth participants offering groups for 30 weeks during the academic year for the first year of operation. Each group would have a co-facilitator. The groups would reach 875 teen participants. Costs would include:

Replication Partner Certification $26,000.00
Supplies @ $2000 per group $70,000.00
Staff-35 groups x 4 hours x $25 x 2 leaders $168,000.00
Transportation @ $2000 per group $70,000.00
Local Trainer plus fringe $90,000.00
Administrative Overhead @ 10% of Staff Cost $26,000.00
Total One Year Cost $450,000.00

With 875 youth participants, the per youth cost is $514.

The above is a good example of a community-based or after-school model. In-school implementation typically reduces the staffing costs by over half, since a classroom teacher is often the 2nd facilitator. Transportation costs are also much lower.

Funding Overview

Wyman's Teen Outreach Program is a youth development program with evidence that it prevents teen pregnancy, school drop-out, course failure, and suspension. Funding streams that support youth development and after school programs, middle and high school education, and pregnancy prevention are most commonly used to support the program. TOP also has a strong emphasis on community service so federal service-learning funding streams as well as private foundations interested in community service are potential sources of support for the program.

Funding Strategies

Improving the Use of Existing Public Funds

Funding available for youth development, and pregnancy and sexually-transmitted disease prevention may be considered to fund Wyman's Teen Outreach Program. To the extent that existing pregnancy prevention and youth development programs are not evidence-based, a jurisdiction can consider redirecting funds toward Wyman's Teen Outreach Program to get better outcomes. This could potentially be achieved by training and supporting staff in existing after school and pregnancy prevention programs to help them integrate use of the TOP model in their program. TOP can also be implemented during the school day, with the redirection of teacher or school counselor time to facilitate the program.

Allocating State or Local General Funds

State grant funds supporting youth development and pregnancy prevention programs may be considered for TOP. Some state health departments fund TOP through teen pregnancy and violence prevention grants.

Maximizing Federal Funds

Formula Funds:

  • The TOP program has received funding through child welfare agencies in some jurisdictions with Title IV-B funding.
  • The 21st Century Community Learning Centers block grant is the largest federal funding source dedicated to after school and youth development programs. State education agencies administer the funds to local school districts and community agencies. TOP has received 21 CCLC funds from state education departments for implementation of the program in after school settings.
  • Title I can potentially support curricula purchase, training and teacher salaries in schools that are operating school wide Title I programs. In order for Title I to be allocated, school administrators would have to understand that TOP can contribute to overall academic achievement.
  • The Personal Responsibility Education Program (PREP), administered by the Administration on Children, Youth, and Families (ACYF), Office of Adolescent Health (OAH) provides $55 million annually by formula to states and territories for evidence-based programs that educate adolescents on both abstinence and contraception to prevent pregnancy and sexually transmitted infections.
  • Temporary Assistance to Needy Families (TANF) is a formula grant that states use to provide cash assistance and work supports to needy families. One of the four stated purposes of TANF funding is to prevent and reduce out-of-wedlock pregnancies and many states have used TANF to support a wide array of youth development programs that can help to prevent pregnancy.
  • The Social Services Block Grant Program (SSBG) provides states very flexible dollars to fund a variety of social service programs. State social service agencies may allocate some portion of these funds toward pregnancy programs.
  • The Community Development Block Grant (CDBG) program is administered from the federal Department of Housing and Urban Development to localities to support community economic development. Fifteen percent of these funds can be used to support a wide range of public services. Cities may choose to direct some portion of these funds to pregnancy prevention programs and youth development programs.

Discretionary Grants: Relevant discretionary grants include grants focused on pregnancy prevention that are administered by the Department of Health and Human Services Office of Adolescent Health (OAH) and Family and Youth Services Bureau (FYSB), and the Centers for Disease Control grants for replication of evidence-based programs for teen pregnancy prevention. In addition, Learn and Service grants administered by the federal Corporation for National and Community Service can potentially support TOP, given its focus on service learning. Finally, the strong education and drop-out prevention outcomes align with discretionary grants focused on drop-out prevention and academic achievement in high need communities that are administered by the Department of Education.

Foundation Grants and Public-Private Partnerships

Foundations, particularly those with a focus on pregnancy prevention, youth development, can be a good source of funding for TOP. Foundations with a particular interest in evidence-based interventions should be explored.

Debt Financing

No information is available

Generating New Revenue

Several fund raising approaches might be useful in supporting TOP. These could include fundraising by local civic organizations, and local businesses and industries. Schools or community organizations with existing youth development programs could seek donors or hold fundraising events to support the relatively low cost of training and curriculum, using that training to improve the quality of existing programming.

Data Sources

All information comes from the responses to a questionnaire submitted by the purveyor, The Wyman Center, to the Annie E. Casey Foundation.

Program Developer/Owner

Original Developer: Brenda HostetlerCurrent Owner/Purveyor: Wyman Center

Program Outcomes

  • Academic Performance
  • Sexual Risk Behaviors
  • Teen Pregnancy

Program Specifics

Program Type

  • Civic Responsibility/Education
  • Leadership and Youth Development
  • School - Individual Strategies
  • Skills Training
  • Social Emotional Learning

Program Setting

  • School
  • Community
  • Residential Facility

Continuum of Intervention

  • Universal Prevention

Program Goals

A nine month program that engages high school students in a minimum of 20 hours of community service learning annually with weekly meetings. The goal is to reduce rates of teen pregnancy, course failure, and academic suspension.

Population Demographics

TOP is open to all 9th-12th graders who attend schools that have adopted the program. The developers have implemented and evaluated this program with middle school youth as well; however, these evaluations failed to meet Blueprints quality standards.

Target Population

Age

  • Late Adolescence (15-18) - High School
  • Early Adolescence (12-14) - Middle School

Gender

  • Both

Race/Ethnicity

  • All

Race/Ethnicity/Gender Details

Results provided by gender (male/female) and race (black/white/Hispanic/other). Bull et al. (2016) found a program effect only for Hispanic participants.

Other Risk and Protective Factors

Teen Outreach is not targeted to youth with specific risk factors. Rather, the program views adolescence itself as a risk factor. Protective factors include enhanced feelings of autonomy and connectedness with adults, community, and school through community service.

Risk/Protective Factor Domain

  • Individual
  • School
  • Neighborhood/Community

Risk/Protective Factors

Risk Factors

School: Low school commitment and attachment

Protective Factors

Individual: Academic self-efficacy, Problem solving skills, Prosocial behavior*, Prosocial involvement*, Skills for social interaction

School: Opportunities for prosocial involvement in education*

Neighborhood/Community: Opportunities for prosocial involvement*, Rewards for prosocial involvement


*Risk/Protective Factor was significantly impacted by the program

See also: Wyman's Teen Outreach Program® Logic Model (PDF)

Brief Description of the Program

Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. As a result, teens are better able to navigate challenges during the teenage years - a time when decisions matter.

TOP is designed to meet the developmental needs of middle (6th-8th grades) and high school (9th-12th grades) teens in a variety of settings, including in school, after-school, through community organizations or in systems and institutional settings. Curriculum topics include: emotion management, problem-solving, decision-making, goal-setting, health and wellness, healthy decision making, self-understanding, social identity, empathy, communication, relationships and community.

Description of the Program

Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. TOP aims to achieve four goals:

  • Improve social, emotional, and life skills
  • Support development of a positive sense of self
  • Strengthen connections to others
  • Improve academic outcomes and decrease risky behavior

These goals are accomplished through implementation of the following core program components:

  • High quality facilitation by caring, responsive and knowledgeable TOP staff who build strong, supportive relationships with teens, and create engaging and empowering "TOP Club" experiences
  • At least 25 weekly peer group meetings
  • At least 12 lessons of TOP curriculum provided by facilitators who tailor the content to the teens' needs
  • At least 20 hours of meaningful community service learning, which includes planning, action and reflection

The TOP curriculum includes lessons that address content on adolescent development: skill-building, connections with others, and learning about one's self. Within each area of content, the curriculum includes foundational, intermediate, and advanced lessons that are designed to meet the developmental needs of young people in grades 6 through 12. Each of the 120+ lessons is written for a 45-minute time period with an additional extension activity available if time allows. The TOP curriculum is designed to allow for sequenced programming over multiple years (e.g., as part of a whole-school middle school approach), with a minimum duration of 9 months.

The community service component has four phases: preparation, action, reflection, and celebration. Most of the time is spent in action, with reflection happening continuously throughout the community service experience. Service can involve direct action (e.g., tutoring, building a playground or garden), indirect action (e.g., bake sales, blood drives), and civic action for older students (e.g., voter registration, public speaking, educational theatre and awareness programs). There is a process through which students are able to choose the specific type of service they want to engage in.

Wyman's national replication model involves partnering with school districts or community organizations who implement TOP in schools and other settings. Wyman uses a Training of Trainers (TOT) model which is offered quarterly in St. Louis, Missouri. In some cases with large school districts, Wyman will offer the training onsite at schools. The TOT training is five days long and is facilitated by two Wyman TOP Trainers. The first 2.5 days cover the facilitator training, during which participants learn how to directly deliver TOP to teens. The next 2.5 days focus on teaching participants how to train others in delivering TOP and how to structure their TOP implementation to ensure the fidelity and quality of their TOP implementation within their home organizations.

Wyman's National Network team provides technical assistance to TOP replication partners through a variety of means, including conference calls and webinars. In addition, outcome and implementation data collection and reporting are supported through Wyman Connect, Wyman's online TOP data system. Wyman's National Network team works directly with replication partners to regularly monitor fidelity and quality of implementation, with re-certification occurring at the end of each program year.

Theoretical Rationale

Teen Outreach is a preventive program that aims to use a developmentally-based approach to achieve positive outcomes in preventing pregnancy and academic failure. A developmental perspective suggests that successful adolescent adjustment (i.e. fewer problem behaviors) requires a simultaneous sense of autonomy and relatedness among peers and adults. In Teen Outreach volunteer activities, adolescents find themselves in help-giving rather than help-receiving situations. This, in turn, enhances their sense of autonomy and self-efficacy while at the same time fosters bonds with the community and school. The qualities of autonomy and relatedness are viewed as essential tools in navigating adolescence and making sound decisions around school and sexuality.

Theoretical Orientation

  • Skill Oriented
  • Cognitive Behavioral
  • Self Efficacy
  • Attachment - Bonding

Brief Evaluation Methodology

The efficacy of the Teen Outreach Program has been typically studied using quasi-experimental designs. In these designs, students at each studied site were divided into Teen Outreach program groups and comparison groups. Youth in each group filled out confidential questionnaires within the first few weeks of the program/academic year and the last few weeks of the program/academic year. Data collected included sociodemographic characteristics, whether they had ever caused a pregnancy or been pregnant, whether they had failed any courses in the prior school year, and whether they had been suspended academically in the prior school year. The post-test changed the reference time period to the program/academic year itself. To determine the efficacy of the Teen Outreach program, the post-test data were compared across the two groups, controlling for past behaviors and individual characteristics.

Walsh-Buhi et al., (2016) tested the program using a cluster randomized trial of 28 public schools in 12 nonmetropolitan Florida counties. The schools were paired on structural and demographic characteristics and randomized into the intervention or control groups. A total of 7,667 students across two cohorts participated in the program. Primary outcomes regarding sexual activity included ever having had sex, recent sex, recent risky sex, and intention to have risky sex. Participants were assessed at baseline in the fall of the school year and at follow-up the next spring, approximately 9 months after baseline, with a 62% retention rate.

Robinson et al. (2016) evaluated the program in a community setting. The researchers randomly assigned 4769 youth from Louisiana and 966 youth from Rochester to the intervention and control conditions. The youth were recruited through convenience sampling methods. Measures of sexual initiation and use of birth control were collected through self-reports at baseline and immediately after the intervention period. In Louisiana the control condition received no programming, and in Rochester the control condition received a work-readiness intervention.

Bull et al. (2016) examined subjects from eight Boys & Girls clubs in Denver over 4 years. The control condition consisted of TOP programming alone, and the intervention condition consisted of TOP programming plus the text message supplement. Participants completed self-report measures at baseline and immediately at program completion.

Outcomes (Brief, over all studies)

Allen et al. (1997) found that the Teen Outreach Program had statistically weak to moderate effects on problem behavior outcomes. The risk of each of the problem behaviors was less than half for the Teen Outreach than for the comparison group. Specifically, risk of becoming pregnant was 41% as large as the comparison group, risk of course failure was 42% of the comparison group, and risk of suspension was 39% of the comparison group. The hypothesized psycho-social mechanism of an increased sense of autonomy and relatedness among peers and adults was found to be significant, but only among middle school youth (Allen, 1994). No theoretically sound mechanisms have been identified and successfully tested for high school youth.

Allen and Philliber (2001) found that pregnancy prevention effects were similar across all youth except that those who were already parents benefited more than non-parents. With respect to preventing course failure, Teen Outreach was only effective for females, was more effective for minorities than for whites, and for those who had prior academic suspensions than for those without prior suspensions. No significant differences in the effectiveness of Teen Outreach in preventing academic suspension were found.

Allen et al. (1990) sought to identify what Teen Outreach programmatic conditions were most effective in producing successful outcomes. No programmatic differences were found in effectiveness whether the program was offered during the normal class day or after-school or whether the program used a little or a lot of the Teen Outreach curriculum. However, the number of volunteer hours and number of classroom hours (especially for younger participants relative to older participants) had a significant and positive influence in reducing problem behaviors.

Walsh-Buhi et al. (2016) found that at posttest, intervention participants in Cohort 1 were significantly less likely than control group youth to report recent sex, risky sex, and sexual intentions, and Cohort 2 program youth had significantly reduced risky sexual intentions; however, there were no significant impacts of the program across both cohorts, overall.

Robinson et al. (2016) reported no significant effects.

Bull et al. (2016) found no significant effects overall, but for Hispanic participants only, the intervention condition reported significantly fewer pregnancies than those in the intervention condition.

Outcomes

Two studies assessed the effect of the Teen Outreach Program® on the three different problem behaviors (Allen et al, 1997; Allen and Philliber, 2001):

  • The risk of pregnancy among Teen Outreach participants was 41% and 53% as large as the comparison group. The effect was larger among those who were already parents than among non-parents.
  • The risk of course failure was 42% and 60% as large as the comparison group, respectively. The program was only successful for females. Also, minorities and those with prior academic suspensions benefitted more from the program than did whites and those without suspensions.
  • The risk of academic suspension was 39% and 52% as large as the comparison group, respectively.

Studies with No Effects

  • Walsh-Buhi et al., (2016) found that, at follow-up, compared to the control group, participants in the intervention group showed no significant improvements in risky sexual behaviors overall.
  • Robinson et al. (2016) reported no significant effects on initiation of sexual intercourse or use of birth control.
  • Bull et al. (2016) found no effects overall but in comparison to the control condition, the intervention condition showed significantly fewer pregnancies for Hispanic participants only.

Mediating Effects

Only one study investigated possible mediating effects. According to Allen et al. (1994), program effectiveness for middle schoolers was mediated by the degree to which youth viewed that the program enhanced their autonomy and relatedness among peers and adults.

Effect Size

The program consistently demonstrated effect sizes in the weak to moderate range. The effect size of Teen Outreach in the odds of causing pregnancy or becoming pregnant were .53; the effect size in the odds of failing a course were .60; and the effect size in the odds of failing a course were .52. The effect size in the odds of causing or becoming pregnant were .34 for parents and .82 for non-parents. The effect size in the odds of failing a course were .52 for females (compared to no effect for males); .54 for minority students (compared to .74 for majority students); and .57 for youth with prior academic suspensions (.34 for those without prior suspensions). The effect size in the odds of being academically suspended was .52. In Walsh-Buhi et al., (2016), odds ratios ranged from .40-.71, indicating some small to medium program effects on risky sexual behaviors for one cohort receiving the program, though there were no impacts overall. Robinson et al. (2016) reported no significant effects. Bull et al. (2016) did not report an effect size for the significant effect found in the study.

Generalizability

These studies are not generalizable to the nation or to non-studied Teen Outreach programs. This limitation is due to participant self-selection into the program (except for Allen et al., 1997 and Walsh-Buhi et al., 2016), lack of information on site selection and/or the impact of site selection, and because the demographic distributions of the students do not reflect distributions at other sites or across the nation.

Otherwise, the program reduced course failure more among girls than boys and among minorities than non-minorities.

Potential Limitations

  • In two studies--Allen & Philliber (2009) and Allen et al. (1994)--the authors do not indicate how the studied sites were selected or from what years the data were drawn. A third study (Allen et al., 1990) provides the year of the data but does not indicate the site selection method.
  • Only one study randomly assigned all youth to treatment or comparison groups (Allen et al., 1997), although even in this study, the youth that made up the sampling frame had already expressed an interest in the program.
  • In each study, the youth self-selected into the Teen Outreach program. Thus, the results can only be generalized to the type of youth who would be inclined to join this type of program.
  • The authors (Allen & Philliber, 2001) do not indicate which years the data were collected; how the 60 sites were selected; or whether data was collected on all Teen Outreach students in each site, as opposed to sampling Teen Outreach students within sites.
  • Program participants select the program freely or respond affirmatively to a recommendation. None of the program participants are actually assigned to the program, and certainly not randomly assigned. Many in the comparison groups were nominated by students in the intervention group.

Walsh-Buhi et al., (2016):

  • No overall impacts on behavioral outcomes
  • No tests for differential attrition
  • Some baseline differences
  • A few small iatrogenic effects in subgroups

Robinson et al. (2016):

  • No information on reliability or validity of sexual behavior measures
  • Possible violation of intent to treat
  • Significant differences between groups in ethnicity at baseline
  • Tests for differential attrition used baseline equivalence with the analysis sample but without comparison to randomized sample
  • No effect on behavioral outcomes

Bull et al. (2016):

  • No information on reliability or validity of sexual behavior measures
  • Significant differences between groups in pregnancy at baseline
  • Some effects of baseline measures on attrition
  • Effect on behavioral outcome for Hispanics only

Notes

Another study of the program neither examined program outcomes nor used a control groups. A 4-week pilot program (Devine et al., 2014) was conducted that added text messages to the main program. A total of 96 teens were enrolled in the study and received on average 11 messages per week related to the program's content with references to celebrities and musicians in whom the participants indicated they were interested. More than half of these texts asked for a response from the participant and an automated message was sent acknowledging the receipt of the response. Text reminders regarding upcoming sessions were also sent out. Teens without mobile phones were set up with a buddy system to read text messages with another participant. Satisfaction surveys were collected from the participants during and at the end of the study.

Devine, S., Bull, S., Dreisbach, S., & Shlay, J. (2014). Enhancing a teen pregnancy prevention program with text messaging: Engaging minority youth to develop TOP® plus text. Journal of Adolescent Health, 54(3), S78-S83.

Endorsements

Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
SAMHSA : 2.2-2.3

Peer Implementation Sites

DeVonne Bernard
Director, Teen Outreach Program
Wyman Center
1401 LaSalle Lane, Suite 220
St. Louis, MO 63104
(314) 471-3864
Devonne.Bernard@wymancenter.org

Alexandra Peralta
Program Performance Officer
Children's Services Council of Palm Beach County
2300 High Ridge Road
Boynton Beach, FL 33426
(561) 374-7612
Alexandra.Peralta@cscpbc.org

Program Information Contact

Christina Donald
Wyman Center
600 Kiwanis Drive
St. Louis, Missouri 63025
Tel: 314-717-2071
Website 1: www.wymancenter.org
Website 2: www.teenoutreachprogram.com

References

Study 1

Certified Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach program. Journal of Community Psychology, 29(6), 637-655.

Study 2

Certified Allen, J., Philliber, S., Herrling, S., & Kuperminc, G. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729-742.

Study 3

Allen, J., Philliber, S., & Hoggson, N. (1990). School-based prevention of teen-age pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach program. American Journal of Community Psychology, 29(4), 505-523.

Study 4

Allen, J., Kupermind, G., Philliber, S., & Herre, K. (1994). Programmatic prevention of adolescent problem behaviors: The role of autonomy, relatedness, and volunteer service in the Teen Outreach program. American Journal of Community Psychology, 22(5), 617-639.

Study 5

Walsh-Buhi, E. R., Marhefka, S. L., Wang, W., Debate, R., Perrin, K., Singleton, A., ... & Daley, E. M. (2016). The impact of the Teen Outreach Program on sexual intentions and behaviors. Journal of Adolescent Health, 59, 283-290.

Study 6

Robinson, W. T., Seibold-Simpson, S. M., Crean, H. F., & Spruille-White, B. (2016). Randomized trials of the Teen Outreach Program in Louisiana and Rochester, New York. American Journal of Public Health, 106(51), 539-544.

Study 7

Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text messaging, Teen Outreach Program, and sexual health behavior: A cluster randomized trial. American Journal of Public Health, 106(51), S117 - S124.

Study 1

Evaluation Methodology

The motivation for this evaluation was to learn whether the Teen Outreach program had greater efficacy for those students who were more at-risk for problem behaviors such as pregnancy and academic difficulties.

Design: In this mix of a quasi-experimental and randomized study, data were collected from 3,277 youth in grades 9-12 (1,673 program youth, 1,604 comparison youth) from 60 sites over the course of 4 years. The actual years of the data collection were not reported. No information is provided as to how the 60 sites were selected.

About 20% of the study youth were randomly assigned to either the Teen Outreach group or the comparison group. The remaining Teen Outreach youth chose to participate in the program by requesting it after hearing about it during school announcements or agreeing to participate after having been referred by teachers or counselors. The remaining comparison group youth were selected through one of two methods: (a) program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) program participants nominated youth whom they thought were from similar backgrounds and encountered similar life problems. The authors reported that analysis found no statistically significant differences between the randomized and non-randomized samples.

Youth in the Teen Outreach program and in the comparison group filled out self-report questionnaires within the first few weeks of the beginning of the program and academic year. Program youth did this in their Teen Outreach classroom, while the comparison group youth filled out the questionnaire in study hall or some other school setting. The youth reported demographic characteristics and whether they had ever been pregnant or caused a pregnancy, whether they had failed any courses in the prior school year, and whether they had been suspended academically in the prior school year. At the end of the program and academic year, the youth repeated the questionnaire but reported only on outcomes for the academic year of the program. The Teen Outreach sample experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%.

Sample characteristics: Of the 3,277 sampled youth, the mean age was 16 years old, with the majority in 9th or 10th grades. The average mother's education level was slightly over a high school graduate, and slightly over half (about 55%) of the youth lived in two-parent homes. About 19% of the sample had prior academic suspensions and 32% had failed at least one course. About 8% had either caused or been pregnant in the past and about 3% were teen parents. The samples differed only slightly on gender and race profiles.

Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June). Students were asked their age, their grade in school, their race/ethnicity (black, white, Hispanic, other), their mother's education level (1=not a high school graduate through 4=college graduate), whether they lived in a two-parent household, whether they had been suspended from school in the previous school year, whether they had failed courses in the previous school year, whether they had ever caused a pregnancy or been pregnant, and whether they were currently a parent.

Analysis: This study used three logistic regressions in an intent-to-treat analysis. Each regression's binary dependent variable represented a particular problem behavior outcome at program exit (i.e., 1 if student became or caused a pregnancy in program year, 0 otherwise; 1 if student failed a class in program year, 0 otherwise; and 1 if student was academically suspended in program year, 0 otherwise). The independent variable that indicated program participation reflected the effect of the program on the three different outcome variables. Control variables included demographics and the baseline outcomes of pregnancy, course failure, and academic suspension status at the beginning of the year. In addition to general program effectiveness, interactions between program participation and other factors (demographic background, behavioral risks, familial risks) were tested to determine whether program effectiveness differed for students with different characteristics.

Outcomes

Implementation fidelity: Although program facilitators were trained, the content of the program (i.e., which components of the curriculum were used and to what extent) and the quality of the implementation at each site were not measured.

Baseline equivalence: Only two relatively small but significant differences in sociodemographics and background risk factors existed between the program participants and the comparison groups. Females represented 29.1% of the comparison group and only 24.6% of the program participants, and 46.1% percent of the comparison group identified as black, compared with 44.3% percent of the program participants.

Differential attrition: The Teen Outreach sample experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%. Attriters were more likely than non-attriters to become pregnant or cause a pregnancy and/or fail courses in school. They were also more likely to be younger, male, members of a minority group, and living in single-parent households. However, attriters were equally likely to come from the Teen Outreach sample as from the comparison sample.

Teen pregnancy: Self-reports showed that Teen Outreach participants had 53% (p<.01) of the risk of pregnancy exhibited by the comparison group. The only characteristic that resulted in significant differences by individual or behavioral characteristics was for teen parent status. Parents in Teen Outreach were 82% less likely to become or cause pregnancy than the comparison group, while non-parents in Teen Outreach were 34% less likely to become or cause pregnancy than non-parents in the comparison group. Note that Teen Outreach was no more or less effective for youth who had experienced prior pregnancy but were not parents.

Course failure: Self-reports indicate that Teen Outreach participants had 60% (p<.001) of the risk of course failure exhibited by the comparison group. Significant differences in the effect of the program were found with respect to gender, race/ethnicity, and prior academic suspension status. The program was significantly effective for females (Teen Outreach females had 52% of the risk of course failure compared to females in the comparison group), but not for males. Teen Outreach was significantly effective for all races, but especially effective for minorities. Minority students in Teen Outreach exhibited 52% (p<.001) of the odds of course failure compared to minorities in the comparison group. Program youth with prior academic suspensions had 43% (p<.001) of the risk of course failure compared to the comparison group youth, but for youth without prior academic suspensions this figure was 66%.

Academic suspension: Students participating in Teen Outreach exhibited 52% (p<.01) of the risk of academic suspension compared to the comparison group. No significant differences in the effectiveness of Teen Outreach were found by individual or behavior characteristics.

Effects of differential attrition: To assess whether differential attrition affected the results, the researchers re-ran the models using multiple datasets in which the missing data were imputed based on assumptions. In the pregnancy model, all the main effects and interactions remained significant. In the course failure model, participation and interactions remained significant through the model that assumed problem behaviors at twice the baseline rate, but not in a model that assumed problem behaviors at four times the baseline rate.

Study 2

Evaluation Methodology

Design: From 1991 through 1995, 25 Teen Outreach programs participated in this evaluation of the Teen Outreach program. Total participation included 342 Teen Outreach Program youth and 353 control group youth in grades 9 through 12.

In the spring prior to each academic year, all known Teen Outreach programs in the country were contacted and asked if they would participate in a random assignment evaluation of the program the following school year. Approximately 10% of the known sites (25 sites) eventually participated. Sites that refused to participate consistently reported three main reasons why they refused: a) random assignment of students was too awkward or unwieldy; b) they did not find enough interested students to fill both a control and a treatment group; and c) they found random assignment objectionable because it would require "turning away interested students."

At the early stages of data analysis, two sites had attrition rates between 80 and 90 percent and one site showed a significantly higher rate of problem behaviors for the control group than the program group. These three sites were removed, and 22 sites were used in the final analysis.

The 22 sites provided a sample of 342 students who participated in Teen Outreach and 353 who did not. Students indicated interest in the program by enrolling as part of the health curriculum, as an academic elective or as an after-school activity. After the students indicated their interest, they were randomly assigned to either the Teen Outreach program or the control group.

Over the academic year duration, attrition was 5.3% for the Teen Outreach program and 8.4% for members of the control group.

Sample characteristics: The typical student in the sample was about 16 years old, in 9th or 10th grade, female (roughly 85%), and black (about 67%). The high rate of female participation is explained by teachers and counselor's viewing the program as part of pregnancy prevention and therefore recommended more females than males.

Measures: Students filled out brief self-report questionnaires within a few weeks of the start of the academic year and within a few weeks of the end of the academic year. Demographic data collected included age, grade level, race/ethnicity (black, white, Hispanic, other), household composition (one- or two-parent), and parents' education level (four categories ranging from not graduating high school to college graduate). The problem behaviors questions included whether they had ever been pregnant or caused a pregnancy, whether they had failed any courses during the prior year, and whether they had been suspended academically in the prior year. In the posttest, the problem behaviors referred to the academic year that they were completing at the time.

Analysis: Logistic regression examined the effects of program participation on problem behaviors at exit (pregnancy, course failure, and academic suspension) while controlling for problem behaviors at entry and demographic characteristics.

Outcomes

Baseline Equivalence: No statistically significant differences in demographic characteristics existed across the groups. However, the control group exhibited significantly higher rates of problem behaviors at entry. The potential effects of this are discussed in the outcomes section.

Differential Site Attrition: No analysis was made to determine whether the three sites that dropped out of the study had students who were significantly different than the other sites demographically or in terms of problem behaviors at entry. However, an alternative analysis was conducted with the three sites included and all effects of the program remained at the same significance levels. Unfortunately, the method used to impute data in order to run the alternative analysis was not reported.

Differential Student Attrition: Comparing attriters with completers, the researchers found that students who dropped out of the remaining 22 program sites were not significantly different than those who completed the program except that they were more likely to have had or caused a prior pregnancy, been academically suspended, been younger and been male. Comparing attriters across group status found that attrition was not linked to being in the control or program group.

Posttest: The Teen Outreach program had statistically significant and moderate effects on problem behavior outcomes. The risk of each of the problem behaviors was less than half for the Teen Outreach than for the control group. Specifically, risk of pregnancy was 41% as large as the control group, risk of course failure was 42% of the control group, and risk of suspension was 39% of the control group.

Observed program effects could be the result of the Teen Outreach group having significantly lower rates of problem behavior at entry. Youth with fewer troubles may be more open to the program and may experience change more easily than the more troubled control group. To test this possibility, the researchers created a dichotomous variable that indicated whether the site had a higher number of problems at entry in the Teen Outreach group relative to the control group. The models were re-run with this new variable interacted with program participation. None of these interactions were significant, leading to the conclusion that the lack of equivalence between Teen Outreach and control group problem behaviors at entry was not responsible for the observed program effects.

Dosage effects: Two indicators of program intensity were the number of classroom sessions attended and the number of hours of volunteer service. The models were re-run with continuous variables that represented dosage in place of the program participation dichotomous variable. No significant effects were found for program intensity on pregnancy or academic suspension, but a significant effect was found for number of volunteer hours on course failure.

Limitations

The following limitations were found in this evaluation:

  • The site sample was not random. Sites that were able to manage the random assignment of youth were more likely to agree, which may indicate that the participating sites are better run, which may bias the results towards higher effect sizes.
  • The site sample was small. Only 10% (25) of the invited sites chose to participate.
  • The method used to impute data when assessing the effects of removing three sites from the analysis was not described, nor were the results of the analysis of imputed data presented.
  • Program intensity factors were substituted for program participation and found to be mostly insignificant.

Study 3

Evaluation Methodology

Design: This evaluation was designed to assess both individual and program-level factors that may influence the effectiveness of the Teen Outreach program in reducing problem behaviors in youth. Data were collected at the site- and the individual-level during the 1986-1987 academic/program year. Of the 70 Teen Outreach sites in the nation, the study sampled 35 sites in 30 schools. The authors did not indicate how the evaluation sites were selected, but reported that the sample was geographically disperse and the sampled sites did not differ significantly from the non-sampled site. The authors did not indicate which site characteristics were used to draw this conclusion.

Program participants could have entered the program by electing to participate in response to school-wide announcements or by agreeing to participate after they had been targeted by teachers or counselors. After the program youth were chosen, the comparison youth were quickly selected through one of two methods: (a) the program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) teachers and counselors who had made referrals to Teen Outreach identified other youth whom they thought had similar sociodemographic backgrounds and experienced similar problems as the youth chosen for the program.

Participants in the study included 632 Teen Outreach participants and 855 comparison students in grades 7-12. Attrition over the course of the program year was 2.4% among Teen Outreach students and 4.8% among comparison students. An additional 6.5% of the Teen Outreach students did not complete exit questionnaires and were excluded from the analysis. Consistent with the intent-to-treat principle, the analysis used all subjects with available data.

Sample characteristics: Students ranged in age from 11 to 19. The typical member of the sample was 16 years old and in 10th grade. Almost 70% of the sample was female, about half were white, and about a third were black. The mean parents' education for the sample was slightly more than a high-school graduate and slightly more than half of the youth lived in two-parent households.

Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).

In the entry questionnaire, students were asked a set of demographic questions (age, grade in school, race/ethnicity, mother's and father's education levels, and whether they lived in a two-parent household). Next, students were asked a set of questions about three specific problem behaviors (whether they had been suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant).

In the exit questionnaire, they were asked the same questions regarding problem behaviors, but the time reference period was the academic year just ending. In addition to the three primary problem behaviors, they were also asked if they intended not to return to school in the fall.

The researchers chose to sum these behaviors into an entry problem behavior syndrome score (ranging from 0 to 3 problem behaviors) and an exit problem behavior syndrome score (ranging from 0 to 4 problem behaviors). By doing so, they addressed the difficulty of having low base rates on individual problem behaviors and also were consistent with the understanding at the time that a "syndrome" was more meaningful than individual inter-correlated behaviors. The mean entry problem behavior syndrome scores were .80 for the Teen Outreach group and .66 for the comparison group (p<.05).

Site-level data were collected from facilitators at each site across the following dimensions:

  • Intensity measures: the average number of volunteer hours worked and the average number of classroom hours;
  • Structural measures: whether the program was offered during school or after school and whether it was for-credit or not-for-credit; and
  • Curriculum measures: a four point likert-scale measured how much (none, a little, a lot, almost all) facilitators used the curriculum overall and by curriculum category. The categories were identified by factor analysis to be an orientation section, a self-awareness section, a social-awareness section, and a developmental section.

Analysis: The data used for this study were multi-level in nature, but the analysis was not. Individual students were nested within programs and each was assigned the average program characteristics from that site. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify the role of site-level factors in the efficacy of the program. As such, the regression models were run with individual-level characteristics as controls, with site-level intensity measures as explanatory factors, with site-level structural measures as explanatory factors, and with site-level curriculum used as explanatory factors.

Outcomes

Baseline Equivalence: Compared to the Teen Outreach participants, the comparison group youth had fathers with a slightly higher level of education, were more likely to live in a two-parent household, were less likely to have failed courses in the prior year, and had fewer problem behaviors on average.

Differential Attrition: Without reporting figures, the authors report "no significant effects of loss-to-follow-up on demographic or problem behavior measures, nor were there significant interactions of loss-to-follow-up with membership in the program versus comparison group for any of these measures."

Posttest: At the beginning of the program, the Teen Outreach participants had more problem behaviors, on average, than the comparison group (p<.001). At program exit, the Teen Outreach participants had fewer problem behaviors, on average, than the comparison group (p<.05). Thus, compared to non-participants, participating in Teen Outreach was associated with a reduced number of problem behaviors.

The average number of volunteer hours at a site had a significant and positive but weak effect on reducing the number of problem behaviors (beta=-.09, p<.05). The number of classroom hours also had a positive effect in reducing the number of problem behaviors, but the effect was stronger for the younger participants (beta for grade x class hours = .10, p<.05). Whether a program was offered during the school day for credit or after-school as an activity had no significant impact in the effectiveness of the program in reducing problem behaviors. Also, the effectiveness of the program was not significantly altered by the amount of the overall curriculum used or by different usage of the categories of the curriculum.

Limitations

The major limitations of this study include:

  • The evaluation's overall lack of generalizability. This inability to generalize is due to student self-selection into the program, lack of clarity around site selection, and demographic characteristics of the sample that do not generalize to the population.
  • The estimates of program-level factors are made using the individual as the unit of analysis. Thus, standard errors are likely under-estimated and the results are likely biased upwards.
  • Non-equivalence exists between the treatment group and comparison group, possibly because neither group was randomly selected.
  • Control group participants were nominated by program participants or teachers rather than randomly selected.
  • Using a problem behavior syndrome approach assumes that each of the behaviors is weighted the same. Further, the impact of the program cannot be differentiated between reducing pregnancy or academic failures.

Study 4

Evaluation Methodology

Design: This quasi-experimental study used individual and site-level data from 66 different Teen Outreach programs across the country. The researchers did not indicate how the 66 sites were selected or from which program year(s) the data were drawn.

Participants enrolled in Teen Outreach by a variety of ways: it could have been part of the required curriculum, they could have chosen it as an elective or taken it as an after-school activity; or they could have agreed to take the program in response to a recommendation from a teacher or guidance counselor. Previous research has shown that the method of entry into the program is unrelated to outcomes. After choosing the program youth, the investigators quickly selected comparison youth through one of three methods:

  1. Program participants nominated youth who they thought would fill out the questionnaire the same way they did; or
  2. Program participants nominated youth whom they thought were from similar backgrounds and encountered similar life problems; or
  3. Comparison youth were randomly selected from a pool of youth who had wanted to enroll in the Teen Outreach but the program was already at capacity.

Within the 66 sites, data were collected from 1,020 Teen Outreach youth and 1,013 comparison group youth in grades 7 - 12. Attrition was 2.4% among Teen Outreach students and 2.7% among comparison students.

Sample characteristics: Students ranged in age from 11 to 19 years old. The typical member of the sample was 16 years old and in the 10th grade. About 70% of the sampled students were female, and roughly 40% were black and 40% were white. The mean parents' education level was just above a high school graduate and slightly more than half of the youth came from two-parent households.

Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June). Students were asked their age, their grade in school, their race/ethnicity, their parents' education level (1=not a high school graduate through 4=college graduate), whether they lived in a two-parent household, whether they had been academically suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant. In the exit questionnaire, the reference time period was the academic year just ending.

To measure the degree to which programs fostered senses of autonomy and relatedness, a questionnaire was administered to students and to facilitators during the final month of the program. This questionnaire was intended to measure (a) the extent to which the program was viewed as promoting adolescent autonomy and relatedness; and (b) whether the quality of the volunteer experience was likely to enhance the students' sense of competence and autonomy.

The questionnaire was structured similar to the Perceived Competence Scale for Children, which is intended to reduce the effects of a pull for social desirability. For each item, two contrasting stems were presented side by side (e.g., "some kids feel like their facilitators like them a lot" and "other kids feel like their facilitator just likes them OK."). The youth were asked to pick one of the two statements that they most agreed with and then decide whether it was "really true" or "sort of true." The autonomy and relatedness questions focused on whether the youth felt like they could make their own decisions within the program, whether their views were respected, etc. The quality of volunteer experience questions focused on whether the youth felt like they were learning new skills, whether they might like to continue volunteering after the program ended, etc. The two scales had adequate internal consistency (alpha=.87 and alpha = .72, respectively).

The questionnaire was also given to facilitators who were asked to answer as they thought their class would answer. The adolescent autonomy and relatedness scale had an alpha of .81 and the quality of volunteer experience scale had an alpha of .52. The authors report that the "validity of this inventory approach has been established in other studies, in spite of its reduction of alpha levels".

Students were given the standard problem behavior questionnaire at the beginning of the program and at the end. Both program youth and facilitators were given the autonomy and relatedness questionnaire and the quality of volunteer experience questionnaire at some point in the last month of the program. Students were assured of confidentiality and that their data would not be shared with school officials.

Analysis: The data used for this study were multi-level in nature. Individual students were nested within programs and each was assigned the average program values for program level measures. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify whether feelings of autonomy and connectedness and quality volunteer experiences mediated the program's effectiveness. Also, to control for potential unmeasured site-wide factors, the authors included residualized problem behavior change scores for the comparison group for each site.

Linear regression models were run with the number of problem behaviors at exit as the dependent variable. The covariates were entered sequentially, as follows: individual-level data (number of problem behaviors at entry, grade level, residualized problem behavior change score); then the primary variable of interest (promotion of autonomy and relatedness scale in regression #1 and quality of volunteer experience in regression #2); and, finally, interactions of grade level with the primary variable of interest. Each model was run twice, once using student ratings and once using facilitator ratings.

Outcomes

Baseline Equivalency: The Teen Outreach participants and the comparison group were very similar with respect to demographic characteristics and program behaviors at entry. The only statistically significant difference was that the proportion of females was slightly higher among the Teen Outreach group. This difference can be attributed to adult referrals into a program that they view as primarily a pregnancy prevention program.

Differential Attrition: Attrition was similarly low in both the treatment and comparison groups.

Posttest: At program entry, there was no significant difference in the mean number of problem behaviors (ranging from 0 to 3) between the Teen Outreach youth and the comparison youth (.61 vs.64, respectively). After controlling for problem behaviors at program entry, the mean number of problem behaviors was significantly lower (p<.001) for youth in the Teen Outreach program (.48 vs.67, respectively).

When testing for site-level effects, the researchers used averaged site-level data from both the students and facilitators. In the student rating model, programs with higher scores on the promotion of autonomy and relatedness scales were significantly more effective in reducing problem behaviors (p<.01). However, this effect was only true for middle school students (p<.05). Likewise, in the facilitator model, the only significant effect of promotion of autonomy and relatedness was for middle school students (p<.01).

The regressions were run again with the quality of volunteer experience as the key explanatory variable. In the student rating model, programs with higher scores on the quality of volunteer experience scale were significantly more effective in reducing problem behaviors ( p<.05), but once again this effect only held for middle school students. In the facilitator rating model, no significant effects of the quality of volunteer experience was found. Additionally, the effect of the mean number of volunteer hours on program success was not significant.

The authors conducted an analysis to determine if the small effect found for quality of volunteer experience (i.e., enhancing autonomy and relatedness) on reducing problem behaviors for middle school students was a mediating effect of the volunteer experience itself. The authors conclude that, in fact, a portion of the volunteer experience effect is mediated through the perceived autonomy and relatedness aspects of the program.

In sum, the proposed psycho-social mechanisms were significant only for middle schoolers. This was surprising because previous research has shown that the overall Teen Outreach program is more effective, in general, for high school youth. The authors concluded that middle school youth are more in need of an environment that fosters this sort of development, while high school students may already have achieved a certain level of autonomy and relatedness. High school youth are therefore less dependent on getting this specific developmental competency within the program and instead benefit from other aspects of the program.

Limitations

The major limitations of this study include:

  • The evaluation's overall lack of generalizability. This inability to generalize is due to student self-selection into the program, lack of clarity around site selection, and demographic characteristics of the sample that do not generalize to the population.
  • The estimates of program-level factors are made using the individual as the unit of analysis. Thus, standard errors are likely under-estimated and the results are likely biased upwards.
  • Non-equivalence exists between the treatment group and comparison group because neither group was randomly selected.
  • Control group participants were nominated by program participants or teachers rather than randomly selected.
  • There is no independent measure of autonomy and relatedness. The measure used in this study substantially overlaps with the students' negative or positive experience of the program.
  • Success in the Teen Outreach program may influence the measure of autonomy and relatedness rather than vice-versa.

Study 5

Evaluation Methodology

Design:

Recruitment: Schools were recruited for participation if they were located in nonmetropolitan Florida counties (those with a population of less than 900,000), if they had the capacity to implement the program, and if they had higher rates of adolescent births, STIs, high school dropout, graduation, or out-of-school suspension rates compared to other communities. A total of 28 public schools within 12 counties were selected for program implementation, with two cohorts of 9th grade classes recruited for participation (n=9,220 students eligible, 7,667 consenting).

Assignment: The 28 schools were matched into pairs based on county, courses offered, school size, region/proximity, and presence of block scheduling, and then randomly assigned to either the intervention (n=14 schools, 3,533 students) or control group (n=14, 4,134). In the intervention group, the intervention was delivered as a supplement to the usual health curriculum while control schools continued with health education as usual.

Attrition: One pair of schools (1 treatment, 1 control) dropped out during the first year. At follow-up, cohort 1 had a retention rate of 85% (n=3,404) and cohort 2 had a retention rate of 75% (n=2,758).

Sample:

Across both cohorts the sample was evenly split on gender with a mean age of 14.6 at baseline. Nearly 60% of respondents were white, with 20% reporting as Hispanic, 10% as Black, and the remaining 10% as other races/ethnicities. Approximately 15% of the sample reported recent sexual activity; of those who reported recent sexual activity, 27% reported recent risky sex. About 32% of the entire sample intended to have sex within the next year, with 8% of those intending risky sex in that same time frame.

Measures:

Participants were given a series of self-report questions at baseline and at posttest, approximately 9 months after baseline. Outcomes of interest were: ever had sex, sexual intercourse in the previous 3 months, not using a condom in the previous 3 months, and ever been pregnant or gotten someone pregnant. Secondary outcomes included intention to have sex in the next year, and, among those who intended to have sex in the next year, their intention to use a condom or not.

Analysis:

The effects of the program were evaluated using mixed-effects logistic regression models with adjustments for clustering in schools, controlling for baseline demographics and behaviors. Many different subgroup impacts were examined, including differential impacts of the program across cohorts, genders, and by baseline behaviors.

Intent-to-Treat: Multiple imputation was used to account for missing data, with all cases used in the analysis.

Outcomes

Implementation Fidelity:

Fidelity was not assessed.

Baseline Equivalence:

There were significant differences between groups in racial/ethnic makeup, but no other baseline differences in demographics or behaviors.

Differential Attrition:

There was a higher rate of attrition in the second cohort than the first, and while the study states that they test for condition equivalence of those retained at posttest, the results of the attrition analyses were not presented.

Posttest:

Overall, at posttest, there were no significant differences between the treatment and control groups on any measures using the pooled Cohort 1 and Cohort 2 data. However, subgroup impacts were much more positive, especially in Cohort 1. Treatment participants in the first cohort were significantly less likely than control group youth to report recent sex, risky sex, and sexual intentions. Females showed lower odds of recent and risky sex, while males showed decreased odds of recent sex. Of those inexperienced at baseline, intervention group participants were significantly less likely to report both ever having had sex and recent sex; females were significantly less likely to report ever having been pregnant, ever having sex, and having recent sex. Of those in Cohort 1 considered experienced at baseline, intervention participants were significantly less likely to report risky sex; females were significantly less likely to report recent sex and risky sex. There was an iatrogenic effect of the program for Cohort 1 males who were sexually experienced at baseline, as they actually displayed greater odds of risky sex than their control group counterparts.

The intervention participants in Cohort 2 reported significantly reduced intentions to have risky sex compared to the control group. However, of those considered experienced at baseline, intervention participants had significantly higher odds of recent sex than those in the control group-an iatrogenic effect.

Long-Term:

The follow-up assessment occurred shortly after treatment ended.

Study 6

Evaluation Methodology

Design:

Recruitment: The researchers recruited 4769 youth in Louisiana and 966 youth in Rochester via convenience sampling methods. Recruitment took place in school and community settings during the academic years of 2013-2015, and the sample contained three "annual cohorts." Eligible participants included those who lived in the targets areas, spoke English well enough to complete the study materials, and had not previously participated. In Louisiana youth aged 12 to 17 years were included and in Rochester youth aged 11 to 14 years were included.

Assignment: Individuals were randomly assigned to the conditions in Louisiana and 33 clusters were randomly assigned to the conditions in Rochester. In Louisiana the control condition received no programming, and in Rochester the control condition received a work-readiness intervention.

Attrition: Of the 4769 youth in Louisiana, 2428 (51%) completed the follow-up survey, and of the 966 youth in Rochester, 824 (85%) completed the follow-up survey.

Sample:

The sample targeted areas with high rates of adolescent pregnancy and included multiple regions across Louisiana and Rochester. In Louisiana, the sample was 90% black and 60% female, with an average age of 13.9 years (SD = 1.59). In Rochester, the sample was 64% black and 52% female with a mean age of 12.3 years (SD = 1.11).

Measures:

Participants completed self-report outcome measures. Sexual activity was measured through a yes/no question about whether the participant had previously engaged in intercourse; the posttest initiation measure included only those who had not already had intercourse at baseline. A yes/no question asked about whether the participant had engaged in intercourse without birth control in the previous 3 months.

Analysis:

Logistic regression models were used to analyze predictors of the binary outcomes, with gender, age, race, ethnicity, and cohort year used as covariates. The baseline outcome was included in models of the use of birth controls but not for initiation of sexual intercourse (all analyzed subjects had not previously engaged in sexual intercourse). The analysis was done at the individual level with multivariable logistic regression in Louisiana, and at the clustered level with multilevel modeling in Rochester.

Intent-to-Treat: The researchers stated that they used intent-to-treat analyses by including all individuals who completed the posttest. However, they also stated that "During the course of the study, 3 study sites were dropped from the program."

Outcomes

Implementation Fidelity:

The researchers reported that "process measures" were recorded after each session by facilitators to assess fidelity. Approximately 10% of sessions were observed and evaluated as well and high ratings were given by evaluators. However, only 21% of youth in Louisiana and 48% of youth in Rochester in the intervention condition attended at least 75% of the sessions.

Baseline Equivalence:

The researchers tested for baseline equivalence between the intervention and control conditions on demographic variables and outcome measures and found no significant differences in Louisiana. In Rochester, the intervention group included significantly more Hispanics. However, the tests used the analysis sample of those with posttest data rather than the randomized sample.

Differential Attrition:

Differential attrition was not analyzed directly. Tests for baseline equivalence for the analysis sample are relevant but lacked comparison to the randomized sample, included only 5-6 baseline measures, and showed one difference.

Posttest:

No significant effects on outcome variables were found.

Long-Term:

Long-term effects were not tested.

Study 7

Evaluation Methodology

Design:

Recruitment: Participants were recruited from eight Boys & Girls clubs in Denver over 4 years. Eligible participants included active club members between 14 and 18 years of age. A total of 3643 youth fit all eligibility criteria, and of these, 852 consented to participate in the study.

Assignment: Randomization was clustered by site and by year, creating 32 clusters in total. Each site was assigned to the intervention condition for two of the years and to the control condition for the other two years. The intervention sites received TOP programming plus the text-message-based supplement, and control sites received just the TOP programming. The sample included 436 in the intervention condition and 315 in the control condition at baseline.

Attrition: Participants completed self-report measures at baseline and immediately at program completion. At posttest, 72.7% of the intervention group and 75.7% of the control group completed the assessment.

Sample:

The clubs primarily served minority youth and those living in poverty. The mean age was 14.90 (SD = 1.02) in the intervention condition and 14.98 (SD = 1.14) in the control condition. The intervention condition was 42.41% Hispanic, 20.25% black, and 9.18% white, and the control condition was 43.81% Hispanic, 18.73% black, and 7.94% white.

Measures:

Measures were administered through self-reports. Participants reported how many times they had had sex in the previous three months and how many of these times were without protection. They also reported whether they had "accessed contraceptive or sexually transmitted infection services in the past 9 months" and whether they had gotten pregnant. The study reported no information on the validity and reliability of the measures.

Analysis:

Multilevel regression models were used to analyze the outcome variables, with a large number of covariates including the baseline outcomes. Sub-analyses compared Hispanic to non-Hispanic participants. Bonferroni-Hochner corrections were used to correct for multiple tests.

Intent-to-Treat: The authors stated that subjects were used in the originally assigned conditions, regardless of the level of engagement with the program. Also, sensitivity analysis included missing data with full information maximum likelihood estimation and gave the same results.

Outcomes

Implementation Fidelity:

No information on implementation fidelity was given, but the subjects attended an average of 11.9 sessions of the 25.

Baseline Equivalence:

Table 2 showed only 1 significant difference out of 22 tests, but the table used the analysis sample. The significant difference was for pregnancy.

Differential Attrition:

Differential attrition was tested by condition and by 22 baseline variables. Those remaining in the study reported higher rates of protected sex and living with both parents and lower likelihood of having sex. Additionally, intervention participants lost to follow-up were significantly older than those who remained in the study. However, tests for the interaction of the baseline variables and condition showed that the differential attrition was similar across conditions. The tests for baseline equivalence using the analysis sample also suggest a limited influence of attrition.

Posttest:

The program showed no significant effects for the full sample. Hispanic participants in the intervention condition reported significantly fewer pregnancies than those in the intervention condition, but the authors noted that this effect may be influenced by the significant baseline difference between conditions for pregnancy.

Long-Term:

No long-term effects were reported.

Contact

Blueprints for Healthy Youth Development
University of Colorado Boulder
Institute of Behavioral Science
UCB 483, Boulder, CO 80309

Email: blueprints@colorado.edu

Sign up for Newsletter

If you are interested in staying connected with the work conducted by Blueprints, please share your email to receive quarterly updates.

Blueprints for Healthy Youth Development is
currently funded by Arnold Ventures (formerly the Laura and John Arnold Foundation) and historically has received funding from the Annie E. Casey Foundation and the Office of Juvenile Justice and Delinquency Prevention.