A nine month program that engages high school students in a minimum of 20 hours of community service learning annually with weekly meetings. The goal is to reduce rates of teen pregnancy, course failure, and academic suspension.
Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
SAMHSA : 2.2-2.3
Tori Gale
Director, Partner Development
Wyman Center
600 Kiwanis Drive
Eureka, Missouri 63025
Email: Tori.Gale@wymancenter.org
Website 1: www.wymancenter.org
Website 2: https://wymancenter.org/national-network/top/
Original Developer: Brenda Hostetler
Current Owner/Purveyor: Wyman Center
Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. As a result, teens are better able to navigate challenges during the teenage years - a time when decisions matter.
TOP is designed to meet the developmental needs of middle (6th-8th grades) and high school (9th-12th grades) teens in a variety of settings, including in school, after-school, through community organizations or in systems and institutional settings. Curriculum topics include: emotion management, problem-solving, decision-making, goal-setting, health and wellness, healthy decision making, self-understanding, social identity, empathy, communication, relationships and community.
Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. TOP aims to achieve four goals:
These goals are accomplished through implementation of the following core program components:
The TOP curriculum includes lessons that address content on adolescent development: skill-building, connections with others, and learning about one's self. Within each area of content, the curriculum includes foundational, intermediate, and advanced lessons that are designed to meet the developmental needs of young people in grades 6 through 12. Each of the 120+ lessons is written for a 45-minute time period with an additional extension activity available if time allows. The TOP curriculum is designed to allow for sequenced programming over multiple years (e.g., as part of a whole-school middle school approach), with a minimum duration of 9 months.
The community service component has four phases: preparation, action, reflection, and celebration. Most of the time is spent in action, with reflection happening continuously throughout the community service experience. Service can involve direct action (e.g., tutoring, building a playground or garden), indirect action (e.g., bake sales, blood drives), and civic action for older students (e.g., voter registration, public speaking, educational theatre and awareness programs). There is a process through which students are able to choose the specific type of service they want to engage in.
Two studies assessed the effect of the Teen Outreach Program® on the three different problem behaviors (Allen et al, 1997; Allen and Philliber, 2001):
Studies with No Effects
Study 1
Allen and Philliber (2001) conducted a mixed randomized and quasi-experimental design study with 3,277 high school youth. Self-report assessments of pregnancy, course failure, and academic suspension were administered within the first few weeks of the beginning of the program and academic year, and again at program exit at the end of the academic year.
Study 2
Allen et al. (1997) conducted a randomized controlled trial with 695 adolescents across 25 sites nationwide. The randomization assigned students individually and, less frequently, by classrooms to an intervention group (n = 342) or control group (n = 353). Self-report assessments of pregnancy, course failure, and academic suspension were conducted at baseline and 9 months later (posttest).
Study 3
Allen et al. (1990) conducted a quasi-experimental study with 1,487 youth in grades 7-12. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).
Study 4
Allen et al. (1994) conducted a quasi-experimental study with 2,033 youth (grades 7-12) across 66 sites. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).
Study 5
Walsh-Buhi et al. (2016) tested the program using a cluster randomized trial of 28 public schools in 12 non-metropolitan Florida counties. A total of 7,667 students across two cohorts participated in the program. Primary outcomes regarding sexual activity included ever having had sex, recent sex, recent risky sex, and intention to have risky sex, with assessments conducted at baseline and at posttest, approximately 9 months after baseline. Participants in Walsh-Buhi et al. (2016) included 2 cohorts of youth. For cohort 1 youth, the baseline survey was administered in fall 2012, and follow-up survey was administered in spring 2013. For cohort 2 youth, the baseline was administered in fall 2013 and follow-up was administered between May and September 2014. Daley et al. (2015) reported posttest and follow-up results for cohort 1 only (n = 4,063 youth). Daley et al. (2019) also reported on follow-up results (10-12 months post-program implementation) for cohort 1.
Study 6
Robinson et al. (2016) evaluated the program in a community setting. The researchers randomly assigned 4,769 youth from Louisiana and 966 youth from Rochester, NY to the intervention and control conditions. Measures of sexual initiation and use of birth control were collected through self-reports at baseline and immediately after the intervention period.
Study 7
Bull et al. (2016) conducted a cluster randomized control trial between September 2011 and September 2014 with 852 youth across 32 Boys & Girls clubs in Denver, CO. The control condition consisted of TOP programming alone, and the intervention condition consisted of TOP programming plus the text message supplement. Participants completed self-report measures at baseline (pretest) and immediately at program completion, after four years of programming (posttest).
Study 1
Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach program. Journal of Community Psychology, 29(6), 637-655.
Study 2
Allen, J., Philliber, S., Herrling, S., & Kuperminc, G. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729-742.
School: Low school commitment and attachment
Individual: Academic self-efficacy, Problem solving skills, Prosocial behavior*, Prosocial involvement*, Skills for social interaction
School: Opportunities for prosocial involvement in education*
Neighborhood/Community: Opportunities for prosocial involvement*, Rewards for prosocial involvement
*
Risk/Protective Factor was significantly impacted by the program
See also: Wyman's Teen Outreach Program Logic Model (PDF)
Teen Outreach Program® Facilitator Training is designed for those who will directly deliver TOP® to teens. The primary goal of this training is to prepare facilitators to begin their TOP® club(s) and to know where they can access additional information and support. Those supervising the implementation of TOP® may also benefit from attending so they are clear on the messages delivered in training.
TOP® Facilitator Training Content includes:
The TOP® Facilitator Training does not include:
It is highly recommended that providers select facilitators with experience and proven skills working with teens. Wyman operates under the assumption that providers will hold their own staff orientations and trainings specific to the population they serve and the settings in which their programs operate.
The TOP® Training of Trainers is designed for those who will deliver the TOP® Facilitator Training in their network. The primary goal of this 5-day training is to prepare trainers to be able to deliver the content and approach of the TOP® Facilitator Training and to share where additional information and support may be found. Those supervising TOP® trainers may also benefit from attending so they are clear on the messages delivered in the training.
TOP® Training of Trainers content includes:
TOP® Training of Trainers content does not include the following and these can be addressed with individual partner technical assistance:
It is highly recommended that partners select trainers with experience and proven skills in facilitation and training adults. Wyman operates under the assumption that partners will hold their own staff orientations and trainings specific to planning and organizing their network of TOP® trainers, providers and clubs.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
Year 1 costs: Inclusive of 1st year license, 5 curriculum sets, 1 Training of Trainers (TOT), 2 Training of Facilitators (TOF), program implementation guidance and monitoring, access to online data collection and reporting system = $17,500 for up to 500 teens served. Costs increase by $1,000 for every 250 teens served beyond 500.
Five curricula are included in Year 1 costs. Additional curricula are available only to Certified Replication Partners and their provider organizations. Individual sets are $700 plus shipping costs (10% of total if under $1,000; 5% of total if over $1,000).
Materials Available in Other Language: The TOP Curriculum and the TOP Teen Pre and Post Surveys are available in Spanish.
Licensing is included in Year 1 costs.
It is recommended that organizations give a least 4-6 weeks of program planning/start-up time to account for training to be completed and program recruitment (if applicable).
Space is usually donated by schools or other interested organizations. If not donated, costs for space will vary.
Year 2 and beyond: Additional curricula beyond 5 sets included in Year 1 costs are available for $700 plus shipping costs (10% of total if under $1,000; 5% of total if over $1,000).
Program materials and art supplies are estimated at $2,000 per TOP Club. Costs for snacks/drinks and teambuilding supplies should also be considered.
In addition to salary costs of facilitators and coordinators, we recommend organizations plan for 20% of salary for benefits and a minimum of a 3% annual salary increase for staff. An evaluation or data specialist is not required, however, some organizations include this role when staffing their program.
It is recommended that $2,000 per group/club should be budgeted for costs of transportation of participants to and from community service learning experiences.
Ongoing training after initial training is not required, however, it is recommended that organizations plan for professional development of staff to ensure quality of programing. Costs for additional training provided by Wyman include:
Access to webinars, check-in calls, and email inquiries are included in the annual license fee for program monitoring.
Specialized technical assistance is available and generally starts at $175/hr but varies based on project needs.
The cost for fidelity monitoring and evaluation is included in Year 1 and annual renewal fees.
Year 2 and beyond: Licensing, fidelity monitoring and evaluation is $5,500 per year for up to 500 teens. Costs increase $1,000 for every 250 teens served beyond 500.
No information is available
No information is available
This example describes a sponsoring organization supporting 10 facilitators each with 3-4 groups of 25 youth participants offering groups for 30 weeks during the academic year for the first year of operation. Each group would have a co-facilitator. The groups would reach 875 teen participants. Costs would include:
| Replication Partner Certification | $17,500.00 |
| Supplies @ $2000 per group | $70,000.00 |
| Staff-35 groups x 4 hours x $25 x 2 facilitators | $168,000.00 |
| Transportation @ $2000 per group | $70,000.00 |
| Local Trainer plus fringe | $90,000.00 |
| Administrative Overhead @ 10% of Staff Cost | $26,000.00 |
| Total One Year Cost | $441,500.00 |
With 875 youth participants, the per youth cost is $505.
The above is a good example of a community-based or after-school model. In-school implementation typically reduces the staffing costs by over half, since a classroom teacher is often the 2nd facilitator. Transportation costs are also much lower.
Wyman's Teen Outreach Program is a youth development program with evidence that it prevents teen pregnancy, school drop-out, course failure, and suspension. Funding streams that support youth development and after school programs, middle and high school education, and pregnancy prevention are most commonly used to support the program. TOP also has a strong emphasis on community service so federal service-learning funding streams as well as private foundations interested in community service are potential sources of support for the program.
Funding available for youth development, and pregnancy and sexually-transmitted disease prevention may be considered to fund Wyman's Teen Outreach Program. To the extent that existing pregnancy prevention and youth development programs are not evidence-based, a jurisdiction can consider redirecting funds toward Wyman's Teen Outreach Program to get better outcomes. This could potentially be achieved by training and supporting staff in existing after school and pregnancy prevention programs to help them integrate use of the TOP model in their program. TOP can also be implemented during the school day, with the redirection of teacher or school counselor time to facilitate the program.
State grant funds supporting youth development and pregnancy prevention programs may be considered for TOP. Some state health departments fund TOP through teen pregnancy and violence prevention grants.
Formula Funds:
Discretionary Grants: Relevant discretionary grants include grants focused on pregnancy prevention that are administered by the Department of Health and Human Services Office of Adolescent Health (OAH) and Family and Youth Services Bureau (FYSB), and the Centers for Disease Control grants for replication of evidence-based programs for teen pregnancy prevention. In addition, Learn and Service grants administered by the federal Corporation for National and Community Service can potentially support TOP, given its focus on service learning. Finally, the strong education and drop-out prevention outcomes align with discretionary grants focused on drop-out prevention and academic achievement in high need communities that are administered by the Department of Education.
Foundations, particularly those with a focus on pregnancy prevention, youth development, can be a good source of funding for TOP. Foundations with a particular interest in evidence-based interventions should be explored.
Several fund raising approaches might be useful in supporting TOP. These could include fundraising by local civic organizations, and local businesses and industries. Schools or community organizations with existing youth development programs could seek donors or hold fundraising events to support the relatively low cost of training and curriculum, using that training to improve the quality of existing programming.
All information comes from the responses to a questionnaire submitted by the purveyor, The Wyman Center, to the Annie E. Casey Foundation.
Original Developer: Brenda HostetlerCurrent Owner/Purveyor: Wyman Center
A nine month program that engages high school students in a minimum of 20 hours of community service learning annually with weekly meetings. The goal is to reduce rates of teen pregnancy, course failure, and academic suspension.
TOP is open to all 9th-12th graders who attend schools that have adopted the program. The developers have implemented and evaluated this program with middle school youth as well; however, these evaluations failed to meet Blueprints quality standards.
Results provided by gender (male/female) and race (black/white/Hispanic/other). Bull et al. (2016) found a program effect only for Hispanic participants.
Teen Outreach is not targeted to youth with specific risk factors. Rather, the program views adolescence itself as a risk factor. Protective factors include enhanced feelings of autonomy and connectedness with adults, community, and school through community service.
School: Low school commitment and attachment
Individual: Academic self-efficacy, Problem solving skills, Prosocial behavior*, Prosocial involvement*, Skills for social interaction
School: Opportunities for prosocial involvement in education*
Neighborhood/Community: Opportunities for prosocial involvement*, Rewards for prosocial involvement
*Risk/Protective Factor was significantly impacted by the program
Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. As a result, teens are better able to navigate challenges during the teenage years - a time when decisions matter.
TOP is designed to meet the developmental needs of middle (6th-8th grades) and high school (9th-12th grades) teens in a variety of settings, including in school, after-school, through community organizations or in systems and institutional settings. Curriculum topics include: emotion management, problem-solving, decision-making, goal-setting, health and wellness, healthy decision making, self-understanding, social identity, empathy, communication, relationships and community.
Wyman's evidence-based Teen Outreach Program® (TOP®) is a positive youth development program designed to build teens' educational success, life and leadership skills, and healthy behaviors and relationships. TOP aims to achieve four goals:
These goals are accomplished through implementation of the following core program components:
The TOP curriculum includes lessons that address content on adolescent development: skill-building, connections with others, and learning about one's self. Within each area of content, the curriculum includes foundational, intermediate, and advanced lessons that are designed to meet the developmental needs of young people in grades 6 through 12. Each of the 120+ lessons is written for a 45-minute time period with an additional extension activity available if time allows. The TOP curriculum is designed to allow for sequenced programming over multiple years (e.g., as part of a whole-school middle school approach), with a minimum duration of 9 months.
The community service component has four phases: preparation, action, reflection, and celebration. Most of the time is spent in action, with reflection happening continuously throughout the community service experience. Service can involve direct action (e.g., tutoring, building a playground or garden), indirect action (e.g., bake sales, blood drives), and civic action for older students (e.g., voter registration, public speaking, educational theatre and awareness programs). There is a process through which students are able to choose the specific type of service they want to engage in.
Teen Outreach is a preventive program that aims to use a developmentally-based approach to achieve positive outcomes in preventing pregnancy and academic failure. A developmental perspective suggests that successful adolescent adjustment (i.e. fewer problem behaviors) requires a simultaneous sense of autonomy and relatedness among peers and adults. In Teen Outreach volunteer activities, adolescents find themselves in help-giving rather than help-receiving situations. This, in turn, enhances their sense of autonomy and self-efficacy while at the same time fosters bonds with the community and school. The qualities of autonomy and relatedness are viewed as essential tools in navigating adolescence and making sound decisions around school and sexuality.
Study 1
Allen and Philliber (2001) conducted a mixed randomized and quasi-experimental design study with 3,277 high school youth. Self-report assessments of pregnancy, course failure, and academic suspension were administered within the first few weeks of the beginning of the program and academic year, and again at program exit at the end of the academic year.
Study 2
Allen et al. (1997) conducted a randomized controlled trial with 695 adolescents across 25 sites nationwide. The randomization assigned students individually and, less frequently, by classrooms to an intervention group (n = 342) or control group (n = 353). Self-report assessments of pregnancy, course failure, and academic suspension were conducted at baseline and 9 months later (posttest).
Study 3
Allen et al. (1990) conducted a quasi-experimental study with 1,487 youth in grades 7-12. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).
Study 4
Allen et al. (1994) conducted a quasi-experimental study with 2,033 youth (grades 7-12) across 66 sites. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).
Study 5
Walsh-Buhi et al. (2016) tested the program using a cluster randomized trial of 28 public schools in 12 non-metropolitan Florida counties. A total of 7,667 students across two cohorts participated in the program. Primary outcomes regarding sexual activity included ever having had sex, recent sex, recent risky sex, and intention to have risky sex, with assessments conducted at baseline and at posttest, approximately 9 months after baseline. Participants in Walsh-Buhi et al. (2016) included 2 cohorts of youth. For cohort 1 youth, the baseline survey was administered in fall 2012, and follow-up survey was administered in spring 2013. For cohort 2 youth, the baseline was administered in fall 2013 and follow-up was administered between May and September 2014. Daley et al. (2015) reported posttest and follow-up results for cohort 1 only (n = 4,063 youth). Daley et al. (2019) also reported on follow-up results (10-12 months post-program implementation) for cohort 1.
Study 6
Robinson et al. (2016) evaluated the program in a community setting. The researchers randomly assigned 4,769 youth from Louisiana and 966 youth from Rochester, NY to the intervention and control conditions. Measures of sexual initiation and use of birth control were collected through self-reports at baseline and immediately after the intervention period.
Study 7
Bull et al. (2016) conducted a cluster randomized control trial between September 2011 and September 2014 with 852 youth across 32 Boys & Girls clubs in Denver, CO. The control condition consisted of TOP programming alone, and the intervention condition consisted of TOP programming plus the text message supplement. Participants completed self-report measures at baseline (pretest) and immediately at program completion, after four years of programming (posttest).
Study 1
Allen and Philliber (2001) found that pregnancy prevention effects were similar across all youth except that those who were already parents benefited more than non-parents. With respect to preventing course failure, Teen Outreach was only effective for females, was more effective for minorities than for whites, and for those who had prior academic suspensions than for those without prior suspensions. No significant differences in the effectiveness of Teen Outreach in preventing academic suspension were found.
Study 2
Allen et al. (1997) found that the Teen Outreach Program had statistically significant effects on all three problem behavior outcomes. The intervention group had fewer pregnancies, course failures, and school suspensions than the control group.
Study 3
Allen et al. (1990) sought to identify what Teen Outreach programmatic conditions were most effective in producing successful outcomes. No programmatic differences were found in effectiveness whether the program was offered during the normal class day or after-school or whether the program used a little or a lot of the Teen Outreach curriculum. However, the number of volunteer hours and number of classroom hours (especially for younger participants relative to older participants) had a significant and positive influence in reducing problem behaviors.
Study 4
The hypothesized psycho-social mechanism of an increased sense of autonomy and relatedness among peers and adults was found to be significant, but only among middle school youth (Allen, 1994). No theoretically sound mechanisms have been identified and successfully tested for high school youth.
Study 5
Walsh-Buhi et al. (2016) found that at posttest, intervention participants in Cohort 1 were significantly less likely than control group youth to report recent sex, risky sex, and sexual intentions, and Cohort 2 program youth had significantly reduced risky sexual intentions; however, there were no significant impacts of the program across both cohorts, overall. When examining results for Cohort 1 only, compared to control, treatment reduced the number of youth ever having sex at posttest (Daley et al., 2015); in addition, treatment reduced the rates of ever being pregnant or causing a pregnancy at posttest (Daley et al., 2015) and 10-12 months after the posttest (Daley et al., 2019).
Study 6
Robinson et al. (2016) reported no significant effects.
Study 7
Bull et al. (2016) found no significant effects overall, but for Hispanic participants only, the intervention condition reported significantly fewer pregnancies than those in the control condition.
Two studies assessed the effect of the Teen Outreach Program® on the three different problem behaviors (Allen et al, 1997; Allen and Philliber, 2001):
Studies with No Effects
Only one study investigated possible mediating effects. According to Allen et al. (1994), program effectiveness for middle schoolers was mediated by the degree to which youth viewed that the program enhanced their autonomy and relatedness among peers and adults.
Allen & Philliber (2001) and Allen et al. (1997) found small-medium to medium effect sizes. The odds ratios were .42-.53 for causing pregnancy or becoming pregnant, .39-.60 for failing a course, and .41-.52 for being suspended. In Walsh-Buhi et al., (2016), odds ratios ranged from .40-.71, indicating some small to medium program effects on risky sexual behaviors for one cohort receiving the program, though there were no impacts overall. Robinson et al. (2016) reported no significant effects. Bull et al. (2016) did not report an effect size for the significant effect found in the study. Allen & Philliber (2001) and Allen et al. (1997) found small-medium to medium effect sizes. The odds ratios were .42-.53 for causing pregnancy or becoming pregnant, .39-.60 for failing a course, and .41-.52 for being suspended. In Walsh-Buhi et al., (2016), odds ratios ranged from .40-.71, indicating some small to medium program effects on risky sexual behaviors for one cohort receiving the program, though there were no impacts overall. Robinson et al. (2016) reported no significant effects. Bull et al. (2016) did not report an effect size for the significant effect found in the study.
These studies are not generalizable to the nation or to non-studied Teen Outreach programs. This limitation is due to participant self-selection into the program (except for Allen et al., 1997 and Walsh-Buhi et al., 2016), lack of information on site selection and/or the impact of site selection, and because the demographic distributions of the students do not reflect distributions at other sites or across the nation.
Otherwise, the program reduced course failure more among girls than boys and among minorities than non-minorities.
Study 1: Allen and Philliber (2001)
Study 2: Allen et al. (2997)
Study 3: Allen et al., (1990)
Study 4: Allen et al., (1994)
Study 5: Walsh-Buhi et al., (2016):
Study 6: Robinson et al. (2016):
Study 7: Bull et al. (2016):
Another study of the program neither examined program outcomes nor used a control groups. A 4-week pilot program (Devine et al., 2014) was conducted that added text messages to the main program. A total of 96 teens were enrolled in the study and received on average 11 messages per week related to the program's content with references to celebrities and musicians in whom the participants indicated they were interested. More than half of these texts asked for a response from the participant and an automated message was sent acknowledging the receipt of the response. Text reminders regarding upcoming sessions were also sent out. Teens without mobile phones were set up with a buddy system to read text messages with another participant. Satisfaction surveys were collected from the participants during and at the end of the study.
Devine, S., Bull, S., Dreisbach, S., & Shlay, J. (2014). Enhancing a teen pregnancy prevention program with text messaging: Engaging minority youth to develop TOP® plus text. Journal of Adolescent Health, 54(3), S78-S83.
Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
SAMHSA : 2.2-2.3
DeVonne Bernard
Director, Teen Outreach Program
Wyman Center
1401 LaSalle Lane, Suite 220
St. Louis, MO 63104
(314) 471-3864
Devonne.Bernard@wymancenter.org
Alexandra Peralta
Program Performance Officer
Children's Services Council of Palm Beach County
2300 High Ridge Road
Boynton Beach, FL 33426
(561) 374-7612
Alexandra.Peralta@cscpbc.org
Tori Gale
Director, Partner Development
Wyman Center
600 Kiwanis Drive
Eureka, Missouri 63025
Email: Tori.Gale@wymancenter.org
Website 1: www.wymancenter.org
Website 2: https://wymancenter.org/national-network/top/
Certified Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach program. Journal of Community Psychology, 29(6), 637-655.
Certified Allen, J., Philliber, S., Herrling, S., & Kuperminc, G. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729-742.
Allen, J., Philliber, S., & Hoggson, N. (1990). School-based prevention of teen-age pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach program. American Journal of Community Psychology, 29(4), 505-523.
Allen, J., Kupermind, G., Philliber, S., & Herre, K. (1994). Programmatic prevention of adolescent problem behaviors: The role of autonomy, relatedness, and volunteer service in the Teen Outreach program. American Journal of Community Psychology, 22(5), 617-639.
Walsh-Buhi, E. R., Marhefka, S. L., Wang, W., Debate, R., Perrin, K., Singleton, A., ... & Daley, E. M. (2016). The impact of the Teen Outreach Program on sexual intentions and behaviors. Journal of Adolescent Health, 59, 283-290.
Daley, E. M., Buhi, E. R., Wang, W., Singleton, A., Debate, R., Marhefka, S., . . . Ziemba, R. (2015). Evaluation of Wyman's Teen Outreach Program in Florida: Final impact report for Florida Department of Health. Findings from the replication of an evidence-based teen pregnancy program.
Daley, E. M., Marhefka, S. L., Wang, W., Noble, C. A., Mahony, H., Arzola, S., . . . Walsh-Buhi, E. R. (2019). Longitudinal evaluation of the Teen Outreach Programme: Impacts of a health promotion programme on risky sexual behaviours. Health Education Journal. Advance online publication. doi 10.1177/0017896919857777
Robinson, W. T., Seibold-Simpson, S. M., Crean, H. F., & Spruille-White, B. (2016). Randomized trials of the Teen Outreach Program in Louisiana and Rochester, New York. American Journal of Public Health, 106(51), 539-544.
Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text messaging, Teen Outreach Program, and sexual health behavior: A cluster randomized trial. American Journal of Public Health, 106(51), S117 - S124.
Evaluation Methodology
The motivation for this evaluation was to learn whether the Teen Outreach program had greater efficacy for those students who were more at-risk for problem behaviors than other students.
Design:
Recruitment: In this mix of a quasi-experimental and randomized study, data were collected from 3,277 youth in grades 9-12 from 60 sites over the course of 4 years. The actual years of the data collection were not reported. No information is provided as to how the 60 sites were selected.
Assignment: The intervention group included 1,673 students and the comparison group included 1,604 students. About 20% of the study youth were randomly assigned to either the Teen Outreach group or the comparison group. The remaining Teen Outreach youth chose to participate in the program by requesting it after hearing about it during school announcements or agreeing to participate after having been referred by teachers or counselors. The remaining comparison group youth were selected through one of two methods: (a) program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) facilitators or guidance counselors nominated youth whom they thought were from similar backgrounds and had similar levels of problems.
Assessments/Attrition: The baseline assessment came within the first few weeks of the academic year, while the posttest came at the end of the academic year. The intervention group experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%.
Sample characteristics: Of the 3,277 sampled youth, the mean age was 16 years old, with the majority in 9th or 10th grades. The average mother's education level was slightly over a high school graduate, and slightly over half (about 55%) of the youth lived in two-parent homes. About 19% of the sample had prior academic suspensions and 32% had failed at least one course. About 8% had either caused or been pregnant in the past and about 3% were teen parents. The samples differed only slightly on gender and race profiles.
Measures:
The measures came from a self-report questionnaire that program youth completed in their Teen Outreach classroom and comparison group youth completed in study hall or some other school setting. Students were told that the information they provided would be kept strictly confidential, and that only identity codes would be included on their questionnaires. They were specifically reassured that none of their answers would be available to program facilitators or to other school officials, and that no data which in any way identified them would be reported.
The outcome measures came from questions about three behaviors over the last academic year: 1) having gotten pregnant or caused a pregnancy, 2) having failed any course, and 3) having been suspended. Each outcome took a binary form of yes or no. According to the authors, prior research suggested that student reports on these outcomes were generally accurate and that any inaccuracies were unbiased with respect to student condition.
Analysis: This study used three logistic regressions to estimate program effects. Control variables included demographics and the baseline outcomes of pregnancy, course failure, and academic suspension status at the beginning of the year. Checks for clustering within classroom using multilevel models were reported as a supplement.
Along with complete-case analyses, the study presented analyses using all cases under three forms of imputation. First, dropouts were assumed to have the same rate of problem behavior as completers. Second, dropouts were assumed to have twice the rate of problem behaviors as completers. Third, dropouts were assumed to have had four times and eight times the rate of problem behavior of completers (up to a 100% rate).
Intent to Treat: The study used all students with complete data in the main analysis and reported on results using all students after imputation of missing data.
Outcomes
Implementation Fidelity:
Although Teen Outreach sites were expected to provide a minimum of 20 hours per year of volunteer experience, participants averaged 37.4 hours, with a median of 25 hours. No other measures of implementation fidelity were reported.
Baseline Equivalence:
Tests reported in Table 1 showed two significant differences in 17 tests. The significant differences appeared small. Females represented 29.1% of the comparison group and 24.6% of the program participants, and 46.1% percent of the comparison group identified as black, compared with 44.3% percent of the program participants.
Differential attrition:
The Teen Outreach sample experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%. Attriters were more likely than non-attriters to become pregnant or cause a pregnancy and/or fail courses in school. They were also more likely to be younger, male, members of a minority group, and living in single-parent households. However, the authors stated that these differences were similar across conditions (i.e., there was no interaction of attrition and condition).
Posttest:
The authors did not report results separately for the randomized sample. However, they stated that, in allowing a variable for random assignment to interact with the effects of interest, the analysis found no statistically significant differences in program effects between the randomized and non-randomized samples.
Teen pregnancy: Self-reports showed that intervention participants had significantly fewer pregnancies than the comparison group (OR = .53). In seven interaction tests, only one emerged as significant: the intervention had a stronger effect among those who were teen parents at baseline than others.
Course failure: Self-reports showed that intervention participants had significantly fewer course failures than the comparison group (OR = .60). In seven interaction tests, three emerged as significant. The intervention had stronger benefits for females than males, for racial minorities than majorities, and for students with prior suspensions than students with no prior suspensions.
Academic suspension: Self-reports showed that intervention participants had significantly fewer suspensions than the comparison group (OR = .52). No significant interaction effects emerged in seven tests.
Multiple imputation results: In the pregnancy model, all the main effects and interactions remained significant. In the course failure model, participation and interactions remained significant when imputing problem behaviors at twice the baseline rate, but not when imputing problem behaviors at four times the baseline rate.
Multilevel results. Models with random effects for schools did little to change the results.
Long-Term:
Not examined.
Evaluation Methodology
Design:
Recruitment: From 1991 through 1995, 25 schools with Teen Outreach programs participated in this evaluation. With all known Teen Outreach programs in the country contacted, 10% (n = 25) agreed to participate. Although non-random, the sample was national in scope. Students at the sites indicated interest in the program by enrolling as part of the health curriculum, as an academic elective, or as an after-school activity. Total participation included 695 students in grades 9 through 12.
Assignment: The students were randomly assigned within schools either individually when there was program over-enrollment or, less frequently, by classroom. The intervention group (n = 342) took courses that included the program curriculum and activities, while the control group (n = 353) took the usual health or social studies courses.
Assessments/Attrition: The baseline assessment came within the first few weeks of the academic year, while the posttest came at the end of the academic year. At the early stages of data analysis, two sites had attrition rates between 80 and 90 percent and one site showed a significantly higher rate of baseline problem behaviors for the control group than the program group. These three sites were removed, and 22 sites were used in the main analysis. For the remaining 22 sites, attrition was 5.3% for the Teen Outreach program and 8.4% for the control group.
Sample characteristics: The typical student in the sample was about 16 years old, in 9th or 10th grade, female (roughly 85%), and black (about 67%). The high rate of female participation is explained by teachers and counselor's viewing the program as part of pregnancy prevention and therefore recommended more females than males.
Measures: The measures came from a self-report questionnaire that program youth completed in their Teen Outreach classroom and control youth completed in study hall or other school setting. The problem behavior questions asked students whether they had 1) been pregnant or caused a pregnancy, 2) failed any courses during the prior year, and 3) been suspended academically in the prior year. In the posttest, the problem behaviors referred to the past academic year.
A validity check used data from one of the sites that provided administrative data on course failures and school suspensions. For that site, over-reporting was common but was similar across conditions and across baseline and posttest assessments.
Analysis: Logistic regression examined the effects of the program while controlling for baseline outcomes and demographic characteristics. Although some students were randomized by classroom, the analysis did not check or adjust for clustering.
Intent to Treat: Dropping three schools may have violated the intent-to-treat criterion, but the exclusions related more to data quality than low program participation, and the authors stated that including the three schools did not change the significance of the program effects.
Outcomes
Implementation Fidelity: While required to provide 20 hours of volunteer service, program participants averaged 45.8 hours. The study reported no other measures of implementation fidelity.
Baseline Equivalence:
No statistically significant differences in demographic characteristics existed across the groups (Table 1). However, the control group exhibited significantly higher rates on all three problem behavior outcomes. The differences came in part from the three sites that were dropped from most of the analysis. Among the other 22 sites, the differences may have come from completing the baseline measures a few weeks after the program start. The potential effects of baseline differences for the remaining 22 sites were examined as part of the outcome analysis.
Differential Attrition:
Among sites, no analysis was made to determine whether the three sites that dropped out of the study had students who were significantly different than the other sites demographically or in terms of problem behaviors at entry. However, an alternative analysis was conducted with the three sites included and all effects of the program remained at the same significance levels. The method used to impute data in order to run the alternative analysis was not reported.
Among students in the remaining 22 programs, dropouts were significantly different at baseline than completers on prior pregnancy, academic suspension, age, and gender. However, comparisons across conditions found that attrition was not linked to being in the control or program group (i.e., there was no interaction of attrition with condition).
Posttest:
The Teen Outreach program had statistically significant and moderate effects on all three problem behavior outcomes. The odds ratios equaled .42 for pregnancy, .39 for course failure, and .41 for school suspension. Interaction tests showed one group difference in program effects, with females benefitting more from the program than males. Further tests including the three sites dropped from the main analysis found that the program effects remained at the same significance levels.
To check for bias from condition differences at baseline among the 22 analysis sites, the study first created a dichotomous variable that indicated whether the site had a higher number of problem behaviors at entry in the control group than the intervention group. It next estimated models that allowed the program to interact with the new variable. None of these interactions was significant, leading the authors to conclude that baseline non-equivalence did not explain the observed program effects. Lastly, they dropped the four schools with the greatest imbalance, finding that the significance levels of the program effects were unchanged.
Long-Term:
Not examined.
Evaluation Methodology
Design: This evaluation was designed to assess both individual and program-level factors that may influence the effectiveness of the Teen Outreach program in reducing problem behaviors in youth. Data were collected at the site- and the individual-level during the 1986-1987 academic/program year. Of the 70 Teen Outreach sites in the nation, the study sampled 35 sites in 30 schools. The authors did not indicate how the evaluation sites were selected, but reported that the sample was geographically disperse and the sampled sites did not differ significantly from the non-sampled site. The authors did not indicate which site characteristics were used to draw this conclusion.
Program participants could have entered the program by electing to participate in response to school-wide announcements or by agreeing to participate after they had been targeted by teachers or counselors. After the program youth were chosen, the comparison youth were quickly selected through one of two methods: (a) the program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) teachers and counselors who had made referrals to Teen Outreach identified other youth whom they thought had similar sociodemographic backgrounds and experienced similar problems as the youth chosen for the program.
Participants in the study included 632 Teen Outreach participants and 855 comparison students in grades 7-12. Attrition over the course of the program year was 2.4% among Teen Outreach students and 4.8% among comparison students. An additional 6.5% of the Teen Outreach students did not complete exit questionnaires and were excluded from the analysis. Consistent with the intent-to-treat principle, the analysis used all subjects with available data.
Sample characteristics: Students ranged in age from 11 to 19. The typical member of the sample was 16 years old and in 10th grade. Almost 70% of the sample was female, about half were white, and about a third were black. The mean parents' education for the sample was slightly more than a high-school graduate and slightly more than half of the youth lived in two-parent households.
Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).
In the entry questionnaire, students were asked a set of demographic questions (age, grade in school, race/ethnicity, mother's and father's education levels, and whether they lived in a two-parent household). Next, students were asked a set of questions about three specific problem behaviors (whether they had been suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant).
In the exit questionnaire, they were asked the same questions regarding problem behaviors, but the time reference period was the academic year just ending. In addition to the three primary problem behaviors, they were also asked if they intended not to return to school in the fall.
The researchers chose to sum these behaviors into an entry problem behavior syndrome score (ranging from 0 to 3 problem behaviors) and an exit problem behavior syndrome score (ranging from 0 to 4 problem behaviors). By doing so, they addressed the difficulty of having low base rates on individual problem behaviors and also were consistent with the understanding at the time that a "syndrome" was more meaningful than individual inter-correlated behaviors. The mean entry problem behavior syndrome scores were .80 for the Teen Outreach group and .66 for the comparison group (p<.05).
Site-level data were collected from facilitators at each site across the following dimensions:
Analysis: The data used for this study were multi-level in nature, but the analysis was not. Individual students were nested within programs and each was assigned the average program characteristics from that site. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify the role of site-level factors in the efficacy of the program. As such, the regression models were run with individual-level characteristics as controls, with site-level intensity measures as explanatory factors, with site-level structural measures as explanatory factors, and with site-level curriculum used as explanatory factors.
Outcomes
Baseline Equivalence: Compared to the Teen Outreach participants, the comparison group youth had fathers with a slightly higher level of education, were more likely to live in a two-parent household, were less likely to have failed courses in the prior year, and had fewer problem behaviors on average.
Differential Attrition: Without reporting figures, the authors report "no significant effects of loss-to-follow-up on demographic or problem behavior measures, nor were there significant interactions of loss-to-follow-up with membership in the program versus comparison group for any of these measures."
Posttest: At the beginning of the program, the Teen Outreach participants had more problem behaviors, on average, than the comparison group (p<.001). At program exit, the Teen Outreach participants had fewer problem behaviors, on average, than the comparison group (p<.05). Thus, compared to non-participants, participating in Teen Outreach was associated with a reduced number of problem behaviors.
The average number of volunteer hours at a site had a significant and positive but weak effect on reducing the number of problem behaviors (beta=-.09, p<.05). The number of classroom hours also had a positive effect in reducing the number of problem behaviors, but the effect was stronger for the younger participants (beta for grade x class hours = .10, p<.05). Whether a program was offered during the school day for credit or after-school as an activity had no significant impact in the effectiveness of the program in reducing problem behaviors. Also, the effectiveness of the program was not significantly altered by the amount of the overall curriculum used or by different usage of the categories of the curriculum.
Limitations
The major limitations of this study include:
Evaluation Methodology
Design: This quasi-experimental study used individual and site-level data from 66 different Teen Outreach programs across the country. The researchers did not indicate how the 66 sites were selected or from which program year(s) the data were drawn.
Participants enrolled in Teen Outreach by a variety of ways: it could have been part of the required curriculum, they could have chosen it as an elective or taken it as an after-school activity; or they could have agreed to take the program in response to a recommendation from a teacher or guidance counselor. Previous research has shown that the method of entry into the program is unrelated to outcomes. After choosing the program youth, the investigators quickly selected comparison youth through one of three methods:
Within the 66 sites, data were collected from 1,020 Teen Outreach youth and 1,013 comparison group youth in grades 7 - 12. Attrition was 2.4% among Teen Outreach students and 2.7% among comparison students.
Sample characteristics: Students ranged in age from 11 to 19 years old. The typical member of the sample was 16 years old and in the 10th grade. About 70% of the sampled students were female, and roughly 40% were black and 40% were white. The mean parents' education level was just above a high school graduate and slightly more than half of the youth came from two-parent households.
Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June). Students were asked their age, their grade in school, their race/ethnicity, their parents' education level (1=not a high school graduate through 4=college graduate), whether they lived in a two-parent household, whether they had been academically suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant. In the exit questionnaire, the reference time period was the academic year just ending.
To measure the degree to which programs fostered senses of autonomy and relatedness, a questionnaire was administered to students and to facilitators during the final month of the program. This questionnaire was intended to measure (a) the extent to which the program was viewed as promoting adolescent autonomy and relatedness; and (b) whether the quality of the volunteer experience was likely to enhance the students' sense of competence and autonomy.
The questionnaire was structured similar to the Perceived Competence Scale for Children, which is intended to reduce the effects of a pull for social desirability. For each item, two contrasting stems were presented side by side (e.g., "some kids feel like their facilitators like them a lot" and "other kids feel like their facilitator just likes them OK."). The youth were asked to pick one of the two statements that they most agreed with and then decide whether it was "really true" or "sort of true." The autonomy and relatedness questions focused on whether the youth felt like they could make their own decisions within the program, whether their views were respected, etc. The quality of volunteer experience questions focused on whether the youth felt like they were learning new skills, whether they might like to continue volunteering after the program ended, etc. The two scales had adequate internal consistency (alpha=.87 and alpha = .72, respectively).
The questionnaire was also given to facilitators who were asked to answer as they thought their class would answer. The adolescent autonomy and relatedness scale had an alpha of .81 and the quality of volunteer experience scale had an alpha of .52. The authors report that the "validity of this inventory approach has been established in other studies, in spite of its reduction of alpha levels".
Students were given the standard problem behavior questionnaire at the beginning of the program and at the end. Both program youth and facilitators were given the autonomy and relatedness questionnaire and the quality of volunteer experience questionnaire at some point in the last month of the program. Students were assured of confidentiality and that their data would not be shared with school officials.
Analysis: The data used for this study were multi-level in nature. Individual students were nested within programs and each was assigned the average program values for program level measures. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify whether feelings of autonomy and connectedness and quality volunteer experiences mediated the program's effectiveness. Also, to control for potential unmeasured site-wide factors, the authors included residualized problem behavior change scores for the comparison group for each site.
Linear regression models were run with the number of problem behaviors at exit as the dependent variable. The covariates were entered sequentially, as follows: individual-level data (number of problem behaviors at entry, grade level, residualized problem behavior change score); then the primary variable of interest (promotion of autonomy and relatedness scale in regression #1 and quality of volunteer experience in regression #2); and, finally, interactions of grade level with the primary variable of interest. Each model was run twice, once using student ratings and once using facilitator ratings.
Outcomes
Baseline Equivalency: The Teen Outreach participants and the comparison group were very similar with respect to demographic characteristics and program behaviors at entry. The only statistically significant difference was that the proportion of females was slightly higher among the Teen Outreach group. This difference can be attributed to adult referrals into a program that they view as primarily a pregnancy prevention program.
Differential Attrition: Attrition was similarly low in both the treatment and comparison groups.
Posttest: At program entry, there was no significant difference in the mean number of problem behaviors (ranging from 0 to 3) between the Teen Outreach youth and the comparison youth (.61 vs.64, respectively). After controlling for problem behaviors at program entry, the mean number of problem behaviors was significantly lower (p<.001) for youth in the Teen Outreach program (.48 vs.67, respectively).
When testing for site-level effects, the researchers used averaged site-level data from both the students and facilitators. In the student rating model, programs with higher scores on the promotion of autonomy and relatedness scales were significantly more effective in reducing problem behaviors (p<.01). However, this effect was only true for middle school students (p<.05). Likewise, in the facilitator model, the only significant effect of promotion of autonomy and relatedness was for middle school students (p<.01).
The regressions were run again with the quality of volunteer experience as the key explanatory variable. In the student rating model, programs with higher scores on the quality of volunteer experience scale were significantly more effective in reducing problem behaviors ( p<.05), but once again this effect only held for middle school students. In the facilitator rating model, no significant effects of the quality of volunteer experience was found. Additionally, the effect of the mean number of volunteer hours on program success was not significant.
The authors conducted an analysis to determine if the small effect found for quality of volunteer experience (i.e., enhancing autonomy and relatedness) on reducing problem behaviors for middle school students was a mediating effect of the volunteer experience itself. The authors conclude that, in fact, a portion of the volunteer experience effect is mediated through the perceived autonomy and relatedness aspects of the program.
In sum, the proposed psycho-social mechanisms were significant only for middle schoolers. This was surprising because previous research has shown that the overall Teen Outreach program is more effective, in general, for high school youth. The authors concluded that middle school youth are more in need of an environment that fosters this sort of development, while high school students may already have achieved a certain level of autonomy and relatedness. High school youth are therefore less dependent on getting this specific developmental competency within the program and instead benefit from other aspects of the program.
Limitations
The major limitations of this study include:
Evaluation Methodology
Design:
Recruitment: Schools were recruited for participation if they were located in nonmetropolitan Florida counties (those with a population of less than 900,000), if they had the capacity to implement the program, and if they had higher rates of adolescent births, STIs, high school dropout, graduation, or out-of-school suspension rates compared to other communities. A total of 28 public schools within 12 counties were selected for program implementation. Two cohorts of 9th grade classes were recruited for participation in the Walsh-Buhi et al. (2016) study, with 9,220 students found eligible and 7,667 consenting to participate. Daley et al. (2019) offered a longitudinal analysis of youth enrolled in the study beginning in fall 2012 (i.e., cohort 1).
Assignment: The schools were paired on structural and demographic characteristics and randomized into the intervention or control groups. Specifically, the 28 schools reported in Walsh-Buhi et al. (2016) were matched into pairs based on county, courses offered, school size, region/proximity, and presence of block scheduling, and then randomly assigned to either the intervention (n=14 schools, 3,533 students) or control group (n=14, 4,134). In the intervention group, the intervention was delivered as a supplement to the usual health curriculum while control schools continued with health education as usual. On page 16, Daley et al. (2015), reporting only on cohort 1, stated, "Passive parental consent was obtained after random assignment for a total of 4,063 youth throughout the study (1,845 intervention and 2,218 comparison)."
Attrition: One school dropped out during the first year so the school and its matched pair were dropped for cohort 2. As reported in Walsh-Buhi et al. (2016), as stated in the text on p. 285, there were 3,988 students in cohort 1 that completed a baseline measure and 3,404 at follow-up, resulting in a retention rate of 100% at the school level and 85% at the student level. For cohort 2, 3,679 students were assessed at baseline and 2,758 completed a follow-up survey for a retention rate of 93% at the school level and 75% at the student level. Overall (with both cohorts combined), the retention rate was 80% (7,667 at baseline and 6,162 at follow-up). Daley et al. (2019) reported on only cohort 1 and showed a student-level retention rate at the one-year follow-up of 67% (n = 2,660). It is worth noting that the baseline participant numbers are reported somewhat inconsistently in Daley et al. (2019). Figure 1 reports 3,988 students with baseline data while Table 2 reports 3,836.
Sample: Across both cohorts the sample was evenly split on gender with a mean age of 14.6 at baseline. Nearly 60% of respondents were white, with 20% reporting as Hispanic, 10% as Black, and the remaining 10% as other races/ethnicities. Approximately 15% of the sample reported recent sexual activity; of those who reported recent sexual activity, 27% reported recent risky sex. About 32% of the entire sample intended to have sex within the next year, with 8% of those intending risky sex in that same time frame.
Measures: Outcomes of interest were self-reports of: ever had sex (ever sex), sexual intercourse in the previous 3 months (recent sex), not using a condom in the previous 3 months (recent risky sex), and ever been pregnant or gotten someone pregnant (ever pregnant). Secondary outcomes included intention to have sex in the next year (sex intentions), and, among those who intended to have sex in the next year, their intention to use a condom or not (risky sex intentions). Daley et al. (2015) reported on just two of these measures: ever sex and ever pregnant.
Analysis: Walsh-Buhi et al. (2016) used two-level generalized linear mixed effects models with logit link function, controlling for baseline demographics (age, gender, race/ethnicity) and pretest behaviors. School IDs were treated as the source of random effects. Daley et al. (2015) reported the use of the Bonferroni multiple comparisons correction to the ever sex question (for cohort 1). For Daley et al. (2019), three levels (i.e. students within classrooms, nested within schools in the same county) of random effects were included to account for clustering effects. The researchers also attempted to fit a model with school as a random factor, but those models had convergence problems. Repeated measures were calculated across the three time points (baseline, first follow-up, and second follow-up) for each individual, and the random variation of each observation was taken into account. All models included fixed effects for demographic variables (gender, age group, and race/ethnicity) and the treatment condition. Models also included randomization block (e.g., school pair) as a fixed effect except for the outcomes corresponding to the question "Have you ever been pregnant or gotten someone pregnant?" That outcome had low prevalence, overall, so for those models, blocks were collapsed into larger more inclusive categories based on the number of students and geographic setting of each school. Walsh-Buhi et al. (2016) ran many different subgroup analyses, including differential impacts of the program across cohorts, genders, and by baseline behaviors. Daley et al. (2019) performed a subgroup analysis by gender.
Intent-to-Treat: Multiple imputation was used to account for missing data, with all cases used in the analyses for both Walsh-Buhi et al. (2016) and Daley et al. (2019).
Outcomes
Implementation Fidelity: Fidelity was not assessed in Walsh-Buhi et al. (2016). As reported in Daley et al. (2015, 2019), for cohort 1, the intervention was implemented in 70 individual classes. Of the 70 classes, 51 classes (73%) offered at least 25 weekly sessions, as prescribed (range: 23-57 sessions). Relatively few classes-8 (or 11%)-received at least 20 community service project hours, as prescribed. No classes received programming over a span of 9 months.
Baseline Equivalence: As reported by Walsh-Buhi et al. (2016) in Tables 2 and 3 (p. 287), there were no significant differences between groups at baseline for the combined cohorts. When examined by cohort, out of 3 socio-demographic and 10 pretest variables tested, there was one baseline difference for cohort 1 (race/ethnicity). For Daley et al. (2019), out of the 3 socio-demographic (Table 2, p. 8) and 6 pre-test measures (Table 3, p. 9), no significant baseline differences were detected. As for the analysis sample, Walsh-Buhi et al. (2016) and Daley et al. (2019) state that they tested for condition equivalence of those retained at posttest; neither presented the results of the attrition analyses. Daley et al. (2015), however, reported no demographic or pretest differences between conditions for the analysis sample with one exception, youths in the control group were more likely to answer "yes" to the ever had sex question at both the posttest and follow-up (Tables III3 and III4).
Differential Attrition: As mentioned previously, equivalence for the analysis sample was not presented in either Walsh-Buhi et al. (2016) or Daley et al. (2019), but some differences (in favor of the treatment group) were reported in Daley et al. (2015). Daley et al. (2015) reported that at both the posttest and follow-up, completers (i.e., youth with complete follow-up information who also answered each question at baseline) were significantly more likely to be younger, female, and more likely to report being White or 'other' rather than Black compared to attritors (i.e., those missing at the follow-up). Completers were also less likely to answer 'yes' to both the "ever had sex" pretest question and the "ever been pregnant or got someone pregnant" question at baseline. Differences between completers and attritors were not reported in Walsh-Buhi et al. (2016) or Daley et al. (2019).
Posttest: Overall, at posttest, there were no significant differences reported in Walsh-Buhi et al. (2016) between the treatment and control groups on any measures using the pooled cohort 1 and cohort 2 data. However, impacts were much more positive in cohort 1. Walsh-Buhi et al. (2016) found that treatment participants in the first cohort were significantly less likely than control group youth to report recent sex (OR = 0.70), risky sex (OR = 0.71), and sexual intentions (OR = 0.81). Daley et al. (2015) found for cohort 1, compared to control, exposure to treatment significantly reduced the number of youth ever having sex by 3.7% and significantly reduced the rates of ever being pregnant or causing a pregnancy by 1.14%. Reporting on subgroup effects, Walsh-Buhi et al. (2016) found that females showed lower odds of recent and risky sex, while males showed decreased odds of recent sex. Of those inexperienced at baseline, intervention group participants were significantly less likely to report both ever having had sex and recent sex; females were significantly less likely to report ever having been pregnant, ever having sex, and having recent sex. Of those in cohort 1 considered experienced at baseline, intervention participants were significantly less likely to report risky sex; females were significantly less likely to report recent sex and risky sex. There was an iatrogenic effect of the program for cohort 1 males who were sexually inexperienced at baseline, but sexually active at follow-up, as they actually displayed greater odds of risky sex than their control group counterparts.
For Walsh-Buhi et al. (2016), the intervention participants in cohort 2 reported significantly reduced intentions to have risky sex compared to the control group. However, of those considered experienced at baseline, intervention participants had significantly higher odds of recent sex than those in the control group - an iatrogenic effect.
Long-Term: Daley et al. (2019) found that, for cohort 1, compared to the control group, ten to 12 months after the intervention ended, the treatment group was less likely to report ever having been pregnant or having got someone pregnant (OR = 0.55). There was no long-term impact, however, on having sex (Daley et al., 2015, 2019). When examining treatment effects by gender, statistically significant differences were found only for female participants. No significant difference between treatment and control groups were found at the follow-up for the outcomes sex in the past 3 months, sex without a condom in the past 3 months, intention to have sex, and intention to not use a condom. Researchers did find an effect of the treatment on ever having sex among males at second follow-up (Daley et al., 2019), but not at first follow-up (Walsh-Buhi et al., 2016).
Evaluation Methodology
Design:
Recruitment: The researchers recruited 4769 youth in Louisiana and 966 youth in Rochester via convenience sampling methods. Recruitment took place in school and community settings during the academic years of 2013-2015, and the sample contained three "annual cohorts." Eligible participants included those who lived in the targets areas, spoke English well enough to complete the study materials, and had not previously participated. In Louisiana youth aged 12 to 17 years were included and in Rochester youth aged 11 to 14 years were included.
Assignment: Individuals were randomly assigned to the conditions in Louisiana and 33 clusters were randomly assigned to the conditions in Rochester. In Louisiana the control condition received no programming, and in Rochester the control condition received a work-readiness intervention.
Attrition: Of the 4769 youth in Louisiana, 2428 (51%) completed the follow-up survey, and of the 966 youth in Rochester, 824 (85%) completed the follow-up survey.
Sample:
The sample targeted areas with high rates of adolescent pregnancy and included multiple regions across Louisiana and Rochester. In Louisiana, the sample was 90% black and 60% female, with an average age of 13.9 years (SD = 1.59). In Rochester, the sample was 64% black and 52% female with a mean age of 12.3 years (SD = 1.11).
Measures:
Participants completed self-report outcome measures. Sexual activity was measured through a yes/no question about whether the participant had previously engaged in intercourse; the posttest initiation measure included only those who had not already had intercourse at baseline. A yes/no question asked about whether the participant had engaged in intercourse without birth control in the previous 3 months.
Analysis:
Logistic regression models were used to analyze predictors of the binary outcomes, with gender, age, race, ethnicity, and cohort year used as covariates. The baseline outcome was included in models of the use of birth controls but not for initiation of sexual intercourse (all analyzed subjects had not previously engaged in sexual intercourse). The analysis was done at the individual level with multivariable logistic regression in Louisiana, and at the clustered level with multilevel modeling in Rochester.
Intent-to-Treat: The researchers stated that they used intent-to-treat analyses by including all individuals who completed the posttest. However, they also stated that "During the course of the study, 3 study sites were dropped from the program."
Outcomes
Implementation Fidelity:
The researchers reported that "process measures" were recorded after each session by facilitators to assess fidelity. Approximately 10% of sessions were observed and evaluated as well and high ratings were given by evaluators. However, only 21% of youth in Louisiana and 48% of youth in Rochester in the intervention condition attended at least 75% of the sessions.
Baseline Equivalence:
The researchers tested for baseline equivalence between the intervention and control conditions on demographic variables and outcome measures and found no significant differences in Louisiana. In Rochester, the intervention group included significantly more Hispanics. However, the tests used the analysis sample of those with posttest data rather than the randomized sample.
Differential Attrition:
Differential attrition was not analyzed directly. Tests for baseline equivalence for the analysis sample are relevant but lacked comparison to the randomized sample, included only 5-6 baseline measures, and showed one difference.
Posttest:
No significant effects on outcome variables were found.
Long-Term:
Long-term effects were not tested.
Evaluation Methodology
Design:
Recruitment: Participants were recruited from eight Boys & Girls clubs in Denver over 4 years. Eligible participants included active club members between 14 and 18 years of age. A total of 3643 youth fit all eligibility criteria, and of these, 852 consented to participate in the study.
Assignment: Randomization was clustered by site and by year, creating 32 clusters in total. Each site was assigned to the intervention condition for two of the years and to the control condition for the other two years. The intervention sites received TOP programming plus the text-message-based supplement, and control sites received just the TOP programming. The sample included 436 in the intervention condition and 315 in the control condition at baseline.
Attrition: Participants completed self-report measures at baseline and immediately at program completion. At posttest, 72.7% of the intervention group and 75.7% of the control group completed the assessment.
Sample:
The clubs primarily served minority youth and those living in poverty. The mean age was 14.90 (SD = 1.02) in the intervention condition and 14.98 (SD = 1.14) in the control condition. The intervention condition was 42.41% Hispanic, 20.25% black, and 9.18% white, and the control condition was 43.81% Hispanic, 18.73% black, and 7.94% white.
Measures:
Measures were administered through self-reports. Participants reported how many times they had had sex in the previous three months and how many of these times were without protection. They also reported whether they had "accessed contraceptive or sexually transmitted infection services in the past 9 months" and whether they had gotten pregnant. The study reported no information on the validity and reliability of the measures.
Analysis:
Multilevel regression models were used to analyze the outcome variables, with a large number of covariates including the baseline outcomes. Sub-analyses compared Hispanic to non-Hispanic participants. Bonferroni-Hochner corrections were used to correct for multiple tests.
Intent-to-Treat: The authors stated that subjects were used in the originally assigned conditions, regardless of the level of engagement with the program. Also, sensitivity analysis included missing data with full information maximum likelihood estimation and gave the same results.
Outcomes
Implementation Fidelity:
No information on implementation fidelity was given, but the subjects attended an average of 11.9 sessions of the 25.
Baseline Equivalence:
Table 2 showed only 1 significant difference out of 22 tests, but the table used the analysis sample. The significant difference was for pregnancy.
Differential Attrition:
Differential attrition was tested by condition and by 22 baseline variables. Those remaining in the study reported higher rates of protected sex and living with both parents and lower likelihood of having sex. Additionally, intervention participants lost to follow-up were significantly older than those who remained in the study. However, tests for the interaction of the baseline variables and condition showed that the differential attrition was similar across conditions. The tests for baseline equivalence using the analysis sample also suggest a limited influence of attrition.
Posttest:
The program showed no significant effects for the full sample. Hispanic participants in the intervention condition reported significantly fewer pregnancies than those in the control condition, but the authors noted that this effect may be influenced by the significant baseline difference between conditions for pregnancy.
Long-Term:
No long-term effects were reported.