Communities That Care
A prevention system designed to reduce levels of adolescent delinquency and substance use through the selection and use of effective preventative interventions tailored to a community's specific profile of risk and protection.
Program Outcomes
- Alcohol
- Antisocial-aggressive Behavior
- Delinquency and Criminal Behavior
- Tobacco
- Violence
Program Type
- Community - Other Approaches
Program Setting
- Community
Continuum of Intervention
- Universal Prevention
Age
- Early Childhood (3-4) - Preschool
- Infant (0-2)
- Early Adulthood (19-24)
- Late Adolescence (15-18) - High School
- Early Adolescence (12-14) - Middle School
- Late Childhood (5-11) - K/Elementary
Gender
- Both
Race/Ethnicity
- All
Endorsements
Blueprints: Promising
SAMHSA : 3.2 - 3.6
Program Information Contact
Blair Brooke-Weiss
Social Development Research Group
University of Washington School of Social Work
9725 3rd Ave. NE, Suite 401
Seattle, WA 98115-2024
(206) 543-5709
email: bbrooke@myuw.net
www.communitiesthatcare.net
Program Developer/Owner
J. David Hawkins, Ph.D.
University of Washington School of Social Work
Brief Description of the Program
Communities That Care (CTC) is a prevention system, grounded in science that gives communities the tools to address their adolescent health and behavior problems through a focus on empirically identified risk and protective factors. CTC provides a structure for engaging community stakeholders, a process for establishing a shared community vision, tools for assessing levels of risk and protection in communities, and processes for prioritizing risk and protective factors and setting specific, measurable, community goals. CTC guides the coalition to create a strategic community prevention plan designed to address the community's profile of risk and protection with tested, effective programs and to implement the chosen programs with fidelity. CTC instructs the coalition to monitor program implementation and to periodically reevaluate community levels of risk and protection and outcomes, and to make adjustments in prevention programming if indicated by the data. Implementation of CTC is organized into five stages, each with its own series of "benchmarks" and "milestones" to help guide and monitor implementation progress. CTC is installed in communities through a series of six training events delivered over the course of 6 to 12 months by certified CTC trainers.Implementation of CTC is organized into five stages, each with its own series of "benchmarks" and "milestones" to help guide and monitor implementation progress. Technical assistance is provided to local CTC coordinators and prevention coalition members to help ensure completion of these steps and procedures, identify any barriers to successful implementation, and discuss strategies for overcoming obstacles. Assistance is delivered via weekly phone calls and emails and twice-yearly site visits.
Phase 1 is a Community Readiness Assessment phase. Here, attitudinal and organizational characteristics of community members, leaders, and organizations thought to influence the mobilization process are assessed. Important individuals and organizations necessary to initiate CTC are identified.
Phase 2 introduces the community to CTC through a training event that orients key community leaders to prevention science and the community activation processes of CTC. The training defines roles and responsibilities of the key leaders and those of the community prevention board. Key leaders are expected to hold the community prevention board and staff accountable for planning and carrying out CTC and to identify and secure resources necessary to implement preventive interventions planned through the CTC process. Key leaders then identify and invite community members who will make up the community prevention board, or, alternatively, identify an existing coalition in the community to take on the CTC prevention board functions. Prevention board members attend a 2-day orientation training.
In Phase 3, the CTC board completes assessments of levels of youth problem behaviors and risk and protective factors, as well as assessments of existing community resources. Board members participate in a 2-day training on how to evaluate the collected data in terms of community risk and protective factors, and the CTC board prioritizes two to five risk factors to target for preventive action. The profiles of risk and protection provide baseline data for later assessments of the community's progress in changing levels and trends in the factors targeted by the board's prevention plan. Following the prioritization of risk and protective factors, CTC board members attend a 1-day resource assessment training with the goal of identifying gaps in existing policies, programs, and services that address the community's prioritized factors.
In Phase 4, the CTC board develops its community action plan. Community board members attend a 2-day Community Plan Training that reviews tested, evidence-based policies, programs, and actions that have demonstrated effectiveness. The board defines measurable objectives with respect to reducing prioritized risk factors, enhancing protective factors, and reducing substance use and delinquency, and develops a plan to fill gaps in existing services through the implementation of tested, effective policies and programs. The CTC typically encompasses preventive actions from the prenatal period through young adulthood. The board selects policies and programs from a menu of tested preventive interventions for elementary and middle school students. Communities' action plans describe the interventions selected and include work plans to implement those new interventions, monitor and provide feedback on implementation quality, and assess progress towards specified process and outcome goals.
In Phase 5, the chosen preventive interventions are implemented, and implementation quality is monitored by the CTC community prevention board. At the outset of Phase 5, CTC boards receive the Community Plan Implementation Training to develop the skills and plans necessary to implement and monitor their community's action plan and sustain the CTC effort. Beginning in Year 2 and continuing into Year 5, program developers conduct trainings on the selected programs and provide technical assistance to ensure high-quality implementation and monitoring of progress toward implementation and outcome goals. Monitoring of implementation is accomplished through program-specific implementation checklists completed by program providers, checklists completed by community board members and agency supervisors who observe 10% to 15% of program sessions, and participant pre- and post-tests. During Phase 5, the board also engages local media to educate community members about risk and protective factors for adolescent problem behaviors, generates public support for the new preventive interventions indicated and motivates community members to take part in the new preventive interventions.
Outcomes
Primary Evidence Base for Certification
Study 1
Through Grade 7 (Hawkins, Brown et al., 2008):
- Students in control communities were significantly more likely to initiate delinquent behavior between fifth and seventh grades than were students in CTC communities.
Through Grade 8 (Hawkins et al., 2009):
- The incidence of delinquent behavior, alcohol, cigarette, and smokeless tobacco initiation were significantly lower in CTC than in control communities between grades 5 and 8.
- In grade 8, the prevalence of alcohol and smokeless tobacco use in the last 30 days, binge drinking in the past 2 weeks, and the number of different delinquent behaviors committed in the past year in grade 8 were significantly lower in CTC communities compared to control communities.
Through Grade 10, one year after the end of technical assistance (Hawkins et al., 2012):
- The incidence of alcohol use, cigarette use, and delinquency was lower by grade 10 among students in CTC communities than in control communities.
- The prevalence of current cigarette use and past-year delinquent and violent behavior were significantly lower in CTC than in control communities in grade 10.
Through Grade 12, eight years post-baseline (Hawkins et al., 2014; Rowhani-Rahbar et al., 2023):
- Abstinence from drug use, drinking alcohol, smoking cigarettes, and engaging in delinquency were lower in the CTC communities than control communities.
- Students were less likely to ever have committed a violent act or to have carried a handgun in CTC communities, relative to control communities.
Through age 19, nine years after baseline (Oesterle et al., 2015): No main effects.
Through age 21, ten years after baseline and compared to control participants (Oesterle et al., 2018), treatment participants had:
- Higher likelihood of sustained abstinence from gateway drugs (alcohol, tobacco, and marijuana) and marijuana
- Higher likelihood of abstaining from antisocial behavior
- Reduced risk of lifetime engagement in violence
Through age 23, 12 years after baseline and compared to control participants (Kuklinski et al., 2021), treatment participants had significantly lower:
- Alcohol use,
- Illicit drug use, and
- Anti-social behavior.
Community-Level Prevention Service System Outcomes (Brown et al., 2007; Rhew et al., 2011 draft):
- CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004, relative to control communities.
- CTC communities reported higher levels of adoption of a science-based approach to prevention in 2009, 1.5 years after study-funded resources for CTC ended.
Significant Program Effects on Risk and Protective Factors:
- The levels of risk factors targeted by CTC communities were significantly lower among panel students in grade 7 in intervention communities than in control communities after 1.67 years of implementing preventive interventions selected through the CTC process (Hawkins, Brown, et al., 2008).
- Mean levels of targeted risks increased less rapidly between grades 5 and 10, and were significantly lower in grade 10, in CTC than in control communities (Hawkins et al., 2011).
- Protective factors of opportunities for prosocial involvement, recognition for prosocial involvement, and social skills were higher in grade eight in intervention communities than control communities (Kim et al., 2015).
Brief Evaluation Methodology
Primary Evidence Base for Certification
Of the six studies Blueprints has reviewed, one study (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness) and was conducted by the developer.
Study 1
The Community Youth Development Study [CYDS] consists of 20 articles (three certified by Blueprints: Hawkins, Brown et al., 2008; Hawkins et al., 2009, 2012) that examined a cluster randomized controlled trial of 24 communities across seven states. After matching, 12 communities were assigned to the intervention group and 12 to the control group. A total of 4,407 fifth-grade students in the 24 communities were surveyed nine times from 2004 to 2014, roughly from age 12 to age 23. Outcomes included initiation and prevalence of substance use and delinquent behavior.
Study 1
Hawkins, J. D., Brown, E. C., Oesterle, S., Arthur, M. W., Abbot, R. D., & Catalano, R. F. (2008). Early effects of Communities That Care on targeted risks and initiation of delinquent behavior and substance use. Journal of Adolescent Health, 43(1), 15-22.
Hawkins, J. D., Oesterle, S., Brown, E. C., Arthur, M. W., Abbot, R. D., Fagan, A. A., & Catalano, R. F. (2009). Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: A test of Communities That Care. Archives of Pediatric Adolescent Medicine, 163(9), 789-798.
Hawkins, J. D., Oesterle, S., Brown, E. C., Monahan, K. C., Abbott, R. D., Arthur, M. W., & Catalano, R. F. (2012). Sustained decreases in risk exposure and youth problem behaviors after installation of the Communities That Care prevention system in a randomized controlled trial. Archives of Pediatrics & Adolescent Medicine, 166(2), 141-148.
Risk Factors
Individual: Antisocial/aggressive behavior, Early initiation of antisocial behavior*, Early initiation of drug use*, Favorable attitudes towards antisocial behavior*, Favorable attitudes towards drug use*, Gang involvement, Physical violence, Rebelliousness, Stress, Substance use*
Peer: Interaction with antisocial peers, Peer rewards for antisocial behavior, Peer substance use
Family: Family conflict/violence, Family history of problem behavior, Parental attitudes favorable to antisocial behavior, Parental attitudes favorable to drug use, Poor family management*
School: Low school commitment and attachment, Poor academic performance
Neighborhood/Community: Community disorganization, Laws and norms favorable to drug use/crime, Low neighborhood attachment, Perceived availability of drugs, Perceived availability of handguns, Transitions and mobility
Protective Factors
Individual: Clear standards for behavior, Coping Skills, Perceived risk of drug use, Prosocial involvement, Refusal skills, Religious service attendance, Rewards for prosocial involvement, Skills for social interaction
Peer: Interaction with prosocial peers
Family: Attachment to parents, Opportunities for prosocial involvement with parents, Rewards for prosocial involvement with parents
School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school
Neighborhood/Community: Opportunities for prosocial involvement, Rewards for prosocial involvement
*
Risk/Protective Factor was significantly impacted by the program
See also: Communities That Care Logic Model (PDF)
Subgroup Analysis Details
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In Study 1 (Oesterle et al., 2018), at nine years post-baseline, only males showed significant sustained effects; however, there were no significant differences in main effects between men and women 10 years after implementation.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
The sample of students in Study 1 (Hawkins, Brown et al., 2008) was evenly split between males and females. About 70% were White or Caucasian, 9% were Native American, 4% were African American, and 20% were of Hispanic origin.
For information on CTC training description, see: CTC_Training.pdf
Training Certification Process
For information about the CTC Training of Trainers model, see: CTC_TOT.pdf
Program Benefits (per individual):
$3,240
Program Costs (per individual):
$623
Net Present Value (Benefits minus Costs, per individual):
$2,617
Measured Risk (odds of a positive Net Present Value):
86%
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
Start-Up Costs
Initial Training and Technical Assistance
Start-up Costs for One CTC Coalition: Technical assistance and 2-day Local Coalition Coordinator Training to support initial CTC coalition development, not including travel, is $6,059.
Training and Technical Assistance: $23,296. Phase Two 1/2-day Key Leaders and 2-day Community Board Orientation Training for up to 30 participants is $5,429. Phase Three 1-day Community Assessment Training and 1-day Community Resources Assessment Training is $5,933. Phase Four 2-day Community Planning Training, 1/2-day Community Plan Implementation Training 1, and 1/2-day Funding Sustainability Workshop is $7,227. Phase Five 1-day Community Plan Implementation Training 2 is $4,707. These costs include training and technical assistance required to complete CTC coalition training. Travel is not included, and varies by location.
Curriculum and Materials
Included in costs above.
Materials Available in Other Language: Participant materials only are translated into Spanish at same cost as English language version.
Licensing
Agency Licensing: N/A.
Other Start-Up Costs
Administering and analyzing the Communities That Care Youth Survey to all 6th, 8th, 10th, and 12th grade students in the community as part of the needs assessment phase of CTC. These surveys are often funded as part of a statewide survey, in which case there are no costs. CTC Youth Surveys can be purchased from Bach-Harrison, LCC in Salt Lake City, Pride Surveys, and other providers at a cost of about $2.25 per survey (price to be negotiated with provider).
Intervention Implementation Costs
Ongoing Curriculum and Materials
No information is available
Staffing
It is recommended that at least .5 FTE staffing for a local coalition coordinator be provided to support the work of the CTC coalition.
Other Implementation Costs
Costs that will vary by locality include program implementation, administrative support, space, travel, supplies, and communications. Program implementation costs vary significantly across the country depending on programs selected by the CTC coalition.
Implementation Support and Fidelity Monitoring Costs
Ongoing Training and Technical Assistance
Ongoing training and technical assistance can be obtained from certified CTC trainers at a cost to be negotiated with the trainer on a case-by-case basis.
Fidelity Monitoring and Evaluation
Can be obtained from program providers. Cost is variable. Fidelity monitoring of the CTC system is imbedded within the system and carried out by the coalition at no cost.
Ongoing License Fees
No information is available
Other Implementation Support and Fidelity Monitoring Costs
No information is available
Other Cost Considerations
No information is available
Year One Cost Example
In this example, one community is implementing the CTC process with a CTC coalition. The costs cover the CTC process and do not include programming. First year costs would include:
Development technical assistance | $6,059.00 |
Initial orientation trainings for up to 30 participants | $5,429.00 |
Community assessment trainings and TA | $5,933.00 |
Community planning trainings and TA | $11,934.00 |
Travel costs for trainers (8 days @ $250/day) | $2,000.00 |
Salary for one coordinator @ $60,000 | $60,000.00 |
Fringe @ 25% | $15,000.00 |
CTC Youth Survey (800 students @ $2.50/student) | $2,000.00 |
Overhead @ 25% of staff cost | $19,500.00 |
Total One Year Cost | $127,855.00 |
Depending upon what the community decides as its priorities, a community of 20,000 - 25,000 might serve approximately 800 students and their families at a unit cost of $159.82.
Funding Overview
The costs of the CTC model described above are costs associated with organizing and facilitating a community prevention coalition, and gathering, analyzing and using assessment data. They do not include the costs of implementing selected programs to address risk and protective factors. The funding streams typically used to support CTC are federal, state, and local funding streams dedicated to prevention initiatives that have a portion of dollars available for technical assistance, training and research.
Allocating State or Local General Funds
Some states have allocated general funds to establish dedicated funding for prevention programs or put in place changes to budget structures, such as legislative set- asides requiring a certain portion of state agency budgets to be dedicated to evidence-based programs and/or prevention programs. States that are administering prevention funds to local communities may be willing to invest in CTC training to support those communities in making effective use of prevention dollars. In addition, many states have invested some portion of their tobacco settlement funds in substance abuse prevention programs.
Local school districts and schools, public health departments, and cities can provide critical in-kind resources to support administration of the assessment, analysis of the data, and organizing community coalitions.
Maximizing Federal Funds
Formula Funds:
- The Community Development Block Grant (CDBG) program is administered from the federal Department of Housing and Urban Development to localities to support community economic development. Fifteen percent of these funds can be used to support a wide range of public services. Cities fund a variety of prevention and social service programs with these dollars. It is a funding source well-aligned with CTC due to its emphasis on community engagement and development.
- The Title V Maternal and Child Health Block Grant funds public health activities aimed at supporting healthy pregnancy and early childhood. Title V funds typically support state and local public health activities, with an emphasis on supporting the infrastructure needed to support good health outcomes for pregnant women, infants, and children. Partnership with public health offices could lead to support for administration of assessments or coalition-building activities in CTC.
- OJJDP Formula Grant Funds support a variety of improvements to delinquency prevention programs and juvenile justice programs in states. Evidence-based programs are an explicit priority for these funds, which are typically administered on a competitive basis from the state administering agency to community-based programs. Accessing these funds to support CTC activities would require outreach to the state administering agency.
- The Substance Abuse Prevention and Treatment and Community Mental Health Services Block Grants (SABG & MHBG), both of which are administered by SAMHSA, can fund a variety of substance abuse and mental health prevention and intervention activities. Since the passage of the Affordable Care Act, SAMHSA is placing increasing emphasis on supporting prevention efforts with these funds and coordinating activities supported by the two block grants. State administering agencies are required to invest 20 percent of their block grants in primary substance abuse prevention activities. State administering agencies could invest in CTC training and assessment tools to help to inform the development of community level priorities for prevention.
Discretionary Grants: There are relevant federal discretionary grants administered by federal agencies including the Substance Abuse and Mental Health Services Administration (SAMHSA), the Health Resources Service Administration (HRSA), the Office of Juvenile Justice and Delinquency Prevention (OJJDP), and the Centers for Disease Control and Prevention (CDC).
Foundation Grants and Public-Private Partnerships
Foundation grants can be an important potential source of support for CTC. Community foundations, United Way, and other local funders may be interested in supporting administration and analysis of assessments to help inform their own grant making. They also may appreciate the potential for CTC to influence the investment of public dollars.
Data Sources
All information comes from the Social Development Research Group, University of Washington, and Kuklinski, M.R., Briney, J.S., Hawkins, J.D., and Catalano, R.F. (2012). Cost-benefit analysis of Communities That Care outcomes at eighth grade. Prevention Science. 13(2), 150-161.
Program Developer/Owner
J. David Hawkins, Ph.D.University of Washington School of Social WorkSocial Development Research Group9725 3rd Ave. NE, Suite 401Seattle, WA 98115jdh@u.washington.edu
Program Outcomes
- Alcohol
- Antisocial-aggressive Behavior
- Delinquency and Criminal Behavior
- Tobacco
- Violence
Program Specifics
Program Type
- Community - Other Approaches
Program Setting
- Community
Continuum of Intervention
- Universal Prevention
Program Goals
A prevention system designed to reduce levels of adolescent delinquency and substance use through the selection and use of effective preventative interventions tailored to a community's specific profile of risk and protection.
Population Demographics
Communities That Care is designed broadly to fit the age, gender, race, and ethnic make-up of the communities adopting the prevention system.
Target Population
Age
- Early Childhood (3-4) - Preschool
- Infant (0-2)
- Early Adulthood (19-24)
- Late Adolescence (15-18) - High School
- Early Adolescence (12-14) - Middle School
- Late Childhood (5-11) - K/Elementary
Gender
- Both
Race/Ethnicity
- All
Subgroup Analysis Details
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In Study 1 (Oesterle et al., 2018), at nine years post-baseline, only males showed significant sustained effects; however, there were no significant differences in main effects between men and women 10 years after implementation.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
The sample of students in Study 1 (Hawkins, Brown et al., 2008) was evenly split between males and females. About 70% were White or Caucasian, 9% were Native American, 4% were African American, and 20% were of Hispanic origin.
Other Risk and Protective Factors
Because assessment of risk and protective factors (R&P) in a participating community is conducted, and interventions are chosen that focus on the R&P factors that are of highest priority in a community, CTC has a focus on all R&P factors.
Risk/Protective Factor Domain
- Individual
- School
- Peer
- Family
- Neighborhood/Community
Risk/Protective Factors
Risk Factors
Individual: Antisocial/aggressive behavior, Early initiation of antisocial behavior*, Early initiation of drug use*, Favorable attitudes towards antisocial behavior*, Favorable attitudes towards drug use*, Gang involvement, Physical violence, Rebelliousness, Stress, Substance use*
Peer: Interaction with antisocial peers, Peer rewards for antisocial behavior, Peer substance use
Family: Family conflict/violence, Family history of problem behavior, Parental attitudes favorable to antisocial behavior, Parental attitudes favorable to drug use, Poor family management*
School: Low school commitment and attachment, Poor academic performance
Neighborhood/Community: Community disorganization, Laws and norms favorable to drug use/crime, Low neighborhood attachment, Perceived availability of drugs, Perceived availability of handguns, Transitions and mobility
Protective Factors
Individual: Clear standards for behavior, Coping Skills, Perceived risk of drug use, Prosocial involvement, Refusal skills, Religious service attendance, Rewards for prosocial involvement, Skills for social interaction
Peer: Interaction with prosocial peers
Family: Attachment to parents, Opportunities for prosocial involvement with parents, Rewards for prosocial involvement with parents
School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school
Neighborhood/Community: Opportunities for prosocial involvement, Rewards for prosocial involvement
*Risk/Protective Factor was significantly impacted by the program
See also: Communities That Care Logic Model (PDF)
Brief Description of the Program
Communities That Care (CTC) is a prevention system, grounded in science that gives communities the tools to address their adolescent health and behavior problems through a focus on empirically identified risk and protective factors. CTC provides a structure for engaging community stakeholders, a process for establishing a shared community vision, tools for assessing levels of risk and protection in communities, and processes for prioritizing risk and protective factors and setting specific, measurable, community goals. CTC guides the coalition to create a strategic community prevention plan designed to address the community's profile of risk and protection with tested, effective programs and to implement the chosen programs with fidelity. CTC instructs the coalition to monitor program implementation and to periodically reevaluate community levels of risk and protection and outcomes, and to make adjustments in prevention programming if indicated by the data. Implementation of CTC is organized into five stages, each with its own series of "benchmarks" and "milestones" to help guide and monitor implementation progress. CTC is installed in communities through a series of six training events delivered over the course of 6 to 12 months by certified CTC trainers.Description of the Program
Communities That Care (CTC) is a prevention system that gives communities the tools to address adolescent health and behavior problems through a focus on empirically identified risk and protective factors. CTC mobilizes community leaders and a community prevention coalition (called the "community prevention board") to plan and implement a set of tested interventions to reduce elevated risk factors and promote protective factors in the community. According to CTC's theory of change, it should take from 2 to 5 years to observe community-level changes in targeted risk factors in CTC communities, and from 5 to 10 years to observe community-level changes in substance use and delinquency outcomes.Implementation of CTC is organized into five stages, each with its own series of "benchmarks" and "milestones" to help guide and monitor implementation progress. Technical assistance is provided to local CTC coordinators and prevention coalition members to help ensure completion of these steps and procedures, identify any barriers to successful implementation, and discuss strategies for overcoming obstacles. Assistance is delivered via weekly phone calls and emails and twice-yearly site visits.
Phase 1 is a Community Readiness Assessment phase. Here, attitudinal and organizational characteristics of community members, leaders, and organizations thought to influence the mobilization process are assessed. Important individuals and organizations necessary to initiate CTC are identified.
Phase 2 introduces the community to CTC through a training event that orients key community leaders to prevention science and the community activation processes of CTC. The training defines roles and responsibilities of the key leaders and those of the community prevention board. Key leaders are expected to hold the community prevention board and staff accountable for planning and carrying out CTC and to identify and secure resources necessary to implement preventive interventions planned through the CTC process. Key leaders then identify and invite community members who will make up the community prevention board, or, alternatively, identify an existing coalition in the community to take on the CTC prevention board functions. Prevention board members attend a 2-day orientation training.
In Phase 3, the CTC board completes assessments of levels of youth problem behaviors and risk and protective factors, as well as assessments of existing community resources. Board members participate in a 2-day training on how to evaluate the collected data in terms of community risk and protective factors, and the CTC board prioritizes two to five risk factors to target for preventive action. The profiles of risk and protection provide baseline data for later assessments of the community's progress in changing levels and trends in the factors targeted by the board's prevention plan. Following the prioritization of risk and protective factors, CTC board members attend a 1-day resource assessment training with the goal of identifying gaps in existing policies, programs, and services that address the community's prioritized factors.
In Phase 4, the CTC board develops its community action plan. Community board members attend a 2-day Community Plan Training that reviews tested, evidence-based policies, programs, and actions that have demonstrated effectiveness. The board defines measurable objectives with respect to reducing prioritized risk factors, enhancing protective factors, and reducing substance use and delinquency, and develops a plan to fill gaps in existing services through the implementation of tested, effective policies and programs. The CTC typically encompasses preventive actions from the prenatal period through young adulthood. The board selects policies and programs from a menu of tested preventive interventions for elementary and middle school students. Communities' action plans describe the interventions selected and include work plans to implement those new interventions, monitor and provide feedback on implementation quality, and assess progress towards specified process and outcome goals.
In Phase 5, the chosen preventive interventions are implemented, and implementation quality is monitored by the CTC community prevention board. At the outset of Phase 5, CTC boards receive the Community Plan Implementation Training to develop the skills and plans necessary to implement and monitor their community's action plan and sustain the CTC effort. Beginning in Year 2 and continuing into Year 5, program developers conduct trainings on the selected programs and provide technical assistance to ensure high-quality implementation and monitoring of progress toward implementation and outcome goals. Monitoring of implementation is accomplished through program-specific implementation checklists completed by program providers, checklists completed by community board members and agency supervisors who observe 10% to 15% of program sessions, and participant pre- and post-tests. During Phase 5, the board also engages local media to educate community members about risk and protective factors for adolescent problem behaviors, generates public support for the new preventive interventions indicated and motivates community members to take part in the new preventive interventions.
Theoretical Rationale
Communities That Care (CTC) is guided theoretically by the social development model (SDM), which posits that bonding to prosocial groups and individuals and clear standards for healthy behavior are protective factors that inhibit the development of problem behaviors. The SDM hypothesizes that bonding is created when people are provided opportunities to be involved in a social group like a coalition, family, or classroom, when they have the skills to participate in the social group, and when they are recognized for their contributions to the group.
This theoretical framework is applied in CTC in two ways. First, CTC encourages community stakeholders to adopt the SDM in their daily interactions with young people as a strategy for promoting healthy development. A goal in CTC communities is to ensure that all young people are provided developmentally appropriate opportunities, skills, and recognition, as well as healthy standards for behavior, by adults and organizations in the community. Second, the social development model guides the community mobilization and training component of CTC itself. CTC seeks to create opportunities for all interested community stakeholders to participate in developing a shared vision for positive youth development based in prevention science. Through CTC trainings, diverse community representatives develop skills to work together effectively, thus increasing the likelihood that opportunities for interaction lead to rewarding experiences. The CTC process also suggests appropriate recognition activities to enhance the reinforcement of community board members for their participation in the process.
Theoretical Orientation
- Skill Oriented
- Attachment - Bonding
Brief Evaluation Methodology
Primary Evidence Base for Certification
Of the six studies Blueprints has reviewed, one study (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness) and was conducted by the developer.
Study 1
The Community Youth Development Study [CYDS] consists of 20 articles (three certified by Blueprints: Hawkins, Brown et al., 2008; Hawkins et al., 2009, 2012) that examined a cluster randomized controlled trial of 24 communities across seven states. After matching, 12 communities were assigned to the intervention group and 12 to the control group. A total of 4,407 fifth-grade students in the 24 communities were surveyed nine times from 2004 to 2014, roughly from age 12 to age 23. Outcomes included initiation and prevalence of substance use and delinquent behavior.
Outcomes (Brief, over all studies)
Primary Evidence Base for Certification
Study 1
Hawkins, Brown et al. (2008). The levels of risk factors targeted by CTC communities were significantly lower among panel students in grade 7 in intervention communities than in control communities after 1.67 years of implementing preventive interventions selected through the CTC process. Students in control communities were significantly more likely to initiate delinquent behavior between fifth and seventh grades than students in CTC communities.
Hawkins et al. (2009). The incidence of delinquent behavior, alcohol, cigarette, and smokeless tobacco initiation were significantly lower in CTC than in control communities between grades 5 and 8. In grade 8, the prevalence of alcohol and smokeless tobacco use in the last 30 days, binge drinking in the past 2 weeks, and the number of different delinquent behaviors committed in the past year in grade 8 were significantly lower in CTC communities compared to control communities.
Hawkins et al. (2012). One year after the end of technical assistance, grade 10. Mean levels of targeted risks increased less rapidly between grades 5 and 10 in CTC than in control communities and were significantly lower in CTC than control communities in grade 10. Students in CTC communities had 38% lower odds of initiating the use of alcohol, 46% lower odds of beginning to smoke by grade 10, and 21% lower odds of initiating delinquent behavior, relative to students in control communities. Regarding prevalence measures, students in CTC communities had 21% lower odds of smoking cigarettes in the past month, 17% lower odds of reporting any delinquent behavior, and 25% lower odds of reporting any violent behavior in the past year.
Hawkins et al. (2014) and Rowhani-Rahbar et al. (2023). Eight years post-baseline, grade 12.
Students in CTC communities were more likely than students in control communities to have abstained from any drug use, drinking alcohol, smoking cigarettes, and engaging in delinquency. They were also less likely to ever have committed a violent act or carried a handgun.
Oesterle et al. (2015). Nine years post-baseline, age 19. No main effects.
Oesterle et al. (2018). Ten years post-baseline, age 21. The study found a significant treatment main effect on sustained abstinence of gateway drugs (alcohol, tobacco, and marijuana) and marijuana through age 21 among participants who had not initiated use at baseline. The study also found a higher likelihood of abstaining from antisocial behavior and a lower risk of lifetime engagement in violence through age 21 among treatment participants as compared to control participants.
Kuklinski et al. (2021). 12 years post-baseline, age 23. The young adults in the intervention group reported significantly lower alcohol use, illicit drug use, and anti-social behavior than the young adults in the control group.
Protective Factors
Opportunities for prosocial involvement, recognition for prosocial involvement, and social skills were higher in grade eight in the intervention group than the control group (Kim et al., 2015).
Prevention System Outcomes
CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004 relative to control communities (Brown et al., 2007). Furthermore, Rhew et al. (2011) found that many prevention system characteristics, such as higher levels of adoption of a science-based approach to prevention, were sustained through 2009, 1.5 years after study-funded resources for CTC ended.
Outcomes
Primary Evidence Base for Certification
Study 1
Through Grade 7 (Hawkins, Brown et al., 2008):
- Students in control communities were significantly more likely to initiate delinquent behavior between fifth and seventh grades than were students in CTC communities.
Through Grade 8 (Hawkins et al., 2009):
- The incidence of delinquent behavior, alcohol, cigarette, and smokeless tobacco initiation were significantly lower in CTC than in control communities between grades 5 and 8.
- In grade 8, the prevalence of alcohol and smokeless tobacco use in the last 30 days, binge drinking in the past 2 weeks, and the number of different delinquent behaviors committed in the past year in grade 8 were significantly lower in CTC communities compared to control communities.
Through Grade 10, one year after the end of technical assistance (Hawkins et al., 2012):
- The incidence of alcohol use, cigarette use, and delinquency was lower by grade 10 among students in CTC communities than in control communities.
- The prevalence of current cigarette use and past-year delinquent and violent behavior were significantly lower in CTC than in control communities in grade 10.
Through Grade 12, eight years post-baseline (Hawkins et al., 2014; Rowhani-Rahbar et al., 2023):
- Abstinence from drug use, drinking alcohol, smoking cigarettes, and engaging in delinquency were lower in the CTC communities than control communities.
- Students were less likely to ever have committed a violent act or to have carried a handgun in CTC communities, relative to control communities.
Through age 19, nine years after baseline (Oesterle et al., 2015): No main effects.
Through age 21, ten years after baseline and compared to control participants (Oesterle et al., 2018), treatment participants had:
- Higher likelihood of sustained abstinence from gateway drugs (alcohol, tobacco, and marijuana) and marijuana
- Higher likelihood of abstaining from antisocial behavior
- Reduced risk of lifetime engagement in violence
Through age 23, 12 years after baseline and compared to control participants (Kuklinski et al., 2021), treatment participants had significantly lower:
- Alcohol use,
- Illicit drug use, and
- Anti-social behavior.
Community-Level Prevention Service System Outcomes (Brown et al., 2007; Rhew et al., 2011 draft):
- CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004, relative to control communities.
- CTC communities reported higher levels of adoption of a science-based approach to prevention in 2009, 1.5 years after study-funded resources for CTC ended.
Significant Program Effects on Risk and Protective Factors:
- The levels of risk factors targeted by CTC communities were significantly lower among panel students in grade 7 in intervention communities than in control communities after 1.67 years of implementing preventive interventions selected through the CTC process (Hawkins, Brown, et al., 2008).
- Mean levels of targeted risks increased less rapidly between grades 5 and 10, and were significantly lower in grade 10, in CTC than in control communities (Hawkins et al., 2011).
- Protective factors of opportunities for prosocial involvement, recognition for prosocial involvement, and social skills were higher in grade eight in intervention communities than control communities (Kim et al., 2015).
Mediating Effects
As reported in Study 1 (Brown et al., 2013), higher levels of community adoption of a science-based approach to prevention in 2004 predicted significantly lower levels of youth problem behaviors in 2007. In addition, effects of the CTC intervention on youth problem behaviors were mediated fully by community adoption of a science-based approach to prevention, as reported by key community leaders.
Effect Size
In Study 1, Hawkins et al. (2011) reported odds ratios for alcohol use (OR = .62), cigarette use (OR = .54), prevalence of cigarette use (OR = .79), delinquent behavior (OR = .83), and use of violence (OR = .75). Oesterle et al. (2018) reported effects sizes for abstinence of gateway drugs (d = .26), anti-social behavior (d = .14), and incidence of violent behavior (d = -.12).
Generalizability
One study meets Blueprints standards for high quality methods with strong evidence of program impact (i.e., "certified" by Blueprints): Study 1 (Hawkins, Brown et al., 2008). The study took place in 24 communities ranging in size from 1,500 to 50,000 residents across seven states (Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington), in which the treatment group was compared to a control group.
Potential Limitations
Additional Studies (not certified by Blueprints)
Study 2 (Feinberg et al., 2007, 2010)
- The removal of 11 high-poverty CTC school districts from the analyses limited the ability to understand the effects of CTC among lower poverty communities.
- The possibility of a selection bias exists due to the quasi-experimental study design.
Feinberg, M. W., Greenberg, M. T., Osgood, D. W., Sartorius, J., & Bontempo, D. (2007). Effects of the Communities That Care model in Pennsylvania on youth risk and problem behaviors. Prevention Science, 8(4), 261-270.
Feinberg, M. W., Jones, D., Greenberg, M. T., Osgood, D. W., & Bontempo, D. (2010). Effects of the Communities That Care model in Pennsylvania on change in adolescent risk and problem behaviors. Prevention Science, 11, 163-171.
Study 3 (Rhew et al., 2016; Van Horn et al., 2014)
- Treatment and control groups matched on demographic factors, but baseline equivalence not reported
- Design unable to test for differential attrition among students
- Few positive effects of program
- Possible iatrogenic effect on antisocial behaviors among 6th graders
Rhew, I. C., Hawkins, J. D., Murray, D. M., Fagan, A. A., Oesterle, S., Abbott, R. D., & Catalano, R. F. (2016). Evaluation of community-level effects of Communities That Care on adolescent drug use and delinquency using a repeated cross-sectional design. Prevention Science, 17(2), 177-187.
Van Horn, M. L., Fagan, A. A., Hawkins, J. D., & Oesterle, S. (2014). Effects of the Communities That Care system on cross-sectional profiles of adolescent substance use and delinquency. American Journal of Preventive Medicine, 47, 188-197.
Study 4 (Chilenski et al., 2019)
- Repeated cross-sections QED with propensity weights
- Individual respondents not tracked by wave
- Possible violation of intent-to-treat
- No baseline controls
- Incomplete tests for baseline equivalence
- Incomplete tests for differential attrition
Chilenski, S. M., Frank, J., Summers, N., & Lew, D. (2019). Public health benefits 16 years after a statewide policy change: Communities that Care in Pennsylvania. Prevention Science, 20(6), 947-958.
Study 5 (Toumbourou et al., 2019)
- QED with non-random assignment and no matching
- Differences between conditions at baseline
- Individual respondents not tracked by wave, but instead used repeated measures cross-sectional aggregate data
Toumbourou, J. W., Rowland, B., Williams, J., Smith, R., & Patton, G. C. (2019). Community intervention to prevent adolescent health behavior problems: Evaluation of Communities That Care in Australia. Health Psychology, 38(6), 536-544.
Study 6 (Jonkman et al., 2015)
- QED with limited matching
- Incomplete tests for baseline equivalence
- Incomplete tests for differential attrition
- No effects on behavioral outcomes
Jonkman, H., Aussems, C., Steketee, M., Boutellier, H., & Cuijpers, P. (2015). Prevention of problem behaviours among adolescents: The impact of the Communities That Care strategy in the Netherlands (2008-2011). International Journal of Developmental Science, 9, 37-52. https://doi.org/10.3233/DEV-13121
Notes
As an upstream preventive intervention, this program targets and reduces problem behaviors that are associated with increased risk of developing substance use disorder or opioid use disorder later in life.
It should be noted that Communities That Care is a prevention system designed to reduce levels of adolescent delinquency and substance use through the selection and use of effective preventive interventions tailored to a community's specific profile of risk and protection. It is not a program or intervention in the traditional sense, but rather a delivery system for evidence-based programs.
Two additional process studies offer evidence of successful implementation.
Arthur, M. W., Ayers, C. D., Graham, K. A., & Hawkins, J. D. (2003). Mobilizing communities to reduce risk for drug abuse: A comparison of two strategies. In Z. Sloboda & W. J. Bukoski (Eds.), The handbook of drug abuse prevention: Theory, science, and practice(pp. 129-144). New York, NY: Kluwer Academic/Plenum Publishers.
In this study, two community mobilization interventions were successful at mobilizing community boards to plan and implement prevention activities, and both approaches were able to recruit and involve the types of community members they targeted on their planning boards. The Washington State CYAP was successful at involving youth in planning youth-oriented activities. The Communities That Care process used in the Oregon TOGETHER! project was effective at involving key community leaders in organizing prevention boards in their communities. However, the Oregon TOGETHER! project was more successful than the Washington State CYAP project at promoting planning and program activities aimed at specific, empirically based risk factors identified through a community risk assessment process. Even without funding, Oregon TOGETHER! prevention boards were more likely than the funded Washington CYAP community teams to collect empirical indicators of community risk and protective factors, develop action plans describing strategies to reduce prioritized risk factors, and implement programs aimed at reducing these risk factors.
Harachi, T. W., Ayers, C. D., Hawkins, J. D., Catalano, R. F., & Cushing, J. (1996). Empowering communities to prevent adolescent substance abuse: Process evaluation results from a risk- and protection-focused community mobilization effort. The Journal of Primary Prevention, 16(3), 233-254.
At the end of the four-year demonstration of Oregon TOGETHER!, using the CTC model, this study found that 31 communities remained active in the project, which Oregon has institutionalized, and 28 of them were involved in the implementation of risk reduction programs. Within a year after training, 28 boards had completed comprehensive risk-focused prevention plans. Less than a year into the planning and implementation phase, 27 had begun implementing risk reduction strategies. These findings appear to indicate that once put in place in communities, the CTC system can be maintained for several years even without significant dedicated funding.
Endorsements
Blueprints: Promising
SAMHSA : 3.2 - 3.6
Peer Implementation Sites
Heidi Peterson
Certified Prevention Specialist
Director, Communities That Care
Tooele City, Utah
Phone: (435) 843-2188
Fax: (435) 843-2189
E-Mail: heidip@tooelecity.org
Web-Address: www.tooelecity.org
Gery Shelafoe, CPC-R
NorthCare Network
200 West Spring Street
Marquette, MI 59855
906.225.7323
gshelafoe@northcarenetwork.org
Vaughnetta J. Barton, MSW
Communities in Action
Communities That Care
School of Social Work
University of Washington
Mail: Box 354900, Seattle, WA 98195-4900
4101 15th Avenue NE, Seattle, WA
206.221.8641
vjbarton@uw.edu
Program Information Contact
Blair Brooke-Weiss
Social Development Research Group
University of Washington School of Social Work
9725 3rd Ave. NE, Suite 401
Seattle, WA 98115-2024
(206) 543-5709
email: bbrooke@myuw.net
www.communitiesthatcare.net
References
Study 1
Hawkins, J. D., Catalano, R. F., Arthur, M. W., Egan, E., Brown, E. C., Abbot, R. D., & Murray, D. M. (2008). Testing Communities That Care: The rationale, design and behavioral baseline equivalence of the Community Youth Development Study. Prevention Science, 9(3), 178-190.
Brown, E. C., Hawkins, J. D., Arthur, M. W., Briney, J. S., & Abbot, R. D. (2007). Effects of Communities That Care on prevention services systems: Findings from the community youth development study at 1.5 years. Prevention Science, 8(3), 180-191.
Brown, E. C., Hawkins, J. D., Rhew, I. C., Shapiro, V. B., Abbott, R. D., Oesterle, S., . . . Catalano, R. F. (2013). Prevention system mediation of Communities That Care effects on youth outcomes. Prevention Science. doi:10.1007/s11121-013-0413-7
Fagan, A. A., Hanson, K., Briney, J. S., & Hawkins, J. D. (2012). Sustaining the utilization and high quality implementation of tested and effective prevention programs using the Communities That Care prevention system. American Journal of Community Psychology, 49, 365-377.
Fagan, A. A., Hanson, K., Hawkins, J. D., & Arthur, M. W. (2008a). Bridging science to practice: Achieving prevention program implementation fidelity in the Community Youth Development Study. American Journal of Community Psychology, 41, 235-249.
Fagan, A. A., Hanson, K., Hawkins, J. D., & Arthur, M. W. (2008b). Implementing effective community-based prevention programs in the Community Youth Development Study. Youth Violence and Juvenile Justice, 6, 256-278.
Gloppen, K. M., Arthur, M. W., Hawkins, J. D., & Shapiro, V. B. (2012). Sustainability of the Communities That Care prevention system by coalitions participating in the Community Youth Development Study. Journal of Adolescent Health, 51(3), 259-264.
Certified Hawkins, J. D., Brown, E. C., Oesterle, S., Arthur, M. W., Abbot, R. D., & Catalano, R. F. (2008). Early effects of Communities That Care on targeted risks and initiation of delinquent behavior and substance use. Journal of Adolescent Health, 43(1), 15-22.
Hawkins, J. D., Oesterle, S., Brown, E. C., Abbott, R. D., & Catalano, R. F. (2014). Youth problem behaviors eight years after implementing the Communities That Care Prevention System in a Community-Randomized Trial. JAMA Pediatr, 168(2), 122-129.
Certified Hawkins, J. D., Oesterle, S., Brown, E. C., Arthur, M. W., Abbot, R. D., Fagan, A. A., & Catalano, R. F. (2009). Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: A test of Communities That Care. Archives of Pediatric Adolescent Medicine, 163(9), 789-798.
Certified
Hawkins, J. D., Oesterle, S., Brown, E. C., Monahan, K. C., Abbott, R. D., Arthur, M. W., & Catalano, R. F. (2012). Sustained decreases in risk exposure and youth problem behaviors after installation of the Communities That Care prevention system in a randomized controlled trial. Archives of Pediatrics & Adolescent Medicine, 166(2), 141-148.
Kuklinski, M. R., Hawkins, J. D., Plotnick, R. D., Abbott, R. D., & Reid, C. K. (2013). How has the econonmic downturn affected communities and implementation of science-based prevention in the randomized trial of Communities That Care? American Journal of Community Psychology, 51, 370-384.
Oesterle, S., Hawkins, J. D., Kuklinski, M. R., Fagan, A. A., Fleming, C., Rhew, I. C., . . . & Catalano, R. F. (2015) Effects of Communities that Care on males' and females' drug use and delinquency 9 years after baseline in a community-randomized trial. American Journal of Community Psychology, 56(3-4), 217-228.
Oesterle, S., Kuklinski, M. R., Hawkins, D., Skinner, M. L., Guttmannova, K., & Rhew, I. C. (2018). Long-term effects of the Communities that Care trial on substance use, antisocial behavior, and violence through age 21 years. American Journal of Public Health, 108 (5), 659-665.
Quinby, R. K., Hanson, K., Brooke-Weiss, B., Arthur, M. W., Hawkins, J. D., & Fagan, A. A. (2008). Installing the Communities That Care prevention system: Implementation progress and fidelity in a randomized controlled trial. Journal of Community Psychology, 36, 313-332.
Rhew, I. C., Brown, E. C., Hawkins, J. D., & Briney, J. S. (2011). Sustained effects of Communities That Care on prevention service system transformation. (Draft). Seattle, WA: Social Development Research Group, University of Washington School of Social Work.
Kim, B. K. E, Gloppen, K. M., Rhew, I. C., Oesterle, S., & Hawkins, J. D. (2015). Effects of the Communities That Care prevention system on youth reports of protective factors. Prevention Science, 16(5), 652-662. doi:10.1007/s11121-014-0524-9
Rhew, I. C., Oesterle, S., Coffman, D., & Hawkins, J. D. (2018). Effects of exposure to the Communities That Care prevention system on youth problem behaviors in a community-randomized trial: Employing an inverse probability weighting approach. Evaluation & the Health Professions, 41(2), 270-289.
Kuklinski, M. R., Oesterle, S., Briney, J. S., & Hawkins, J. D. (2021). Long-term impacts and benefit-cost analysis of the Communities That Care prevention system at age 23, 12 years after baseline. Prevention Science, 22, 452-463.
Rowhani-Rahbar, A., Oesterle, S., Gause, E. L., Kuklinski, M. R., Ellyson, A. M., Schleimer, J. P., . . . Hawkins, J. D. (2023). Effect of the Communities That Care prevention system on adolescent handgun carrying: A cluster-randomized clinical trial. JAMA Network Open, 6(4), e236699. doi:10.1001/jamanetworkopen.2023.6699
Study 2
Feinberg, M. W., Greenberg, M. T., Osgood, D. W., Sartorius, J., & Bontempo, D. (2007). Effects of the Communities That Care model in Pennsylvania on youth risk and problem behaviors. Prevention Science, 8(4), 261-270.
Feinberg, M. W., Jones, D., Greenberg, M. T., Osgood, D. W., & Bontempo, D. (2010). Effects of the Communities That Care model in Pennsylvania on change in adolescent risk and problem behaviors. Prevention Science, 11, 163-171.
Study 3
Rhew, I. C., Hawkins, J. D., Murray, D. M., Fagan, A. A., Oesterle, S., Abbott, R. D., & Catalano, R. F. (2016). Evaluation of community-level effects of Communities That Care on adolescent drug use and delinquency using a repeated cross-sectional design. Prevention Science, 17(2), 177-187.
Van Horn, M. L., Fagan, A. A., Hawkins, J. D., & Oesterle, S. (2014). Effects of the Communities That Care system on cross-sectional profiles of adolescent substance use and delinquency. American Journal of Preventive Medicine, 47, 188-197.
Study 4
Study 5
Toumbourou, J. W., Rowland, B., Williams, J., Smith, R., & Patton, G. C. (2019). Community intervention to prevent adolescent health behavior problems: Evaluation of Communities That Care in Australia. Health Psychology, 38(6), 536-544.
Study 6
Jonkman, H., Aussems, C., Steketee, M., Boutellier, H., & Cuijpers, P. (2015). Prevention of problem behaviours among adolescents: The impact of the Communities That Care strategy in the Netherlands (2008-2011). International Journal of Developmental Science, 9, 37-52. https://doi.org/10.3233/DEV-13121
Study 1
Summary
The Community Youth Development Study [CYDS] consists of 20 articles (three certified by Blueprints: Hawkins, Brown et al., 2008; Hawkins et al., 2009, 2012) that examined a cluster randomized controlled trial of 24 communities across seven states. After matching, 12 communities were assigned to the intervention group and 12 to the control group. A total of 4,407 fifth-grade students in the 24 communities were surveyed nine times from 2004 to 2014, roughly from age 12 to age 23. Outcomes included initiation and prevalence of substance use and delinquent behavior.
Through Grade 7 (Hawkins, Brown et al., 2008):
- Students in control communities were significantly more likely to initiate delinquent behavior between fifth and seventh grades than were students in CTC communities.
Through Grade 8 (Hawkins et al., 2009):
- The incidence of delinquent behavior, alcohol, cigarette, and smokeless tobacco initiation were significantly lower in CTC than in control communities between grades 5 and 8.
- In grade 8, the prevalence of alcohol and smokeless tobacco use in the last 30 days, binge drinking in the past 2 weeks, and the number of different delinquent behaviors committed in the past year in grade 8 were significantly lower in CTC communities compared to control communities.
Through Grade 10, one year after the end of technical assistance (Hawkins et al., 2012):
- The incidence of alcohol use, cigarette use, and delinquency was lower by grade 10 among students in CTC communities than in control communities.
- The prevalence of current cigarette use and past-year delinquent and violent behavior were significantly lower in CTC than in control communities in grade 10.
Through Grade 12, eight years post-baseline (Hawkins et al., 2014; Rowhani-Rahbar et al., 2023):
- Abstinence from drug use, drinking alcohol, smoking cigarettes, and engaging in delinquency were lower in the CTC communities than control communities.
- Students were less likely to ever have committed a violent act or to have carried a handgun in CTC communities, relative to control communities.
Through age 19, nine years after baseline (Oesterle et al., 2015): No main effects.
Through age 21, ten years after baseline and compared to control participants (Oesterle et al., 2018), treatment participants had:
- Higher likelihood of sustained abstinence from gateway drugs (alcohol, tobacco, and marijuana) and marijuana
- Higher likelihood of abstaining from antisocial behavior
- Reduced risk of lifetime engagement in violence
Through age 23, 12 years after baseline and compared to control participants (Kuklinski et al., 2021), treatment participants had significantly lower:
- Alcohol use,
- Illicit drug use, and
- Anti-social behavior.
Community-Level Prevention Service System Outcomes (Brown et al., 2007; Rhew et al., 2011 draft):
- CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004, relative to control communities.
- CTC communities reported higher levels of adoption of a science-based approach to prevention in 2009, 1.5 years after study-funded resources for CTC ended.
Significant Program Effects on Risk and Protective Factors:
- The levels of risk factors targeted by CTC communities were significantly lower among panel students in grade 7 in intervention communities than in control communities after 1.67 years of implementing preventive interventions selected through the CTC process (Hawkins, Brown, et al., 2008).
- Mean levels of targeted risks increased less rapidly between grades 5 and 10, and were significantly lower in grade 10, in CTC than in control communities (Hawkins et al., 2011).
- Protective factors of opportunities for prosocial involvement, recognition for prosocial involvement, and social skills were higher in grade eight in intervention communities than control communities (Kim et al., 2015).
Evaluation Methodology
Design:
Recruitment: With the aid of drug abuse prevention agencies in seven states, the Community Youth Development Study (CYDS) identified 20 communities that were trying to implement risk- and protection-focused prevention services and 21 other matched communities that were not using a risk- and protection-focused approach. The matching was done within each state on population size, racial and ethnic diversity, economic indicators, and crime rates (one community had two matched comparison communities). Of the 20 pairs of communities, 13 were eligible because they had not yet selected or tested effective preventive interventions, and 12 of the 13 pairs agreed to participate. The final sample thus included 24 communities.
Hawkins, Brown et al. (2008) and Hawkins et al. (2009) examined students from 88 schools in the 24 communities who were in 5th grade before the start of the program (2003-2004). Hawkins et al. (2009) referred to 4,407 consented students (76.4% of those eligible). As noted in Hawkins, Brown et al. (2008), the consented number excluded 13 students who were absent during scheduled data collection dates.
Assignment: Members in each matched pair of eligible communities in a state were randomized to the intervention or control condition by a coin toss prior to recruitment. The 12 intervention communities selected 13, 16, and 14 different prevention programs to implement during the school years of 2004-2005, 2005-2006, and 2006-2007, respectively. The number of consented students was 2,405 in the intervention communities (76.2% of those eligible) and 2,002 in the control communities (76.7% of those eligible).
Assessments/Attrition: Hawkins, Brown et al. (2008) and Hawkins et al. (2009) examined four surveys. A baseline assessment of the fifth-grade students in spring 2004 (wave 1) was followed by assessments for sixth-grade students in spring 2005 (wave 2), seventh-grade students in spring 2006 (wave 3), and eighth-grade students in spring 2007 (wave 4). The last follow-up came about 2.6 years after first implementation of the selected programs but during a period of continued technical assistance from the program.
For the first four surveys, Hawkins, Brown et al. (2008) and Hawkins et al. (2009) reported that among the sample of 4,407 consented students, 26.5% did not have fifth-grade baseline data because they were recruited in sixth grade. With the new additions included, data were missing for 3.9% and 3.8% of the students at the sixth- and seventh-grade follow-ups. In addition, missing data reached 10% for some survey items, and 3.7% of the students were dropped because they did not meet the validity screen. However, both studies appear to have excluded 388 consented students (9% of the sample of 4,407) who moved out of the school district before participating in the study for one semester; students who participated for at least one semester in grade six were tracked and surveyed.
Four long-term surveys followed the initial surveys.
- Hawkins et al. (2012): wave six, spring 2009, grade 10, six years after the program start, one year after the end of program technical assistance, 92% valid survey completion among those who consented.
- Hawkins et al. (2014) and Rowhani-Rahbar et al. (2023): wave seven, spring 2011, grade 12, eight years after program start, three years after the end of program technical assistance, 91% valid survey completion among those who consented.
- Oesterle et al. (2015): wave eight, spring 2012, age 19, nine years after program start, four years after the end of program technical assistance, 89% valid survey completion among those who consented.
- Oesterle et al. (2018): wave nine, spring 2014, age 21, 11 years after program start, six years after the end of program technical assistance, 91% valid survey completion among those who consented.
- Kuklinski et al. (2021): age 23, 12 years after the program start. The CONSORT diagram included in supplementary files shows that about 88% of the randomized participants still living provided data.
Sample:
The sample communities had populations ranging from 1,500 to 50,000 residents. The enrolled student population in a single grade in these communities ranged from 40 to 485, with two communities exceeding 400 students per grade. The sample of students was split evenly between males and females. About 70% were white or Caucasian, 9% were Native American, 4% were African American, and 20% were of Hispanic origin. The students were an average of 11.1 years of age at baseline (grade 5).
Measures:
Hawkins, Brown et al. (2008) and Hawkins et al. (2009, 2012) examined three sets of self-reported outcomes. The authors provided little information on reliability and validity, but to minimize reporting bias, they used standard measures, an anonymous survey to help ensure confidentiality, and a validity screen to indicate misreporting.
- Targeted risk factor. A single measure of the targeted risk factor combined multiple items into a standardized scale. The targeted risk factor (e.g., family conflict, academic failure, friends who engage in delinquent behavior) and the specific measure varied across communities. A community's risk factor was therefore compared to its matched control community rather than to the average across all communities.
- Targeted protective factors. Kim et al. (2015) examined 15 scales measuring protective factors relating to community, family, school, and peers. The alpha values for the scales ranged from .66-.90.
- Onset of delinquent behavior. Students reported the first occurrence and recent prevalence of any one of multiple delinquent behaviors. More severe forms of delinquency were added to later waves as they became developmentally appropriate. At the age 21 follow-up, the measure referred to anti-social behavior.
- Onset of substance use. Students reported the first lifetime use and recent prevalence of multiple substances, including alcohol, marijuana, cigarettes, and other illicit drugs (e.g., inhalants, cocaine, barbiturates, ecstasy, prescription drugs).
Hawkins et al. (2014) and Oesterle et al. (2015, 2018) added a measure of sustained abstinence from substance use and delinquent behavior. Oesterle et al. (2015) added secondary outcomes that measured substance use disorders, major depression, suicidality, educational attainment, and sexual risk behaviors. Rowhani-Rahbar et al. (2023) measured participant self-reports of having carried a handgun in the past year. Kuklinski et al. (2021) used measures of substance use and antisocial behavior as primary outcomes and measures of four-year college completion, major depressive disorder, and generalized anxiety disorder as secondary outcomes.
Analysis:
Hawkins, Brown et al. (2008) and Hawkins et al. (2009) used mixed models that accounted for the hierarchical structure of the data. The specific estimates came from linear, logistic, and survival models depending on the outcome. The models controlled for student and community characteristics and baseline outcomes, except for the incidence or onset outcomes used in survival analyses. The intervention effect came from estimating the average difference within the matched pairs; the degrees of freedom for this effect equaled the number of community-matched pairs (12) minus the number of community-level covariates and intervention effect (3) minus 1 (i.e., df = 8). However, the level-2 sample size of 24 may not be large enough to accurately estimate the standard errors, and the result may be to overstate the significance of the tests.
The analyses of initiation or onset dropped students who had already initiated at baseline. Hawkins, Brown et al. (2008) dropped 22.2% for delinquent behavior and 27.5% for substance use. Hawkins et al. (2009) dropped 21.5% for alcohol use, 8.3% for cigarette use, 2.4% for smokeless tobacco use, 0.6% for marijuana use, 8.6% for inhalant use, and 21.2% for delinquent behavior. All analyses used multiple imputation for missing data resulting from non-response of eligible participants and incomplete items of responders.
Hawkins et al. (2012, 2014), Oesterle et al. (2015, 2018), Rowhani-Rahbar et al. (2023), Kuklinski et al., 2021). Depending on the outcome, the long-term analyses adjusted for clustering of time within students and students within communities with three-level growth models, multilevel discrete time survival models, and generalized linear mixed models (linear, Poisson). All models included student and community covariates and, if possible, the baseline outcome. Degrees of freedom equaled the number of matched pairs (12) minus the number of community level covariates. All the studies imputed missing follow-up data. Models of abstinence and incidence of substance use or delinquency included only those who had not initiated those behaviors at baseline.
Rhew et al. (2018). One non-ITT analysis used inverse probability weights to compare intervention and control youths who remained in their original community through eighth grade. The inverse probability weights came from propensity scores for staying in the community, as predicted by a set of baseline sociodemographic and outcome characteristics. The analysis thus examined a subsample consisting of 3,765 youths who stayed in the communities, and the inverse probability weights were used to adjust for selection differences and make the conditions similar for the subsample. The weighted estimates also adjusted for clustering and imputed missing data.
Intent-to-Treat: Hawkins, Brown et al. (2008) and Hawkins et al. (2009) noted that students in the panel who remained in the sample communities for at least one semester in grade six were tracked and surveyed, even if they left the community. In addition, the analyses used multiple imputation for missing data on student and community characteristics, drug use and delinquent behavior outcomes, and community membership. However, it appears that 388 students (9%) in the randomized communities were not followed because they were not present for one full semester of the study, an exclusion possibly related to lack of program participation and possibly a violation of intent-to-treat.
Otherwise, one non-ITT study (Rhew et al., 2018) examined a subsample of youths who remained in their original community through eighth grade, but the analysis adjusted for selection differences across conditions with inverse probability weights.
Outcomes
Implementation Fidelity:
Quinby et al. (2008). CTC implementation fidelity ratings averaged across three groups of raters show that between 89% and 100% of the CTC milestones in the first four phases of CTC implementation were "completely met" or "majority met" in the 12 intervention communities, indicating that the first four phases of the CTC system were well implemented in the communities in this trial. There was high overall agreement (95%) in milestone ratings among the different groups of raters. Five milestones were rated as either "very challenging" or "mostly challenging" for half or more of the communities: preparing archival data to supplement the CTC Youth Survey Data, identifying resources required for new programs and policy implementation, addressing readiness issues, securing a champion, and engaging all stakeholders to support the community action plan.
Fagan et al. (2008a). By using the CTC model to select and monitor the quality of prevention activities, the 12 CYDS communities replicated 13 prevention programs with high rates of adherence to the programs' core components and in accordance with dosage requirements regarding the number, length, and frequency of sessions. Adherence scores ranged from 73% to 99% across all program replications, indicating that program staff taught the majority of program objectives and ensured completion of most of the program components. Dosage scores were also high, as 94% of the dosage criteria were met across all communities. In addition, 81% of the program cycles delivered all required lessons, in the specified amount of time, and with the recommended frequency of delivery.
Fagan et al. (2012). During the active phase of research (2007, 3.5 years after CTC was begun in communities) and at the end of grant-supported activities (2010, 6.5 years after baseline and 1.5 years following the end of training, TA, and funding to intervention communities), according to agency directors and program providers, CTC communities implemented significantly more tested and effective prevention programs and had higher rates of program sustainability compared to control communities at both time periods. CTC sites reached more children and families with prevention services at each time period, but this was only significant during the intervention stage. Only one significant fidelity effect emerged, which indicated that CTC sites provided more program oversight during the sustainability phase.
Fagan et al. (2008b). According to implementer and observer reports, across all 16 programs implemented in CTC communities, large proportions of required material were taught and core components delivered, nearly all lessons were offered in accord with the length and frequency specified by program developers, implementers were prepared and enthusiastic and used a variety of teaching techniques to convey material, and high levels of engagement from program participants were observed.
Baseline Equivalence:
Hawkins, Brown et al. (2008) reported a test for one baseline outcome, finding no significant difference for the targeted risk measure. Hawkins et al. (2009) reported more detail, finding no statistically significant condition differences for drug use or delinquency (p. 793) and similar community sociodemographic characteristics across conditions (Table 1).
Hawkins, Catalano et al. (2008) used a different baseline sample to evaluate baseline equivalence of the intervention and control communities. They examined three cross-sectional surveys of students: 14,293 in 1998, 12,992 in 2000, and 14,910 in 2002. Students in three grades (sixth, eighth, and tenth) were examined separately for 11 measures of substance use and delinquent behavior (33 measures in total). The first set of tests focused on condition differences in 2002, the last survey year; the second set of tests focused on condition differences in the change from 1998 to 2002. Combined with the 33 measures, the two sets of comparison defined 66 tests. Five of these tests reached significance. First, for 2002, significantly more eighth-grade students in control communities reported having attacked someone with the intention of hurting them. Second, for the 1998-2002 period, the trends differed significantly for eighth-grade family history of substance use (a larger increase in the control communities), 10th-grade binge drinking (prevalence diminishing more in control communities), 10th-grade 30-day marijuana use (prevalence diminishing more in control communities), and being drunk or high at school (prevalence diminishing more in control communities).
Differential Attrition:
For the surveys through grades seven and eight, only about 4% of eligible students did not complete the follow-ups, and Hawkins et al. (2009) added that "There was no systematic bias from differential accretion or differential attrition in control and intervention conditions (analyses not shown)."
For the long-term surveys, attrition ranged from 8-11%. All addressed differential attrition, though they did not present the analyses. Hawkins et al. (2012) stated only that, "There was no systematic bias from differential accretion or differential attrition in control and intervention conditions." Hawkins et al. (2014) stated only that, "Accretion and attrition were similar in both intervention groups." Oesterle et al. (2015) stated only that, "Retention did not differ by intervention condition, but was higher for females than males (94 vs. 88 %, respectively)." Oesterle et al. (2018) reported that retention was significantly higher in the intervention than the control group (92% versus 90%) and among females than males (95% vs. 88%), but did not differ by race, ethnicity, or parents' education.
Posttest:
Hawkins, Brown et al. (2008). The results through grade seven showed two significant program effects in three tests. First, the targeted risk factor at grade seven was significantly higher for students in control than intervention communities (d = .15). Second, the hazard for the onset of delinquent behavior was significantly higher in the control than intervention communities (OR = 1.27). Third, the conditions did not differ significantly in the onset of substance use.
Hawkins et al. (2009). The results through grade eight showed eight significant effects in 15 tests. Students in the control communities reported significantly higher incidence of alcohol use, cigarette use, smokeless tobacco use, and delinquency than students in the intervention communities (odds ratios ranged from 1.41 to 2.34). Students in the control communities also reported higher prevalence of alcohol use, smokeless tobacco use, binge drinking, and delinquency than students in the intervention communities (odds ratios ranged from 1.25 to 1.79).
Brown et al. (2013). Higher levels of community adoption of a science-based approach to prevention in 2004 predicted significantly lower levels of youth problem behaviors in 2007. Effects of the CTC intervention on youth problem behaviors were mediated fully by community adoption of a science-based approach to prevention, as reported by key community leaders.
Kim et al. (2015). Results in grade eight (see Figure 2) showed that the youths in intervention communities scored significant better for three of the 16 protective factor scales than youths in control communities: opportunities for prosocial involvement, recognition for prosocial involvement, and social skills.
Rhew et al. (2018). Non-ITT results in grade eight based on inverse propensity weights examined youths who remained in their original community through grade eight. Of seven substance use outcome measures and one delinquency measure (Table 3), four showed significant effects of the intervention in the weighted models for stayers. The intervention group stayers reported significantly lower past month alcohol use, past two-week binge drinking, past month smokeless tobacco use, and past year delinquent behavior than control group stayers.
Long-Term:
To briefly summarize the more detailed description of the long-term results listed below, consistent intervention effects appear through grades 10 and 12 (one year and three years after the posttest). With a few isolated exceptions, however, the benefits were largely absent at ages 19 and 21 (four and six years after the posttest).
Hawkins et al. (2012) - Wave 6, Grade 10 (one year after the end of technical assistance)
In 20 tests of intervention effects, six were statistically significant. Compared to the control group, the intervention group had significantly fewer risk factors (d = -.12), significantly later initiation (of the subsample that had not yet started) for alcohol use (OR = .62) and cigarette use (OR = .54), and significantly lower prevalence of cigarette use (OR = .79), delinquent behavior (OR = .83), and use of violence (OR = .75).
Hawkins et al. (2014), Rowhani-Rahbar et al. (2023) - Wave 7, Grade 12 (three years after the end of technical assistance). In 37 tests of intervention effects (Hawkins et al., 2014), seven were statistically significant. The most consistent effects showed for sustained abstinence (among those not having started at baseline). The intervention group had significantly higher abstinence rates than the control group for any drug use. Rowhani-Rahbar et al. (2023) found that, from grade 6 through grade 12, youths in the program communities were significantly less likely to report handgun carrying in the past year (odds ratio = .76).
Oesterle et al. (2015) - Wave 8, Age 19 (four years after the end of technical assistance). In 46 tests for main effects using the full sample (see Appendix C), three reached statistical significance, but two of those effects were iatrogenic. A global test adjusting for the multiple comparisons was not significant, which discounts the individual significant effects in either direction.
Moderation tests examined intervention effects separately for males and females but revealed few differences. The 32 tests in Figures 2 and 3 show two significant effects for males (lower prevalence of cigarette use and lower delinquency) and no significant differences for females. However, tests for interactions by gender in Appendix D, which show if the intervention effects differ significantly across men and women, indicated only one significant difference in 72 tests. The difference was for an iatrogenic effect among females of ecstasy use; the effects on the prevalence of cigarette use and delinquency did not differ significantly between males and females.
Oesterle et al. (2018) - Wave 9, Age 21 (six years after the end of technical assistance). In 16 tests for main effects using the full sample (Table 1), three reached statistical significance. For the subsample of non-initiates at baseline, the intervention group reported significantly greater abstinence of gateway drugs (d = .26) and anti-social behavior (d = .14) and significantly lower incidence of violent behavior (d = -.12). However, a global test adjusting for multiple comparisons showed that "the CTC system's overall long-term impact across all primary outcomes through age 21 years did not achieve statistical significance." In addition, there were no significant effects on 17 prevalence measures (Appendix A).
Moderation tests in Table 1 found one significant gender interaction in 16 tests for incidence of use of other drugs, but the separate intervention effects on that outcome proved non-significant for both males and females (Table 2). Despite the lack of significant differences between genders, the authors reported significant intervention effects among males only for abstinence of gateway substance use, tobacco use, marijuana use, and anti-social behavior and for incidence of inhalant use.
Kuklinski et al. (2021) - Wave 10, Age 23. Table 2 shows that three of five primary outcomes and zero of three secondary outcomes were significantly affected by the program. The young adults in the intervention group reported significantly lower alcohol use, illicit drug use, and anti-social behavior than the young adults in the control group.
Community-Level Prevention Service Outcomes:
Brown et al. (2007). CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004 relative to control communities.
Rhew et al. (2011). CTC had evidence of sustained effects on prevention system characteristics 1.5 years after study-funded resources for CTC ended. Leaders from CTC communities reported higher levels of adoption of a science-based approach to prevention and a higher percentage of funding desired for prevention activities in 2009 than did leaders in control communities. CTC communities showed a higher increase over time in community norms against adolescent drug use compared to control communities. There was no evidence of a main effect of CTC on community collaboration for prevention, general community support for prevention, or use of the social development strategy.
Gloppen et al. (2012). The extent to which coalitions continued to meet specific benchmarks was measured 20 months following the end of study support for CTC. Only 1 of 12 CTC coalitions had dissolved by winter of 2009. Although funding had decreased in 7 of the 11 coalitions, two-thirds of the coalitions reported having a paid staff person. CTC coalitions continued to report significantly higher scores on the benchmarks of phases 2 through 5 of the CTC system than did the prevention coalitions in the control communities, indicating that CTC coalitions maintained a more scientific approach to prevention than coalitions that did not receive CTC training.
Kuklinski et al. (2013). Many CTC coalitions were resilient two years after the economic downturn that began December 2007. CTC coalitions continued to implement science-based prevention to a significantly greater degree than control coalitions (7 of 11 coalitions maintained 2007 implementation levels through 2009).
Study 2
Summary
Feinberg et al. (2007, 2010) conducted a quasi-experimental study of CTC in Pennsylvania sites using a state-funded survey of adolescents in grades 6, 8, 10, and 12 in public and private schools. The full data sets available contained data on 92 school districts and 43,842 students in 2001 and 159 school districts and 101,988 students in 2003. Outcomes included student self-reports of substance use and delinquency.
Feinberg et al. (2007):
- In general, the pattern of findings shows that communities employing the CTC model had lower levels of risk factors and problem behaviors (delinquency and alcohol/drug use) than communities not employing CTC.
- When contrasting grade cohorts that were actually exposed to evidence-based programs ("expected impact" cohorts), compared to grade cohorts in the same schools that were not exposed to EBPs combined with students from non-CTC schools, youth in "expected impact" CTC grade cohorts demonstrated significant and beneficial effects for risk/protective factors, academic grades, and delinquency.
Evaluation Methodology
Design: Data was obtained utilizing the results from a state-funded surveillance survey of a large sample of adolescents in public and private schools, the Pennsylvania Youth Survey (PAYS). Although not designed as an evaluation of CTC, PAYS utilized the CTC Youth Survey. The survey results included communities that do and do not have CTC coalitions. The PAYS data was collected in 2001 and 2003, by the PA Commission on Crime and Delinquency. In both years, the sample consisted of students in schools that participated as part of a stratified random sampling procedure or schools that volunteered to participate in the survey. In 2001 and 2003 a random sample of schools was chosen to participate as follows: Schools with over 19 (2001) or 50 (2003) students were divided into 6 regions of the state, and for each of the 4 targeted grades (6, 8, 10, and 12) in each region, a separate random sample was drawn. Each school's grade was assigned a likelihood of participation equivalent to the proportion of the regional student population comprised by the school's grade. In addition to the targeted grades, "piggyback" grades at a school could be included in the survey at a financial discount to the school over the typical survey cost. This procedure yielded 43,842 respondents in 2001 and 38,845 respondents in 2003. In 2003, additional schools volunteered to participate in the survey in order to monitor risks and problems in their communities. The full data sets available contained data on 92 school districts and 43,842 students in 2001 and 159 school districts and 101,988 students in 2003. A total of 50 school districts in 2001, and 102 districts in 2003 were associated with a CTC site. For 2001, the mean time since the initial grant began was 16.8 months. For 2003, the mean time was 42.9 months.
From the full data sets, data was removed for (1) 3,752 students in 2003 because the school district identifier was missing in the data set; (2) 6 school districts whose association with CTC could not be determined; and (3) private schools because CTC sites typically focus programs in public, not private contexts.
Sample: The school districts in the combined 2001-2003 sample had an average of 6% of households below the poverty line (range 1 to 22); and an average of 7.9% female-headed households (range 3-35.2). Although many of the school districts in the rural and small town areas were predominantly white, some areas had predominant minority populations. The average percentage for nonwhites was 6.1 (range 0.7 to 77.8%), and the average percent Hispanic was 2 (range 0.2 to 22.1%).
Measures: The CTC Youth Survey (CTCYS) was the student self-report measure used to assess risk and protective factors for adolescent behavior problems as well as substance use and delinquency. Please see the Study 1 write-up for a detailed description of the CTCYS. The 6 outcome measures used for this study included an 8-item scale assessing delinquent behaviors in the past year; a seven-item "drug involvement" scale assessing use of several substances in the past 30 days (alcohol, smoking and smokeless tobacco, marijuana, LSD, cocaine, inhalants); and 4 single items assessing past 30-day alcohol use, binge drinking, being drunk or high in school, and tobacco use. Binge drinking and being drunk in school were not assessed for 6th graders, given the low rates of these behaviors at that age.
Community demographic information was gathered from 2000 census data aggregated at the school district level by the National Center for Educational Statistics. Variables used included percentage of households below the poverty line, percentage of households with nonwhite racial/ethnic backgrounds, population size, and percentage of female-headed households. The indicator of population density was based on census figures and obtained from publicly available data maintained by the Center for Rural Pennsylvania.
Data was collected from each CTC site regarding which programs they implemented, age groups or grades that participated, and dates of implementation. Each program was then checked to see if it was on the SAMHSA list of effective or model programs and if so, was coded as evidence-based. Grade cohorts were also coded at each school as potentially in the range or out of the range that could be affected by any of the CTC-sponsored programs.
Analysis: Feinberg et al., 2007: Separate analyses were conducted for the two waves of 2001 and 2003 PAYS data, and separately by grade (6th, 8th, 10th, and 12th) since risk and outcomes vary in a nonlinear fashion across grades. Analyses were conducted with statistical techniques that accounted for the clustering of students within school districts (multilevel regression models: SAS Proc Mixed). The indicator of community poverty was included in all analyses as a covariate; and a dichotomous variable representing condition (CTC vs. non-CTC) defined the program effect. Percentage of families living in poverty was used as a control (covariate) in all analyses.
Outcomes
Feinberg et al., 2007
Sample Adjustments and Equivalence: Analyses indicated that CTC districts did not significantly differ from non-CTC districts regarding percentage of nonwhite families, population size, population density, or percentage of female-headed family households. However, results indicate that school districts in CTC sites had a higher average proportion of poor family households (7.66%) compared to non-CTC districts (6.55%). The common region of support was examined and districts outside this region were eliminated. As a result, 11 CTC districts were eliminated that were higher on the poverty variable than any non-CTC districts (8 of these districts contributed data in 2001, and 7 districts contributed data in 2003). After eliminating these districts, the two groups no longer differed on level of poverty or other variables.
Results:
Overall, the pattern of results indicated that CTC school districts had lower levels of some risk factors and rates of substance use and delinquent behaviors than did non-CTC school districts. The results for all CTC sites compared to non-CTC sites indicated 4 significant comparisons favoring CTC in 2001 and 16 in 2003. Findings were particularly strong for the 6th and 12th grades in 2003. In addition, there were many fewer significant comparisons favoring non-CTC sites across the 2 years than would be expected by chance.
The first follow-up analysis included only those CTC grade cohorts that could have been directly impacted by an evidence-based model program. Because numerous CTC sites began implementing programs between 2001 and 2003, there were too few such cases in 2001 to analyze. For the 2003 data, analysis of cohorts most likely to be affected resulted in 22 significant comparisons favoring the CTC sites.
Examination of the distribution of significant effects indicated differential effects by grade and by risk factor/outcome. For the 2003 data, limited to CTC grade cohorts expected to be impacted by evidence-based programs, the greatest number of significant effects was found in the 6th grade. CTC influence on risk factors for the 6th grade included poor family supervision and discipline, friends' behaviors and attitudes, and the individuals own attitudes; analyses also indicated positive effects on each of the outcomes for 6th grade, with the exception of the composite drug use index. For the 6th through the 10th grades, analyses yielded consistent evidence of CTC influence on the outcome of delinquent behavior. And for the 12th grade, evidence of CTC effects was demonstrated for alcohol use and the multi-item measure of drug involvement.
Given that some of the variables were skewed, the data was reanalyzed utilizing models in SAS's PROC GLIMMIX procedure with alternate distributional assumptions. Poisson distributions were utilized and Ordered multinomial models were utilized where appropriate. Results were very nearly identical to the original models utilizing PROC MIXED with assumed normal distributions. In the 2003 expected impact analyses, 21 models yielded significant results with the re-analyses, compared to 22 in the original analyses.
Effect sizes were computed as odds ratios with multinomial models. These analyses were conducted for grades and risk factors/outcomes that had demonstrated significant differences between CTC and non-CTC sites in the 2003 expected impact subsample. Responses were recoded to facilitate analyses; for the variables analyzed, recoding yielded between 7 and 16 response levels. Because there were more than two levels of response for each variable, the resulting odds ratio was not interpreted as the likelihood of use vs. nonuse due to CTC as would be the case if typical logistic models had been used. Instead, the odds ratio resulting from this ordinal logistic model reflects the likelihood of a student in a non-CTC district reporting a higher level of risk or substance use than a student in a CTC district. Odds ratios for the models analyzed ranged from 1.12 to 1.36, with an average of 1.23.
Feinberg et al., 2010
As previous cross-sectional findings were open to the criticism that self-selection of communities into CTC may have biased results, this analysis utilized longitudinal data and tested for impact on change. The authors state that this design to a large extent removes the possibility that selection bias is responsible for the findings. It was not possible to track individuals over time due to the anonymous survey, thus the analyses concern changes in groups of individuals over time. For example, 6th graders responding to the 2001 survey were considered the same grade cohort as the 8th graders from the same school responding to the 2003 survey. Thus, subjects were nested within measurement periods which were nested within school district. In this study, each grade cohort at each school that was ever exposed to an evidence-based program was coded a "1" as having an "expected impact" by a universal EBP. All other grade cohorts (i.e., combining non-CTC and non-expected impact CTC grade cohorts) were coded as 0. Three-level hierarchical models were used to examine change in risk/protective factors, grades, delinquency and substance use over time.
There were no differences between CTC and non-CTC communities' grade cohorts in change of risk/protective factors, grades, and substance use. There was a significant difference between groups in delinquency, with youth in CTC communities, relative to control communities, demonstrating less growth in delinquency. In the models that tested Expected-Impact CTC grade-cohorts and comparison grade-cohorts (i.e., non-CTC grade cohorts and all other CTC grade cohorts), there were significant and beneficial intervention effects for risk/protective factors, academic grades, and delinquency, but not substance use.
Study 3
The two articles in this study used cross-sectional survey data for the same 24 randomized communities as in Study 1. Rhew et al. (2016) examined multiple survey years, while Van Horn et al. (2014) focused on the 2010 survey data.
Summary
Rhew et al. (2016) and Van Horn et al. (2014) also used data from the CYDS but examined a series of cross-sectional surveys between 2000 and 2010. Rhew et al. (2016) pooled the 2000 and 2002 assessments for baseline measures and the 2006 and 2008 assessments for follow-up measures and included 6th, 8th, and 10th graders. Sample sizes over survey years ranged from 4,647 to 5,077 for 6th graders, 4,491 to 4,984 for 8th graders, and 3,854 to 4,726 for 10th graders. Van Horn et al. (2014) examined the 2010 cross-sectional survey with samples of 4,674 eighth graders and 4,810 tenth graders.
Rhew et al. (2016), Van Horn et al. (2014):
Among 6th graders, there was a possible harmful effect as control participants had significantly better outcomes than treatment participants in:
- Antisocial behavior
In comparing change between 6th and 10th grade, the study found significant improvement in treatment communities than control communities for:
- Lifetime use of smokeless tobacco
For tenth-grade students in 2010, the study found lower levels in treatment communities than control communities for:
- Alcohol use
Evaluation Methodology
Design:
Recruitment: The data for this study was from the CYDS, a community-randomized controlled trial of 24 small towns in Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington. The CYDS is discussed further in Study 1. The towns were selected from a larger sample of 41 communities participating in an observational study of the diffusion of science-based prevention strategies. Of these, 24 were eligible and consented to participate.
Assignment: The participating communities were matched within states according to population size, racial/ethnic composition, crime rate, and socioeconomic indicators. From each pair, one town was randomly assigned to the treatment condition the other to the control condition. Communities in the control condition did not receive any training in or funding for CTC activities from the CYDS project during the course of the study.
The repeated cross-sectional design compared all students in community schools at each survey year rather than followed individuals longitudinally. The surveys excluded students present at the start of the program who left the schools and included new students who entered the schools after the program start. All comparisons from baseline to follow-up thus included somewhat different samples of students.
Attrition: Assessments occurred at baseline (2000 and 2002) and follow-up (2006, 2008, and 2010). Across the years of the study, the response rate among 6th graders ranged from 79 to 87%, for 8th graders ranged from 72 to 84%, and for 10th graders ranged from 60 to 78%.
Sample: The sample was approximately half male (47.7 to 49.9%) and largely white (64.1% to 77.2%). The average age was 11.6 among 6th graders, 13.6 among 8th graders, and 15.6 among 10th graders.
Measures: Rhew et al. (2016) examined 9 measures of substance use and 12 measures total. Participants were asked to report use of alcohol, cigarettes, marijuana, and smokeless tobacco in the past 30 days and in their lifetimes. In addition, participants were asked whether or not they participated in binge drinking in the past 2 weeks. The study also included 2 measures of delinquency, based on whether or not the individual had sold illegal drugs, stole or tried to steal a motor vehicle, attacked someone, brought a handgun to school, or were arrested. In assessments after the baseline, 3 additional behaviors were added: stole something worth $5, purposely damaged or destroyed someone else's property, and shoplifted. The study recoded these responses into a count of the number of delinquent behaviors and a dichotomous report of any delinquent behavior. Finally, the study had one measure of change in risk and protective factors prioritized by the community.
Van Horn et al. (2014) created four latent-class measures of substance use and delinquent behaviors for the eighth-grade students: abstainer, experimenter, drug user, and problem behavior. For tenth-grade students, the latent-class analysis added an alcohol use measure to the other four measures.
Analysis: To examine the effects of CTC on youth outcomes, Rhew et al. (2016) used a two-stage ANCOVA approach. In the first stage, the study used individual-level data to estimate the adjusted prevalence of each problem behavior for a community in a given year, by grade. This was then used to create an adjusted pooled baseline level in 2000 and 2002 for each outcome and community and an adjusted pooled follow-up level in 2006 and 2008 for reach outcome and community. In the second stage, the study used a mixed effects model in which the adjusted post-intervention pooled community prevalence was regressed on study condition, the pooled baseline level for the community, percent free or reduced price lunches, and number of students in the community.
Rhew et al. (2016) also conducted analyses of changes in prevalence over time within a specific pseudo cohort followed from 6th grade to 10th grade. The study assumed that most students in 6th grade in 2004 were in 10th grade in 2008, creating the pseudo cohort. The analyses used for this pseudo cohort were similar to those above for the grade samples.
In examining cross-sectional data for 2010, Van Horn et al. (2014) used latent class analysis to assess the intervention impact on the latent classes of substance use/delinquency in the eighth and tenth grades.
Intent-to-Treat: Rhew et al. (2016) used all available data with multiple imputation for missing data. Van Horn et al. (2014) used full information maximum likelihood estimation to include participants with partial missing data. Only those "missing data on all variables" were excluded.
Outcomes
Implementation Fidelity: The study did not include any reports of implementation fidelity.
Baseline Equivalence: As discussed above, the communities were assigned in matched pairs based on population size, racial/ethnic composition, crime rate, and socioeconomic indicators. The study did not provide information on equivalence of baseline outcome scores of either communities or students.
Differential Attrition: Although there were no formal tests for attrition, the study examined differences between conditions on stable demographic characteristics at posttest. Program completers in the treatment group were more likely to be white as compared to the completers in the control group. The study did not discuss differential attrition in the outcome measures.
Posttest: Rhew et al. (2016) found one possible iatrogenic effect on antisocial behavior among 6th graders. Those in the treatment group self-reported significantly more antisocial behavior at posttest than the control group. None of the other outcomes were significant among 6th graders and none of the 12 outcomes were significant for either 8th or 10th graders.
In its analysis of a pseudo cohort, Rhew et al. (2016) found that of 12 outcomes, only lifetime use of smokeless tobacco saw significant improvement from 6th to 10th grade when comparing the treatment to control groups.
Using cross-sectional data from 2010 for eighth graders (n = 4,674) and tenth graders (n = 4,810), Van Horn et al. (2014) found one significant intervention effect in seven tests (see Table 2). Tenth graders in the intervention group reported significantly lower scores on a latent-class measure of alcohol use than those in the control group.
Long-Term: No long-term follow-up was reported.
Study 4
Summary
Chilenski et al. (2019) used a quasi-experimental, repeated cross-sectional design with propensity weighting to assess the effect of the intervention on substance use, delinquency, and depression in Pennsylvania students. Data were collected in six waves over a span of 11 years from students in grades 6, 8, 10, and 12, yielding responses from a total of 470,798 students in 388 school districts. Students reported on recent and lifetime substance use (alcohol, tobacco, marijuana, other illicit drugs) and delinquency (arrests, suspensions, selling drugs) as well as depressive symptoms.
Chilenski et al. (2019) reported that across six waves of data, compared to non-CTC school districts, students in CTC school districts reported significantly lower rates of:
- Substance use
- Delinquency
- Depressive symptoms
Evaluation Methodology
Design:
The program was evaluated in Pennsylvania using a quasi-experimental, repeated cross-sectional design with propensity score weighting at the school district level. Data were collected from participating school districts (N=418) in six waves over 11 years, with assessments occurring every other year for students in grades 6, 8, 10, and 12. After dropping school districts in Philadelphia and Pittsburgh because of limited participation in the youth survey and some other school districts because of missing identifiers, the final number of districts was 388. A total of 470,798 student-level responses were recorded.
Group assignment was reconsidered for each survey year. School districts were included in the intervention group if they had received any CTC services (program, policy, event, or activity) for any age group or grade level within the 12 months preceding data collection. Of the 388 school districts in the analysis sample, 51.6% ever received any CTC services.
For each data wave, inverse probability weighting was used to establish equivalence across the intervention and comparison groups. Weights were created using 27 district-level variables including aggregate indicators of family size, employment, education, demographic composition, population size, and marital status (see Table S4 in supplement).
Sample:
Over all data waves, approximately 53% of students reported lifetime alcohol use (24% recent), 27% reported lifetime cigarette use (12% recent), and 17.7% reported lifetime marijuana use (9% recent).
Measures:
Assessments occurred in 2001, 2003, 2005, 2007, 2009, and 2011 for students in grades 6, 8, 10, and 12 using self-reported items from the CTC Youth Survey. Substance use was measured separately for alcohol, cigarettes, and marijuana, with students reporting any lifetime use and any use in the past 30 days. Severe substance use was assessed with four items reporting recent substance use in school, recent binge drinking, and lifetime/recent use of any other drugs. Delinquency over the preceding 12 months was measured with five items reporting whether a student had been suspended from school, sold illegal drugs, stolen a vehicle, been arrested, and/or assaulted another person. Depression was assessed using the mean of four items (a=.88) measuring agreement with statements like, "sometimes I think that life is not worth it." To strengthen the validity of the measures, students who reported using a fictitious drug were dropped.
Analysis:
The effects of the intervention were evaluated with means comparison and multilevel regression models (students nested within year, year within district) that controlled for individual and district-level characteristics.
Intent-to-Treat: The authors stated that "Due to concerns regarding missing data and spillover effects, school districts that were served by a CTC in 1 year, but later not nominated were dropped for those particular years." School districts in Pittsburgh and Philadelphia (n=30) were dropped from analysis because they were not in areas served by CTC coalitions and, due to the urban setting, were not considered comparable to other school districts in the sample. Students were dropped from analysis if they responded affirmatively to a control item about using a fictitious drug.
Outcomes
Implementation Fidelity:
Not reported. Authors note that CTC was implemented in Pennsylvania in the late 1990s and that many programs and practices implemented in the early years would not meet current standards of evidence-based programming. As a result, any coalitions implementing non-evidence-based programs would fail to meet current standards for CTC implementation fidelity.
Baseline Equivalence:
No differences between conditions on sociodemographic measures after propensity weights were applied, but it was not possible to test for differences on outcome measures.
Differential Attrition:
No typical attrition due to design, but no tests were done for the high levels of missing data in student responses.
Posttest:
Across all data waves, compared to the comparison group, students in CTC intervention school districts had significantly improved outcomes in 12 of 16 tests. They were significantly less likely to report recent alcohol use, recent and lifetime use of cigarettes and marijuana, in-school substance use, binge drinking, and lifetime use of other illicit drugs. Intervention group participants were also significantly less likely to report being arrested (lifetime and past 12 months), school suspension within the past 12 months, and selling illegal drugs within the past 12 months. Adjusted odds ratios ranged from .85-.95. Additional results that included only districts using evidence-based programs in the CTC intervention group showed slightly stronger effects.
Long-Term:
Results indicate consistently lower levels of substance use in CTC school districts, but it appears that the program was still in operation during all six assessment waves and, despite the six waves of follow-up, the results do not constitute true long-term effects.
Study 5
Summary
Toumbourou et al. (2019) used a quasi-experimental, repeated cross-sectional design with no matching. Cross-sectional data were collected across 15 years. A total of 41,328 grade school students (mean grade = 8) reported on their alcohol use, tobacco use, cannabis use, and antisocial behaviors.
Toumbourou et al. (2019) reported that across 15 years of data collection, students in CTC communities, compared to non-CTC communities, reported less:
- Alcohol use
- Tobacco use
- Cannabis use
- Antisocial behavior
In addition, students in CTC communities, compared to non-CTC communities reported:
- A greater average composite of protective factors
- A reduced average composite of risk factors
Evaluation Methodology
Design:
Recruitment: This study evaluated the effect of CTC community intervention exposure compared to Australian youth population trends. Student data were collected as part of the National CTC Youth Survey, administered between 1999 and 2015. Participants in this survey were youth in selected grade levels (unspecified, though average grade was 8) from participating schools in 109 Australian localities (communities). Relevant education authorities and principles gave their consent for their school to be included in the sample. Students were recruited during school hours.
Assignment: CTC was evaluated in this study using a quasi-experimental, cross-sectional design across 15 years. The intervention group in this evaluation included the five of 109 localities that completed all five phases of the CTC process (n = 13,861). The localities differed in the number of times they had initiated a CTC cycle and when those cycles were carried out. Some localities had completed a CTC cycle during the collection period and then discontinued the program. Others had CTC programs ongoing at the end of the data collection period. The other 104 localities served as the comparison group (n = 27,467). There was no formal matching between the CTC communities and the comparison communities.
Assessments/Attrition: The National CTC Youth Survey was implemented no more than once per year during the data collection timeframe. The current study used data from 11 waves of CTC youth surveys (years unspecified). The survey was completed by students at their schools during class time. All data were collected from cross-sectional, aggregated school surveys, and therefore there was not attrition in the typical sense. Student participation in the surveys between localities ranged from 50-90% (average of 73%).
Sample:
Across all collections, students (N = 41,328) had an average age of 13.5 years (SD = 1.7), their average grade was approximately 8, and 51.7% were female. It is unclear as to whether students participated multiple times over the 15-year data collection period.
Measures:
The data for the four primary behavioral outcomes were drawn from 11 cross-sectional CTC Youth Surveys (which specific years had measurements was not disclosed). The CTC Youth Survey is a well-known and validated measure of behavioral outcomes and youth risk and protective factors. Students reported lifetime alcohol, tobacco, and cannabis use separately. The alcohol and cannabis measures were coded to have dichotomous outcomes (tobacco use was already assessed dichotomously). Antisocial behavior was assessed by asking participants to report the number of times in the last 12 months they had 1) carried a weapon, 2) sold illegal drugs, 3) stolen or tried to steal a motor vehicle such as a care or motorcycle, 4) attacked someone with the idea of seriously hurting them, and 5) been drunk or high at school. Response options ranged from never to ten or more times. Antisocial behavior was scored as the count across the five items averaged to range from 0 to 1.
In addition, students reported on eight risk factors and seven protective factors. The risk factors included community substance abuse, low community attachment, parent favorable attitudes toward substance use, poor family management, family conflict, low commitment to school, academic failure, and perceived substance use. The protective factors included community opportunities for prosocial involvement, family attachment, family opportunities for prosocial involvement, family rewards for prosocial involvement, school opportunities for prosocial involvement, school rewards for prosocial involvement, and emotional control. Each of these risk and protective factors consisted of multiple items that were aggregated. These aggregated factors were then combined to form an average composite risk factor score and an average composite protective factor score.
Analysis:
The effects of the intervention were evaluated with multilevel linear mixed effects models with students nested within municipal localities (communities). Behavioral outcomes and risk and protective factor composite scores were predicted by the interaction of the survey year annual trend estimate and CTC status (CTC community vs. non-CTC community). Year 1 estimates served as baseline outcome controls in linear trend models. All analyses controlled for clustering of students within communities, gender, school grade, urban location, government school, survey response rate, community disadvantage, child country of birth, Indigenous status, peer substance use, family antisocial attitudes, and community mobility.
Intent-to-Treat: It appears that all available data were used. However, missingness did vary between outcomes (range: 1% - 15%). In addition, missing data were included in the referent for three variables (government school, child country of birth, and Indigenous stats) and were imputed using regression from correlated variables for gender, peer substance use, family antisocial attitudes, and community mobility. The results using models with imputed data were similar to those using nonimputed data, so the results were presented including the imputed data.
Outcomes
Implementation Fidelity:
No quantitative measures of fidelity were provided. However, the authors stated that an independent process evaluation reported that the intervention communities successfully completed all five phases of the CTC process.
Baseline Equivalence:
At baseline, the CTC communities, relative to the non-CTC communities, had significantly higher levels of alcohol use, cannabis use, family antisocial attitudes, higher community mobility, and lower percentages of non-Australian births.
Differential Attrition:
No typical attrition due to the cross-sectional repeated measures study design.
Posttest:
Across the data collection time frame, aggregated data from students in CTC communities, relative to non-CTC communities, showed significantly stronger annual linear trend decreases in all four behavioral outcomes: alcohol use (adjusted OR = .94), tobacco use (adjusted OR = .97), cannabis use (adjusted OR = .96), and antisocial behaviors (unstandardized b = -.001).
In addition, students in CTC communities reported more protective factors (as assessed by the protective factors composite variable, unstandardized b = .007) and fewer risk factors (as assessed by the risk factors composite variable, unstandardized b = -.007) compared to students in non-CTC communities.
Long-Term:
The results were analyzed using annual linear trends of repeated measures cross-section data across the 15-year data collection period, which coincided with the treatment. Therefore, there were no long-term outcomes.
Study 6
Summary
Jonkman et al. (2015) used a quasi-experimental design with propensity score matching to examine 630 adolescents living in ten neighborhoods in five Dutch cities. Three cities randomly assigned neighborhoods and two cities non-randomly assigned neighborhoods to the intervention and control groups. The adolescents were assessed on substance use, anti-social behavior, and risk and protective factors at baseline and four years later at posttest.
Jonkman et al. (2015) found no significant effects of the intervention on any of the outcome measures.
Evaluation Methodology
Design:
Recruitment: The sample came from ten Dutch neighborhoods in five Dutch cities. In these neighborhoods, 3,368 adolescents aged 12-14 and their parents were contacted. The 785 adolescents who returned both the parental and participant consent forms comprised the baseline sample. The authors stated that the study sample was representative of adolescents in the communities in terms of age, gender, and ethnicity.
Assignment: Although designed as a randomized controlled trial, the study became quasi-experimental. The ten neighborhoods were matched in pairs within cities, based on population size, racial and ethnic diversity, economic indicators, and rates of crime and other problem behaviors of adolescents. Three of the cities agreed to use random assignment of the pairs to the intervention and control neighborhoods, but two cities non-randomly assigned the neighborhoods. Consent rates were low and differed across the five intervention and five control neighborhoods (24.3% and 16.3%, respectively). The final sample of 785 adolescents included 511 (70%) in the intervention condition and 274 (30%) in the control condition.
Given the non-random assignment in two cities and the low consent rate, the study used a quasi-experimental design with propensity score matching for analysis of the data. After matching, the total sample dropped from 274 control and 511 experimental subjects to 230 control and 400 experimental subjects (630 in total). Only adolescents who were comparable according to the propensity score analysis remained in the sample. The study did not discuss the predictors used in the matching.
Assessments/Attrition: Assessments occurred at baseline in 2008 and every year until the posttest in 2011, a four-year period in total. Over three quarters (75.5%) of the adolescents participated in all four waves of data collection, 11.6% in three waves, 7.7% in two waves, and 5.2% in only one wave. However, the multivariate analysis included only the first (baseline) and last (posttest) assessments.
Sample:
At baseline, the adolescents were ages 12-14, with about 55% female and 80% Dutch.
Measures:
All outcome measures were self-reported by the adolescents. The primary outcomes included three alcohol measures, one smoking measure, and one anti-social behavior measure. The secondary measures included 13 risk and protective factors covering family, school, peer group, and community domains. Only those risk and protective measures with Cronbach's alpha coefficients of .70 and higher were used.
Analysis:
After pre-processing the data using propensity score matching, the analysis used mixed models with dummy variables representing the city in which a subject lived. Intervention-by-time interaction terms assessed the program effect from baseline to posttest. The three levels of clustering included measurements, adolescents, and matched sets of communities (with measurement referring to baseline and posttest assessments). The models used "all the information of the adolescents who participated in the first wave of data collection" (p. 43). Both logistic and linear multilevel analyses were used, depending on the type of outcome. Bonferroni corrections were used to control for multiple significance tests.
For initiation of alcohol use and smoking, the analysis used survival models.
Missing Data Methods: The FIML estimation included participants with missing posttest data under the missing-at-random assumption.
Intent-to-Treat: The analysis used all participants with baseline data.
Outcomes
Implementation Fidelity:
The study didn't provide any formal measures of implementation fidelity, but the authors noted that the full program implementation "was hampered by the very small number of tested and effective preventive interventions available in the Netherlands." Also, implementation of the programs was delayed in some neighborhoods.
Baseline Equivalence:
Table 1c shows six significant differences in 25 tests before matching and two significant differences in 24 tests after matching. However, the tests include only sociodemographic measures, not outcomes.
Differential Attrition:
Attrition appeared to be similar across conditions. About 76.3% of the intervention adolescents and 74.1% of the control adolescents participated in all four waves of data collection. In addition, the authors stated that "Examination of the background variables among experimental and control adolescents over the years showed similar patterns of age, gender, ethnicity, and parental work status."
Posttest:
The results for the five substance use measures in Table 2 showed no significant intervention effects, and survival analysis of initiation to alcohol use and smoking also showed no significant interaction effects.
The results for the 13 risk and protective factors in Table 3 showed no significant effects, either for the whole sample or for any of the paired communities.
Long-Term:
It appears that the program was intended to last all four years of the study period, making the last assessment a posttest rather than a long-term follow-up.