RealTeen

Study 1

Study 1 served as the pilot test for RealTeen. This study used an earlier version of the program including 12 sessions.

Evaluation Methodology

Design:

Recruitment: This randomized controlled trial recruited 13- and 14-year-old girls through solicitation e-mails to users of Kiwibox.com, a youth-oriented website. Information packets were sent to the approximately 450 girls (and their parents) who responded to the solicitation, requiring parents to provide written consent to their daughter's participation.

Assignment: Of the roughly 450 girls who submitted interest forms, 236 were enrolled and randomly assigned to the treatment (n=118) or control (n=118) group. The authors do not state why only 236 girls were enrolled in the study; presumably some did not qualify or return consent forms.

Assessments/Attrition: Participants completed online assessments at baseline, posttest, and at 6-month follow-up. Girls in the treatment group completed the posttest survey at the conclusion of the last intervention session, while control group participants completed the posttest 6 weeks after the baseline. The attrition at posttest was 6.8% and the attrition at the 6-month follow-up was 9%.

Sample: The sample was 236 13- and 14-year old girls, from 42 U.S. states and 4 Canadian provinces. The racial composition was 60.6% White, 16.5% Black, 7.2% Latina, 6.8% Asian, and 8.9% other race. The average letter grade participants reported receiving in school was between an "A" and a "B." About half the sample reported living with both a mother and father, and about 90% reported attending a public school.

Measures: The outcome measures consisted of substance use items adapted from the CDC's Youth Risk Behavior Survey, which asked how many times in the past month and week participants used alcohol, cigarettes, marijuana, cocaine, inhalants, methamphetamines, and ecstasy. These responses were then used to create indicators for how many times the following substances were consumed in the last 30 days:

• alcohol,
• cigarettes,
• marijuana,
• poly drug use (cigarettes, marijuana, cocaine, inhalants, methamphetamines, ecstasy), and
• total substance use (alcohol, cigarettes, marijuana, cocaine, inhalants, methamphetamines, ecstasy)

The posttest outcomes measured substance use in the last 30 days, but on average the girls completed the intervention in 1 month, so for those finishing faster than average, the posttest may have captured pre-intervention behaviors. The timing may explain why there were no effects for the posttest.

Additionally, potential mediators, all based on items presumably generated by the investigators, included:

• normative beliefs (10 items, alpha=.89)
• decision making skills (4 items, alpha=.66)
• goal setting skills (3 items, alpha=.69)
• drug resistance/refusal skills (2 items, alpha=.84)
• stress management (3 items, alpha=.67)
• social skills (5 items, alpha=.77)
• self-esteem (4 items, alpha=.85)
• body esteem (4 items, alpha=.74)
• self-efficacy (4 items, alpha=.68)

Analysis: The study determined significant intervention effects using generalized linear models analysis of covariance. Effect sizes were calculated using Cohen's d. Analyses adjusted for baseline outcomes, length of time between pretest and posttest, age, and average letter grade in school. Randomization and analysis were both conducted at the individual level.

Mediation analyses regressed 9 mediator variables on the baseline outcomes, and for the 2 significant predictors, ANCOVA determined significant differences between treatment and control groups at the posttest and 6-month follow-up.

Intent-to-Treat: The study appears to be intent-to-treat. Ten girls in the intervention group failed to complete the full 12 sessions, these girls also did not take the posttest or 6-month follow-up survey, though presumably surveys were offered to these participants.

Outcomes

Implementation Fidelity: Ninety-two percent of the intervention group girls completed all 12 sessions of the prevention program. Further detail on the extent to which girls interacted with the intervention program and the larger website is unknown.

Baseline Equivalence: Authors stated that participants in the intervention and control groups were comparable on demographics (see tests in Table 1), mediating variables, and substance use; however, tests were not shown for the mediating and substance use measures.

Differential Attrition: The attrition rate was 6.8% for the posttest and 9% for the 6-month follow-up; these rates did not differ by condition. Authors indicated that differential attrition analyses were conducted on the same variables analyzed for baseline equivalency, but did not provide the results. No other information was offered on the relationship between baseline characteristics and attrition.

Though the study gave attrition rates for both follow-up surveys, it did not report the effective sample size. Of the 236 girls at baseline, 15 were dropped due to unreliable responses and 9% were lost to attrition, but it appears from the degrees of freedom that analyses included less than 200 participants. It therefore appears that the analyses only included girls who completed both follow-ups, but the study provides no further detail on the effective sample.

Posttest and 6-month follow-up: Those participating in the program showed reduced rates of alcohol, marijuana, poly drug, and total substance use at the 6-month follow-up (respective effect sizes= .20, .20, .19, .20). There were no significant differences for cigarette use at the 6-month follow-up. The results from the posttest administered at the end of the intervention indicated no significant difference for any outcome.

Of the 9 possible mediating effects, only self-efficacy significantly predicted the outcome at baseline and showed a significant difference across conditions at the 6-month follow-up. Normative beliefs also significantly predicted the baseline outcome, but significant difference across conditions were not maintained at the 6-month follow-up. No other mediator significantly predicted the baseline outcome. Overall, the mediation analysis was incomplete and the results were weak.

Study 2

Evaluation Methodology

Design:

Recruitment: Participants were 13-14 year old girls from 48 states and the District of Columbia. They were recruited through Facebook ads (ads were targeted toward just 13-14 year old girls). Those who indicated interest in the study and returned signed consent forms were enrolled in the study.

Assignment: The sample consisted of 788 13-14 year old girls from the United States. Participants were randomly assigned to either the treatment condition or control condition. All participants were given access to the study's website and directed to complete the pretest measures. Intervention group participants (n = 396) were then directed to begin completing the intervention sessions. Control group participants (n = 392) were not given any tasks to complete.

Assessments/Attrition: Participants were assessed at five time points: pretest, posttest, 1-year follow-up, 2-year follow-up, and 3-year follow-up. For intervention group participants, the posttest was administered upon completion of the intervention sessions. For control group participants, the posttest was administered 14 weeks after the pretest was completed. All participants completed the 1-year follow-up 12 months after the completion of their posttest. The 2-year and 3-year follow-ups were completed roughly two and three years after posttest.

Very little information was provided about attrition. The authors indicated that there was 97.5% sample retention for the risk and protective factors measures at posttest, 97% retention at 1-year follow-up, 94% retention at 2-year follow-up, and 93.4% retention at 3-year follow-up, but they did not report on retention rates for the behavioral outcomes.

Sample: Participants were, on average, 13.7 years old (SD = .67), and were 63% White, 17% Black, 15% Latino, 4% Asian, and 16% some other ethnic/racial group. More than 80% of participants came from urban areas, 9% came from large towns, and 10% from small towns. Their most recent average letter grade in school was between "mostly A's" and "mostly B's." In the previous month, 26% had used alcohol, 12% had been binge drinking, 14% had used cigarettes, and 14% had used marijuana.

Measures: The primary outcomes for this study were self-reported frequency, over the last month, of using alcohol, cigarettes, e-cigarettes (at 2- and 3-year follow-ups), marijuana, other drugs (e.g., club drugs, cocaine), as well as self-reported frequency of binge drinking. In addition, participants completed measures of anxiety, depression, body image, coping, stress, media literacy, goal setting, self-esteem, self-efficacy, alcohol refusal, cigarette refusal, marijuana refusal, and peer drug use. Outside of anxiety and depression, these latter measures assessed risk and protective factors and all exhibited acceptable reliability (alphas > .70 across all measurements).

Analysis: The primary outcome measures were analyzed using negative binomial generalized linear regression models. At posttest and 1-year follow-up, these models predicted each drug use measure, independently, from treatment condition while controlling for pretest drug use, age, average letter grade in school, and parents' education. At 2- and 3-year follow-ups, the models only controlled for pretest drug use. All risk and protective factors were analyzed using ordinary least squares regression models. At pretest and 1-year follow-up, the models controlled for pretest score, age, average letter grade in school, and parents' education level. The authors also examined the robustness of their findings by running each model 1) with only the pretest measure as a covariate, and 2) with no covariates. The results were the same under these specifications. No covariates were specified at 2- and 3-year follow-ups.

Intent-to-Treat: No information was provided about intent-to-treat. Authors mention that some participants did not complete all of the intervention sessions but they do not state what was done with their data. In addition, 13 participants (1.6%) had their data removed for reporting extreme scores or unreliable patterns of drug use at posttest and 1-year follow-up; 29 were removed for the same reasons at 2- and 3-year follow-ups.

Outcomes

Implementation Fidelity: The intervention was presented to participants online, and therefore was standardized for everyone. Among intervention participants, 87% completed all sessions; 3% completed zero sessions.

Baseline Equivalence: There were no significant differences between conditions on 11 measures at baseline. These measures included demographic information and baseline substance use. The tests did not include the numerous measures of risk and protective factors.

Differential Attrition: No information was provided concerning differential attrition, but it may be that attrition was only 3-7% across all measures and time points.

Posttest: At posttest, intervention participants, compared to control participants, reported less cigarette usage over the past month (IRR = .419), with 1 of 5 tests for behavioral outcomes reaching significance. For risk and protective factors, intervention participants reported higher self-esteem (B = -.11), goal setting (B = .09), media literacy (B = -.12), and self-efficacy (B = -.12) compared to control participants.

Long-Term: Sustained effects were detected on one of six outcome measures (since e-cigarettes were added at the 2- and 3-year-follow-ups). Intervention participants, compared to control participants, reported less cigarette usage over the past month at the 1-year (IRR = .326), 2-year (IRR = .474) and 3-year (IRR = .353) follow-ups. The other five behavioral outcomes showed mixed results across time. Treatment participants reported less binge drinking at the 1-year follow-up (IRR = .477). At 2-year follow-up, intervention participants, compared to control participants, reported less frequent use of marijuana (IRR = .549) and other drugs (IRR = .365) over the last month. At 3-year follow-up, intervention participants, compared to control participants, reported less frequent use of e-cigarettes (IRR = .146) over the last month.

For risk and protective factors, at the one-year follow-up, intervention participants, compared to control participants, reported higher alcohol refusal skills (B = -.22), cigarette refusal skills (B = -.16), marijuana refusal skills (B = -.18), coping skills (B = .11), and media literacy (B = -.08), as well as lower peer drug use (B = -.60). At the two-year follow-up, intervention participants, compared to control participants, reported lower peer drug use (B = -.55), increased drug refusal skills (B = -.12), improved coping skills (B = .12), self-esteem (B = -.11), media literacy (B = -.11), and self-efficacy (B = -.10). And at the three-year follow-up, intervention participants, compared to control participants, reported lower peer drug use (B = -.76), increased drug refusal skills (B = -.15), self-esteem (B = -.09), media literacy (B = -.09), and self-efficacy (B = -.09).