Sign Up For Newsletter

Blueprints For Healthy Youth Development logo

Year Up

A training and internship program that helps young people with limited post-secondary education get high-quality jobs by learning to work with technology, developing employment skills, and obtaining internships.

Program Outcomes

  • Employment
  • Post Secondary Education

Program Type

  • Employment - Vocational
  • Skills Training

Program Setting

  • Community

Continuum of Intervention

  • Selective Prevention

Age

  • Adult
  • Early Adulthood (19-24)
  • Late Adolescence (15-18) - High School

Gender

  • Both

Race/Ethnicity

  • All

Endorsements

Blueprints: Promising
Social Programs that Work:Suggestive Tier

Program Information Contact

Roberto Zeledon
Chief Marketing Officer
Year Up
45 Milk Street, 9th Floor
Boston MA 2109
Phone: (855) 305-9995
Email: rzeledon@yearup.org
Website: www.yearup.org

Program Developer/Owner

Roberto Zeledon
Year Up


Brief Description of the Program

Year Up provides six months of full-time training in the IT, business operations, sales and customer support, software development, and financial service sectors followed by six-month internships at major employers. The full-time program provides extensive supports and puts a heavy emphasis on the development of professional and technical skills. Students receive weekly stipends to help cover transportation and other program-related expenses. The program targets young adults with a high school diploma or equivalent who are motivated and who, with assistance, can overcome challenges and successfully enter careers in fast-growing technical occupations.

Outcomes

Study 1 (Fein and Hamadyk, 2018; Fein et al., 2021; Fein & Dastrop, 2022): Compared to the control group, the intervention group showed significantly more

  • employment (3 and 5 years after program uptake)
  • average quarterly earnings (3, 5, and 7 years after program uptake)
  • college enrollment (3 years after program uptake)

Risk and Protective Factors

  • increased health insurance rates (3 years after program uptake)
  • reduced financial hardship (3, 5, and 6  years after program uptake)
  • decreased public benefit (5 years after program uptake)
  • improved living arrangements (5 years after program uptake)
  • reported life changes (5 years after program uptake)
  • improved career pathways employment and self-assessed career development (5 years after program uptake)

Study 2 (Roder and Elliott, 2011, 2014): Compared to the control group, the intervention group showed significantly more

  • full-time employment and IT and finance jobs
  • earnings and hourly wages

Risk and Protective Factors

  • college financial assistance and interest in attending college

Brief Evaluation Methodology

Study 1 (Fein and Hamadyk, 2018, Fein et al., 2021): After recruiting a sample of 2,544 young adults located near eight program offices, local staff randomly assigned participants at a 2:1 ratio to the intervention (N = 1,669) and control groups (N = 875). Survey data of participants were collected at 18 months (Fein & Hamadyk, 2018), three years (Fein et al., 2021), and six years (Fein & Dastrop, 2022) after randomization. Administrative records were also collected that followed participants for up to three years (Fein & Hamadyk, 2018), five years (Fein et al., 2021), and seven years (Fein & Dastrop, 2022) after randomization. The primary measures focused on earnings. Secondary and exploratory measures included employment, college enrollment, financial hardship, marriage and childbearing, and personality traits.

Study 2 (Roder and Elliott, 2011, 2014) recruited 195 youths (ages 18-24) in three Eastern cities in the United States for the study and randomly assigned them to the program group or a no-treatment control group. The youths were followed for three years after the end of the program. The analysis examined self-reported measures of earnings, employment, and college attendance.

Study 1

Fein, D., & Hamadyk, J. (2018). Bridging the opportunity divide for low-income youth: Implementation and early impacts of the Year Up program, OPRE Report #2018-65 (and Appendices). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.


Fein, D., Dastrup, S., & Burnett, K. (2021). Still bridging the opportunity divide for low-income youth: Year Up's longer-term impacts, OPRE Report 2021-56. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.


Judkins, D., Walton, D., Durham, G., Litwok, D., & Dastrup, S. (2021). Still Bridging the Opportunity Divide for Low-Income Youth: Year Up's Longer-Term Impacts, Technical Appendices, OPRE Report 2021-56. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.


Fein, D., & Dastrup, S. (2022). Benefits that last: Long-term impact and cost-benefit findings for Year Up, OPRE Report 2022-77. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.


Judkins, D., Roessel, E., & Durham, G. (2022). Career pathways long-term outcomes study: Appendices for PACE six-year impact reports. OPRE Report 2022-69. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.


Race/Ethnicity/Gender Details
Differences in program impacts by gender were not statistically significant, but impacts did vary by race-ethnicity. Results indicated that the intervention increased the earnings of non-Hispanic blacks less than other groups. Overall, however, the program benefitted all groups.

Year Up has standardized training materials, implementation procedures, and playbooks for the delivery of the Year Up program across the network. 

To prepare Year Up staff for their functional roles and to deliver the intervention, Year Up onboards new employees through a collaborative process among the local Operations team, the hiring manager, and Year Up's National office. The manager of each new hire leads all functional onboarding related to organizational or individual responsibilities. A key component to the orientation process is a parallel and continuous self-directed, virtual orientation to train new employees in their specific Year Up role as well as Year Up's organizational practices, program delivery, culture, and more. All new employees participate in a nationally-led, multi-day, new hire in-person training called Baseline In-Person. In addition to this extensive onboarding process, Year Up also offers ongoing training and professional development to staff to ensure staff are equipped in their functional role and are growing professionally. A good deal of this ongoing development is provided by national and regional leadership teams that have long tenure and deep functional expertise in the delivery of different components of the Year Up program (e.g., Admissions and Enrollment, Program Management, Student Support Services, Academic Services and Delivery, Internship Support, Employment Placement Services, Corporate Partner Support and Relationship Management). 

 

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Start-Up Costs

Initial Training and Technical Assistance

$63 cost-per-student (Staff Training)

Curriculum and Materials

$770 cost-per-student (College Fees, including tuition not covered by financial aid for courses providing technical skills training at our college-based locations)

Licensing

$15 cost-per-student (Licensing & Hosting Fees)

Other Start-Up Costs

Employee Recruiting Fees Startup:  $5,000
Postage & Delivery Startup:  $500
Furniture and Equipment Startup:  $2,500
Printing Design & Photo Startup:  $6,000
Office Supplies Startup:  $200
Building Rental & Building Maintenance & Utilities:  $1,457 cost-per-student (also noted under ongoing costs)

Intervention Implementation Costs

Ongoing Curriculum and Materials

Costs Per Student Per Year
Drug Testing & Screening:  $75
Program Supplies:  $20
Printing Design & Photo:  $13  
Catering:  $50

Costs Per Staff Per Year
Employee Recruiting Fees:  $200
Technology:  $2,700
Office Supplies:  $200
Car Rentals:  $200
Staff Training:  $400
Travel (at 1 Learning Community = 40 students):  $1,250
Travel (at 2 Learning Communities = 80 students):  $1,000
Travel (at 3+ Learning Communities = 120+ students):  $750
Meals:  $650
Lodging:  $750

Costs Per Cohort Per Year
Fellow Service:  $4,500
Postage & Delivery:  $500
Awards:  $1,000

Fixed Costs
Registration, License & Fees:  $500
Management Consultants (if greater than year 2):  $20,000
Printing Design & Photo Fixed:  $5,000
Advertising:  $5,000
Catering (if greater than year 1):  $20,000
CC Processing Fees:  $500
Furniture and Equipment:  $5,000
Miscellaneous:  $500
Membership & Subscriptions:  $7,500

Staffing

To serve 1 cohort of 40 students (called an "LC," or Learning Community), Year Up will staff 13 headcount at $1,215,175 annually. This equated to roughly $13,868 cost-per-student in FY2018 (based upon data from all sites).

We cost out our centralized National departments who support our sites at $2,400 per student and $13,000 per full-time staff headcount.

Other Implementation Costs

Learning & Development Weekly Rate: $50 per student per week
Internship Weekly Rate: $150 per student per week, with the exception of our Bay Area market which is $250 per student per week
Other Student Direct Costs: $100 per student per year

Student Transportation: $250 per student per year

Building Rental and Building Maintenance & Utilities: $1,457 cost-per-student

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

Ongoing training, fidelity, technical assistance, and other supports are facilitated by National Program, Human Resources (including a dedicated training team), National Research and Business Intelligence Teams described under administration requirements and cost considerations above. All of these services are included in the centralized cost calculations at $2,400 per student and $13,000 per full-time staff headcount.

Fidelity Monitoring and Evaluation

No information is available

Ongoing License Fees

No information is available

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

Student stipends are $3,600 per student in nearly all locations starting in 2020.

Year One Cost Example

Funding Overview

No information is available

Allocating State or Local General Funds

Historically, 1% of all of Year Up's revenue is derived from any form of government funding, federal, state or local.

Foundation Grants and Public-Private Partnerships

Year Up raises philanthropic capital from a diverse portfolio of individuals (~30% of all philanthropy), corporate giving entities (~10% of all philanthropy), and foundations (~59% of all philanthropy). Philanthropy accounts for about 40-45% of our total revenue. 

Generating New Revenue

Year Up earns Internship Revenue from our corporate partners, who pay market rates to Year Up to host a Year Up student for the internship phase of the program. This innovative financial model is centered on sustainability and scalability, as revenue from corporate partners covers more than half (nearly 60%) of the organization's direct service expenses.

Year Up raises philanthropic capital from individual donors.

Another funding stream is through Year Up Professional Resources, or YUPRO. YUPRO is a Public Benefit Corporation with the social purpose of providing career opportunities for the alumni of Year Up.

Program Developer/Owner

Roberto ZeledonChief Marketing OfficerYear Up45 Milk Street, 9th FloorBoston, MA 2109(855) 305-9995rzeledon@yearup.org www.yearup.org

Program Outcomes

  • Employment
  • Post Secondary Education

Program Specifics

Program Type

  • Employment - Vocational
  • Skills Training

Program Setting

  • Community

Continuum of Intervention

  • Selective Prevention

Program Goals

A training and internship program that helps young people with limited post-secondary education get high-quality jobs by learning to work with technology, developing employment skills, and obtaining internships.

Population Demographics

Economically disadvantaged urban young adults aged 18-24 with a high school degree or GED but limited post-secondary education and job prospects.

Target Population

Age

  • Adult
  • Early Adulthood (19-24)
  • Late Adolescence (15-18) - High School

Gender

  • Both

Race/Ethnicity

  • All

Race/Ethnicity/Gender Details

Differences in program impacts by gender were not statistically significant, but impacts did vary by race-ethnicity. Results indicated that the intervention increased the earnings of non-Hispanic blacks less than other groups. Overall, however, the program benefitted all groups.

Other Risk and Protective Factors

Lack of access to economic opportunity.

Risk/Protective Factor Domain

  • Individual

Risk/Protective Factors

Risk Factors

Protective Factors


*Risk/Protective Factor was significantly impacted by the program

See also: Year Up Logic Model (PDF)

Brief Description of the Program

Year Up provides six months of full-time training in the IT, business operations, sales and customer support, software development, and financial service sectors followed by six-month internships at major employers. The full-time program provides extensive supports and puts a heavy emphasis on the development of professional and technical skills. Students receive weekly stipends to help cover transportation and other program-related expenses. The program targets young adults with a high school diploma or equivalent who are motivated and who, with assistance, can overcome challenges and successfully enter careers in fast-growing technical occupations.

Description of the Program

Year Up provides six months of full-time training (e.g., IT and financial operations) followed by six-month internships at major employers. The full-time program provides extensive supports and puts a heavy emphasis on the development of professional and technical skills. Students receive weekly stipends to help cover transportation and other program-related expenses. The program targets young adults with a high school diploma or equivalent who are motivated and who, with assistance, can overcome challenges and successfully enter careers in fast-growing technical occupations.

In the first (Learning and Development) phase, students typically attend classes at Year Up from 8:30 AM until 3:30 PM four days a week, and for a half day on Wednesdays. The training addresses both occupation-specific and general skills. Fields include information technology (the most common emphasis), business operations, financial operations, software development, and sales and customer support. General skills courses focus on professional and business communication skills. Students gain experience in writing, giving presentations, interacting with clients and colleagues, and developing critical thinking skills.

In the second (Internship) phase, which heavily involves employers, the participants maintain full-time schedules, working at internship sites four and a half days a week and returning to Year Up typically on Wednesdays to share and process internship experiences, attend workshops, and plan post-program career transitions. Towards the end of internships, the emphasis on job search and placement intensifies. Active efforts to support job search and placement continue for up to four months after graduation.

Theoretical Rationale

Intensive and comprehensive interventions that address both general and occupation-specific skills, use work-based learning, and fully engage employers can help to overcome the barriers that nontraditional students often face. Assistance can give young adults without post-secondary credentials the opportunities to access professional careers and further education.

Theoretical Orientation

  • Skill Oriented

Brief Evaluation Methodology

Study 1 (Fein and Hamadyk, 2018, Fein et al., 2021): After recruiting a sample of 2,544 young adults located near eight program offices, local staff randomly assigned participants at a 2:1 ratio to the intervention (N = 1,669) and control groups (N = 875). Survey data of participants were collected at 18 months (Fein & Hamadyk, 2018), three years (Fein et al., 2021), and six years (Fein & Dastrop, 2022) after randomization. Administrative records were also collected that followed participants for up to three years (Fein & Hamadyk, 2018), five years (Fein et al., 2021), and seven years (Fein & Dastrop, 2022) after randomization. The primary measures focused on earnings. Secondary and exploratory measures included employment, college enrollment, financial hardship, marriage and childbearing, and personality traits.

Study 2 (Roder and Elliott, 2011, 2014) recruited 195 youths (ages 18-24) in three Eastern cities in the United States for the study and randomly assigned them to the program group or a no-treatment control group. The youths were followed for three years after the end of the program. The analysis examined self-reported measures of earnings, employment, and college attendance.

Outcomes (Brief, over all studies)

Study 1 (Fein and Hamadyk, 2018; Fein et al., 2021; Fein & Dastrop, 2022): The program significantly improved the primary outcome of earnings in the first three years after program intake, and these impacts persisted undiminished to the end of the five-year follow-up period. In addition, employment (i.e., from more hours worked and higher wages, rather than higher rates of employment) was significantly higher for participants in the treatment compared to the control group 3 years after program uptake. Meanwhile, five years after baseline, employment rates were significantly higher for treatment compared to control participants. It also significantly improved measures of college enrollment for up to three years after program uptake, but these effects were not maintained through the end of the five-year follow-up. At six and seven years after baseline, the intervention group continued to show significantly higher earnings, better jobs, and greater financial security than the control group. There was little benefit for measures of employment, education, psycho-social well-being, family formation, or self-assessed health.

Study 2 (Roder and Elliott, 2011, 2014): The program significantly improved total earnings, hourly wages, full-time employment, and jobs in IT and finance for the intervention group relative to the control group. Overall employment did not differ significantly across conditions, and college attendance was higher for the control group than the intervention group.

Outcomes

Study 1 (Fein and Hamadyk, 2018; Fein et al., 2021; Fein & Dastrop, 2022): Compared to the control group, the intervention group showed significantly more

  • employment (3 and 5 years after program uptake)
  • average quarterly earnings (3, 5, and 7 years after program uptake)
  • college enrollment (3 years after program uptake)

Risk and Protective Factors

  • increased health insurance rates (3 years after program uptake)
  • reduced financial hardship (3, 5, and 6  years after program uptake)
  • decreased public benefit (5 years after program uptake)
  • improved living arrangements (5 years after program uptake)
  • reported life changes (5 years after program uptake)
  • improved career pathways employment and self-assessed career development (5 years after program uptake)

Study 2 (Roder and Elliott, 2011, 2014): Compared to the control group, the intervention group showed significantly more

  • full-time employment and IT and finance jobs
  • earnings and hourly wages

Risk and Protective Factors

  • college financial assistance and interest in attending college

Mediating Effects

Not examined.

Effect Size

Study 1 (Fein & Hamadyk, 2018) reported only a few effect sizes, most notably the values of .28, .18, and .34 for measures of self-assessed career progress.

Generalizability

The study used a large sample from eight major cities across the United States, but the sample was selective in the recruitment of young adults expected to benefit most from the program.

Potential Limitations

Study 1 (Fein and Hamadyk, 2018; Fein et al., 2021; Fein & Dastrop, 2022)

  • Little information on reliability of survey scales and some gaps in measures of employment and college enrollment

Study 2 (Roder and Elliott, 2014)

  • No controls for baseline outcomes
  • No tests for baseline equivalence
  • Evidence of differential attrition
  • Long-term results compromised by entrance into the program of control participants

Notes

Study 1 had a pre-registered analysis plan specifying key hypotheses and outcome measures: Judkins, D., D. Fein, & L. Buron. 2018. Analysis Plan for the PACE Intermediate (Three-Year) Follow-Up Study. OPRE Report # 2018-95. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/analysis-plan-for-the-pace-intermediate-three-year-follow-up-study. The research team subsequently assessed data quality, refined the plan, and publicly registered it on the Open Science Framework website (https://osf.io/2gxuh/?pid=wcus9).

Endorsements

Blueprints: Promising
Social Programs that Work:Suggestive Tier

Program Information Contact

Roberto Zeledon
Chief Marketing Officer
Year Up
45 Milk Street, 9th Floor
Boston MA 2109
Phone: (855) 305-9995
Email: rzeledon@yearup.org
Website: www.yearup.org

References

Study 1

Certified Fein, D., & Hamadyk, J. (2018). Bridging the opportunity divide for low-income youth: Implementation and early impacts of the Year Up program, OPRE Report #2018-65 (and Appendices). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Certified

Fein, D., Dastrup, S., & Burnett, K. (2021). Still bridging the opportunity divide for low-income youth: Year Up's longer-term impacts, OPRE Report 2021-56. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Certified

Judkins, D., Walton, D., Durham, G., Litwok, D., & Dastrup, S. (2021). Still Bridging the Opportunity Divide for Low-Income Youth: Year Up's Longer-Term Impacts, Technical Appendices, OPRE Report 2021-56. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Certified

Fein, D., & Dastrup, S. (2022). Benefits that last: Long-term impact and cost-benefit findings for Year Up, OPRE Report 2022-77. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Certified

Judkins, D., Roessel, E., & Durham, G. (2022). Career pathways long-term outcomes study: Appendices for PACE six-year impact reports. OPRE Report 2022-69. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Study 2

Roder, A., & Elliott, M. (2011). A promising start: Year Up's initial impacts on low-income young adults' careers. Economic Mobility Corporation: New York, NY.

Roder, A., & Elliott, M. (2014). Sustained gains: Year Up's continued impact on young adults' earnings. Economic Mobility Corporation: New York, NY.

Study 1

Design:

Recruitment: Year Up's eight core offices recruited, screened, and selected 2,544 young adults ages 18-24 for the evaluation. To allow for experimental assignment, local admissions teams recruited three eligible applicants for every two open program seats. Each office enrolled at least two cohorts from January 2013 to August 2014. Study locations included Atlanta, the San Francisco Bay area, Boston, Chicago, Washington DC, New York City, Providence, and the Puget Sound area. The young adults eligible for the program had a high school diploma or equivalent and were judged by local staff as likely to benefit from the program. Many had recently experienced financial hardship and very few were employed or in school.

Assignment: Local program staff used an online lottery tool developed by the research team to randomly assign participants within each local site at a 2:1 ratio to the intervention and control groups. Intervention group members (N = 1,669) had the opportunity to participate in Year Up but were not required to enroll. Control group members (N = 875) could not participate in Year Up for three years but received a list of other services available in the community. About 3.4% of the control group joined the program during or after the embargo period but had only a small potential influence on the results, and the analyses kept them in the control group.

Assessments/Attrition: For Fein & Hamadyk (2018), participants completed a baseline survey and an 18-month follow-up survey (about 6 months after the end of the year-long program). The response rate for the follow-up survey was 78% for the treatment group and 73% for the control group. Item non-response among survey responders was under 4%, except for parental college attendance (6.0%), typical high school grades (7.2%), family income (9.5%), and expected near-term future work hours (6.0%).

Additional data from administrative records extended the follow-up period to more than one year after the program end. Data on new hires (from the National Directory of New Hires) accessed in April 2018 covered a period extending from "two quarters before random assignment to three years (12 quarters) after random assignment" (p. 35). Data on college enrollment accessed in April 2017 covered "nearly three years (11 quarters) after random assignment" (p. 35). Data from the administrative records were complete for 98% of the sample (Fein & Hamadyk, 2018).

For Fein et al. (2021), participants completed a baseline survey and a three-year follow-up survey (about 2 years after the end of the year-long program). The response rate for the three-year follow-up survey was 71% overall and the median response occurred at 39 months after random assignment in both the treatment and control groups. Similar to the earlier report, nonresponse rates were less than 5 percent except for parental college attendance (10%), race (9%), Hispanic origin (8%), family income (9%), receipt of WIC or SNAP (7%), and receipt of other public assistance (10%).

Data from a January 2020 extract of earnings data cover a period extending from two quarters before random assignment to five years (20 quarters) after random assignment for all sample members. Since administrative earnings records extend to mid-2019, they do not cover the economic downturn resulting from the coronavirus pandemic. Meanwhile, postsecondary administrative data from a July 2019 extract cover college enrollment outcomes for a five-year (20-quarter) follow-up period following random assignment. Consistent with the earlier report, data from the administrative records were complete for 98% of the sample (Jenkins et al., 2021, Appendix D.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022): Assessments came from a six-year survey (median of 70 months after assignment) and from seven years of records. The response rate was 66% for the survey and 100% for the records.

Sample: All sample members were aged 18-24. A majority of sample members identified as non-Hispanic Black (54%) or Hispanic of any race (31%), while 5.5% identified as White, non-Hispanic and 8.8% identified as "any race, non-Hispanic". Men (59%) outnumbered women (41%), though women were well-represented for a tech-focused training program. Most sample members (68%) lived with their parents, and few (9%) had children. Many struggled in high school: 40% reported usual grades of C or below, and only 10% reported usually receiving As. About half had attended some college. Nearly two-thirds (63%) were in families with annual incomes below $30,000.

Measures: The outcome measures, more than 45 in total, came from two independent sources, survey self-reports, and administrative records. Data on new hires came from extracting records from the National Directory of New Hires (NDNH). The database covered most employment but missed self-employment and independent contracting. Data on college enrollment came from the National Student Clearinghouse (NSC). The database included nearly all private and public 4-year schools and public 2-year schools. Coverage was less complete for private 2-year schools and for-profit 2-year and 4-year schools. The study provided little information on the validity of the survey measures, but most appeared straightforward.

Earnings and Employment: Most measures of earnings came from administrative records. The single measure identified by the researchers as primary was the average quarterly earnings in the 6th and 7th quarters after random assignment (Fein & Hamadyk, 2018) and five years (20 quarters) after random assignment for Fein et al. (2021). Four secondary measures included the annual earnings in years 1, 2, 3, and 5 and employment (Fein & Hamadyk, 2018). A total of 17 measures came from the survey. There were three indicators of career pathways employment (working for $15/hour or more, in a job requiring mid-level skills, and in a program-targeted occupation) and three measures of self-assessed career progress (perceived career progress, confidence in career knowledge, and access to career networks). The surveys also provided 11 measures of earnings, hours worked, and hourly wages.

Post-Secondary Education: The seven measures from administrative records covered quarterly college enrollment, quarterly cumulative college enrollment, and yearly college enrollment. The five measures from the survey included credits received and credentials awarded from college, other institutions, licensing, and any source. The measures in Fein & Hamadyk (2018) were described as exploratory because they were not expected to show effects until later assessments (see the six- and seven-year measures below).

Other Outcomes: The 11 measures from surveys reported in Fein & Hamadyk (2018) included dependence on public assistance, financial hardship, access to health insurance, marriage and childbearing, grit, savvy, and core self-evaluation. These measures were described as exploratory because they were not expected to show effects until later, may have effects in unknown directions, or were of unknown validity. In addition to details on employment and related financial circumstances, the survey reported in Fein et al. (2021) assessed educational history, psycho-social skills, and other outcomes.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022). This report examined more than 80 outcome measures but also sought to minimize false positives by distinguishing between confirmatory, secondary, and exploratory outcomes. The single prespecified confirmatory outcome was average quarterly earnings in the 23rd and 24th follow-up quarters. The secondary outcomes included other measures of quarterly earnings, employment, and occupation. The exploratory outcomes included measures of employment and earnings in a variety of years and quarters; college enrollment and credential receipt; various aspects of financial well-being; and health, stress, living arrangements, and childbearing.

The measures came from a survey completed roughly six years after random assignment and from administrative records drawn from the National Directory of New Hires and the National Student Clearinghouse for a seven-year period. As in previous reports, this report provided little information on the validity of the survey or administrative measures, but most appeared straightforward.

Analysis: Given the 28 baseline covariates, the analysis reported in Fein & Hamadyk (2018) used a multi-step control strategy. It first regressed each outcome on a set of baseline variables for the control group. It next used the regression results to calculate predicted outcome values for the two conditions. The final steps calculated the average difference between actual and predicted values for each condition and, to assess the program impact, subtracted the average condition differences from one another. It appears from Appendix A.2 of Fein & Hamadyk (2018) that the covariates included baseline outcomes.

A technical report published by Judkins et al. (2021) describes the approach to covariate control for the Fein et al. (2021) study, which included some procedural changes from the Fein & Hamadyk (2018) report. The authors selected covariates using the "least absolute shrinkage and selection operator" (LASSO) technique with "10-fold cross-validation" (see page 9 of Judkins et al., 2021 for more details). A set of covariates across three domains were selected. The three domains included: (1) analyses of employment and earnings outcomes that are conducted on the dataset of merged data from the three-year follow-up survey and the National Directory of New Hires (NDNH); (2) analyses of education outcomes (whether based on the survey, National Student Clearinghouse, or local or state college records); and (3) analyses of all other outcomes (most of which concern personal and family well-being and economic independence). The pool of potential covariates was the same for all three domains with one important exception: indicators of pre-baseline earnings based on NDNH data were only allowed in analyses of NDNH-based outcomes. Exhibit A-3 (p. 12-13; Judkins et al., 2021) shows the covariates selected with the LASSO procedure, which appeared to include baseline outcomes.

For both reports (Fein & Hamadyk, 2018; Fein et al., 2021), tests relied on different significance levels, depending on the outcomes. The analyses of the researcher-identified primary and secondary outcomes, called the confirmatory and secondary analyses, used one-tailed tests but were limited in number to minimize the problem of multiple comparisons. The more extensive exploratory analyses used two-tailed tests. But the tables present sufficient information to apply two-tailed tests at .05 to all results.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022). The authors reported regression-adjusted means that controlled for baseline covariates (a subset selected by optimization procedures from a list of 20+ measures contained in Appendix A of Judkins et al., 2022). The tests for differences between the means were one-sided for the confirmatory and secondary outcomes and two-sided for the exploratory outcomes, but the reported probabilities allow for application of two-sided tests to all results. The authors also noted that the reported results did not adjust standard errors to account for variation in effects across offices. Checks estimating standard errors with a cross-office variance term for key outcomes found that all earnings impacts remained highly significant.

Intent-to-Treat: The study included participants assigned to the intervention group but who did not enroll in the program. For those lost to follow-up, the analyses applied weights to adjust for differential nonresponse. In addition, missing data for item non-response was imputed using a weighted "hot deck" matching procedure (Appendix A.1 of Fein & Hamadyk, 2018; Appendix A.1 of Judkins et al., 2021).

Fein & Dastrop (2022) kept participants in their originally assigned condition, regardless of actual participation or non-participation in the program. For the survey data, the analysis used two procedures described in Appendix B of Judkins et al. (2022) to deal with missing data. The first replaced missing items for otherwise complete data; the specifics depended on the outcome measure and the extent of missing data. The second applied weights to adjust for differential attrition or non-response; it did not impute missing data but gave additional weight to the participants most similar to dropouts.

Outcomes

Implementation Fidelity: Nearly all treatment group members (96%) accepted the offer to enroll and began the program. About 81% of the sample and 85% of enrollees completed the six-month learning and development phase. Nearly all those completing the learning and development phase received internships, and 75% of the treatment group members finished the program.

A detailed implementation analysis found that Year Up generally implemented its services with high fidelity to the program model. Overall, treatment group members received substantially higher levels of training and support than control group members. However, the eight offices varied in completion of the program components (range = 61% to 82%).

Baseline Equivalence: Using all 2,544 randomized participants, tests for condition means on 28 baseline measures of sociodemographic characteristics, outcomes, and covariates produced no significant differences (p < .05).

Differential Attrition: The conditions had roughly similar survey response rates of 78% for the treatment group and 73% for the control group (Fein & Hamadyk, 2018). Exhibit D-3 compared the condition means for the baseline measures using all participants and survey respondents. There were no significant differences at the .05 level for the 28 baseline measures in either the full or the analysis sample. The analyses still used a weighting procedure to adjust for nonresponse (see Appendix D.3 of Fein & Hamadyk, 2018). Further, checks in Exhibit D-4 (Fein & Hamadyk, 2018) demonstrated that, for outcome measures available for the full sample, analysis of the subsample of survey respondents produced results similar to those for the full sample. Of note, the primary outcomes relating to earnings were assessed via archival data which had a much higher retention rate.

For the 3-year follow-up (Fein et al., 2021), the conditions had slightly different response rates of 74% in the treatment group and 67% in the control group. Similarly, Fein et al. (2021) explain that: "Analyses of survey data incorporated weights to adjust for differential nonresponse across groups of study participants" (p. 25). Details are listed in Appendix B.2 of Judkins et al., (2021). And consistent with the earlier report, the primary outcome (i.e., earnings) was evaluated using administrative data with a high retention rate.

For the six-year survey, Fein & Dastrop (2022) reported only that the treatment and control groups had 68% and 62% retention, respectively. The analysis of non-response in Judkins et al. (2022) used complete administrative data to compare condition differences in the outcomes for the full sample to differences for the subsample of survey respondents. It found substantial evidence of bias. But, as described in Appendix B.2 of Judkins et al. (2021), the use of weighting helped adjust for differential non-response.

Posttest: Three years after baseline, Fein & Hamadyk (2018) found that approximately 30 of the 45 outcomes showed significant effects, including the primary and secondary measures of earnings and employment.

Earnings and Employment: The primary outcome - average quarterly earnings in the 6th and 7th quarters after random assignment - was significantly higher for the intervention group than the control group (p < .05, two-tailed test). More detail in Exhibit 6-2 of Fein & Hamadyk (2018) revealed significantly lower earnings in year 1, when the program was ongoing and intervention participants were receiving stipends, but significantly higher earnings in years 2 and 3. The same pattern emerged for employment, with significant positive effects in later years. Tests for the survey measures showed significant intervention effects in 16 of 17 outcomes. The authors noted (Fein & Hamadyk, 2018, p. 7) that the increase in earnings came from more hours worked and higher wages rather than from higher rates of employment.

Post-Secondary Education: Tests using the seven administrative measures showed generally positive effects three years after baseline. The intervention group had significantly higher college enrollment in year 1, when the program was ongoing, and significantly lower enrollment in year 2. However, when measuring cumulative months of full-time enrollment (Exhibit 6-7 of Fein & Hamadyk, 2018), the intervention group had significantly higher scores than the control group in all years. Tests for the five survey measures found three significant effects on credits received, a licensing credential received, and a credential received from any source (18 months after baseline).

Other Outcomes: Tests for 11 self-reported outcomes found five significant effects of the intervention 18 months after baseline: having health insurance, receiving less cash or in-kind support, experiencing less financial hardship, perceived savvy, and core self-evaluation (Fein & Hamadyk, 2018).

Long-Term: The effects of the program on earnings and cumulative college enrollment were maintained in years 2, 3, 5, 6, and 7 -- well after the 1-year follow-up period.

Earnings and Employment: Fein et al. (2021) found the large positive earnings impacts persisted undiminished to the end of the five-year follow-up period. That is, the impact on average quarterly earnings in follow-up Quarters 12 and 13 (the sole confirmatory outcome for Fein et al., 2021) was large, at $1,857, and statistically significant. Average earnings were $6,782 for treatment group members and $4,925 for control group members. Impacts of about $2,000 per quarter extended to the end of the five-year follow-up period (Exhibit ES-1 in Fein et al., 2021). The result indicates that impacts persisted virtually undiminished after reaching $1,895 in Quarters 6 and 7-the confirmatory outcome in the short-term report (Fein and Hamadyk 2018).

Secondary analyses estimated impacts on annual earnings for up to five years after random assignment. Fein et al. (2021) report that earnings impacts turn positive and large in Year 2 ($5,267). The size of the impact appears to grow with each additional year of follow-up (reaching $7,830 in Year 5), but statistical tests do not find the increases to be statistically significant (Exhibit 3-2).

Consistent with the earlier report (Fein & Hamadyk, 2018), Fein et al (2021) report survey results that suggest the increase in earnings came from more hours worked and higher wages rather than from higher rates of employment. In contrast to the survey, however, administrative records show small positive employment impacts in a number of quarters during follow-up years 4 and 5 (Exhibit 3-4, Fein et al., 2021). The impacts are small (only 3 percentage points) but statistically significant.

Meanwhile, survey results show impacts in favor of treatment on 5 of 6 indicators of career pathways employment and 2 of 3 indices of self-assessed career development.

Post-Secondary Education: From Year 3 on, college enrollment rates (an exploratory outcome) were virtually identical in the treatment and control groups. At the time of the three-year follow-up survey, Fein et al. (2021) found few members of either group had earned an associate degree or higher, although the fraction was lower for treatment (4 percent) than control (8 percent) group members. Administrative records show a smaller, but statistically significant, two percentage point reduction in degree receipt (see Supplemental Exhibit S-4 of Fein et al., 2021). In addition, whereas in the 18-month survey (Fein & Hamadyk, 2018), treatment group members reported receiving an average of four more credits than control group members did (12 and 8 credits, respectively), the difference had disappeared by the time of the three-year follow-up (see Exhibit 4-2 of Fein et al., 2021): treatment and control group members reported receiving a similar number, averaging 16 and 18 credits, respectively.

The significant reduction in associate degree attainment could be construed as an iatrogenic effect. As an alternative explanation, the authors posit that, because the treatment condition participants were making more money than the control condition, they were less motivated to pursue further education.

Other Outcomes: Results show signs of impacts in secondary "other" outcomes, but effects were not widespread. For example, findings from the three-year survey (Fein et al., 2021) show reductions in public benefit receipt, debt, and financial hardship. Average household income did not increase, which the authors conclude could be because increased earnings both reduced some households' public benefit eligibility and allowed some young adults to live independently. The treatment had a few small effects on living arrangements, but no impacts on two psycho-social outcomes (academic self-confidence and future educational aspirations). In addition, treatment had a small reduction in the average number of reported life challenges but had no impact on perceived stress, social support, core self-evaluation, and grit.

Subgroup findings (exploratory outcomes): Fein et al. (2021) found that 5-year impacts on quarterly earnings were larger for young adults receiving mostly A's or B's in high school ($2,178) than for those receiving mostly C's or below ($1,388), and larger for those who identify as White ($2,931) than for those who identify as Black ($1,570). Variation in impacts across the eight local offices (from $956 to $4,811, with most offices in the $1,348 to $1,807 range) also suggests that local conditions might matter.

Six- and Seven-Year Follow-Up (Fein & Dastrop, 2022). The confirmatory outcome of earnings in quarters 23 and 24 was significantly higher ($1,895) for the treatment group than the control group. Nearly all of the secondary outcomes, which related to earnings for other quarters, occupation, and financial well-being, showed significant program benefits; the treatment group did significantly better than the control group on 16 of 17 tests (two-sided, p < .05). Some of the exploratory outcomes showed significant effects; the treatment group did better than the control group on 22 of 66 tests.

The most consistent results occurred for measures of earnings at seven years, and job quality and financial status at six years. Generally weak results for measures of employment and education suggested that the higher earnings brought about by the program came from better jobs rather than higher employment or education. Null results occurred for exploratory outcomes relating to psycho-social well-being, family formation, or self-assessed health.

Exploratory moderation analyses, based on a large number of tests, found that the program most benefited participants with better high school grades and some college, older ages, and fewer depressive symptoms.

The COVID pandemic began during data gathering, and checks found that program participants enjoyed modest protection from lower earnings and employment during the pandemic. For the confirmatory outcome, the pandemic reduced average quarterly earnings in follow-up quarters 23-24 for all participants but did not change the advantage of the treatment group over the control group.

Study 2

Evaluation Methodology

Design:

Recruitment: Program staff in three cities, Boston, New York City, and Providence, Rhode Island, invited eligible young people to take part in the program. The recruitment in 2007 took place shortly before the recession of 2008. Eligible youths came from low-income backgrounds, were ages 18-24, had a high school degree or GED, and showed strong motivation to improve their opportunities with an intensive, full-year training. A total of 195 agreed to join the study.

Assignment: The research team randomly assigned the participants based on the capacity of the programs in the three cities. Of the 195 youths in the sample, 135 were randomly selected for the intervention group and 60 were randomly selected for the control group. Control participants were told that they were being placed on a waiting list, could re-apply to the program after ten months, and were allowed to pursue employment or postsecondary education or training elsewhere. However, 29% of the control group members eventually participated in the program, which affected the long-term comparisons.

Assessments/Attrition: Roder and Elliott (2011) mentioned that the last survey occurred between 24 and 30 months after random assignment (or 12 and 18 months after the program end). Figures 2 and 3, however, reported quarterly data over two years, suggesting either ongoing surveys after randomization or retrospective reports. The follow-up survey included 164 youths, with completion rates of 84% overall, 89% for the intervention group, and 73% for the control group.

Roder and Elliott (2014) followed the participants for an additional two years, three years after the end of the program. The response rates of the follow-up survey were 73% overall, 76% for the intervention group, and 68% for the control group.

Sample:

Of the 164 young people in the analysis sample, 50% were African American and 34% were Latino. More than half (57%) were male, 81% lived with a parent or guardian, and 18% lived in public housing. All had at least a GED or high school degree, but 8% had a criminal conviction and English was a second language for 15%.

Measures:

For the one-year follow-up (Roder and Elliott, 2011), program staff collected data on students who graduated from the program, while a survey firm conducted interviews with members of the control group and with members of the intervention group who dropped out of the program. For the long-term follow-up (Roder and Elliott, 2014), a single survey firm collected data on all available participants. The study provided no details on the reliability or validity of the self-reports from participants, but the measures were straightforward. The outcomes included:

  • Total earnings
  • Employment, including full-time employment and number of jobs
  • Hourly wages
  • Job type (e.g., IT and finance)
  • Employer-provided benefits
  • College attendance

Analysis:

The analyses simply compared condition means and percentages, without any controls.

Intent-to-Treat: The analyses included all participants with follow-up data, regardless of whether or not they ever attended or graduated from the program. For the long-term data, after many of those initially assigned to the control group had entered and completed the program, treatment-on-the-treated results were reported along with ITT results.

Outcomes

Implementation Fidelity:

The study did not measure the quality of implementation but reported that, of the 120 intervention group members, 90% attended part of the program and 64% graduated on time in July 2008.

Baseline Equivalence:

The study did not examine baseline equivalence of the conditions.

Differential Attrition:

For the one-year follow-up (Roder and Elliott, 2011), the conditions differed substantially in attrition rates - 11% for the intervention group and 27% for the control group. Footnote 2 examined the significance of differences in the attrition rates between the intervention and control group members by baseline characteristics. There were significant differences on three factors. Attrition was higher among control group members than among intervention group members for those ages 18-19, African Americans, and those with English as the primary language.

For the long-term follow-up (Roder and Elliott, 2014), the conditions differed moderately in attrition rates - 24% for the intervention group and 32% for the control group. The appendix examined the significance of differences in the attrition rates between the intervention and control group members by baseline characteristics. There were significant differences on two factors. Attrition was higher among control group members than among intervention group members for those ages 18 to 19 and those with a GED rather than a high school diploma.

Posttest:

The results in Roder and Elliott (2011) covered the quarterly periods during the one-year program and the one-year post-program period. During the program period, when the intervention group was going through the training, the control group had higher employment and earnings. However, during the second year after random assignment, relative to the control group, the intervention group showed significantly more:

  • fulltime employment (but not overall employment),
  • annual earnings in the second and third quarter,
  • earnings per hour, and
  • more jobs in information technology and investment operations.

Long-Term:

The results in Roder and Elliott (2014) covered all four years of the study but focused on the three years after the program end. The comparisons were limited by the entrance of 29% of control participants into the program, which likely lowered employment and earnings of the control group during the training and raised employment and earnings afterward. As a point of comparison, intervention participants made approximately $2000 less per quarter, compared to control participants, while completing the intervention.

Figure A1 in the appendix lists the ITT effects for 25 outcomes, with only three reaching significance at the .05 level, while the text listed some additional significant effects. Relative to the control group, the intervention group showed significantly more:

  • holders of a single job (but not higher employment rates or hours worked),
  • earnings in years 2-3 after the program but not year 4,
  • jobs in information technology and investment operations,
  • recipients of educational financial assistance, such as loans, grants, or scholarships, and
  • interest in attending college (among those not attending college).

Contact

Blueprints for Healthy Youth Development
University of Colorado Boulder
Institute of Behavioral Science
UCB 483, Boulder, CO 80309

Email: blueprints@colorado.edu

Sign up for Newsletter

If you are interested in staying connected with the work conducted by Blueprints, please share your email to receive quarterly updates.

Blueprints for Healthy Youth Development is
currently funded by Arnold Ventures (formerly the Laura and John Arnold Foundation) and historically has received funding from the Annie E. Casey Foundation and the Office of Juvenile Justice and Delinquency Prevention.