A one-semester pre-enrollment program of noncredit developmental courses and support services to help entering community college students in need of academic remediation prepare for college-level coursework.
Blueprints: Promising
Mia Simon
University Director, CUNY Start and Math Start
Mia.Simon@cuny.edu
Website: www.cuny.edu/cunystart
The City University of New York (CUNY)
CUNY Start is a one-semester program designed for associate degree-seeking students, assessed as having significant remedial needs in math and/or English, to help them meet the City University of New York's (CUNY) proficiency milestones and prepare for college credit courses. Students who enroll in CUNY Start defer matriculation for up to a semester to engage in time-intensive math and English developmental courses and attend advisory workshops. They pay only $75 to enroll in the program thereby saving their financial aid dollars for when they are better prepared to succeed in college.
CUNY Start is a one-semester program designed for associate degree-seeking students, assessed as having significant remedial needs in math and/or English, to help them meet the City University of New York's (CUNY) proficiency milestones and prepare for college credit courses. Students who enroll in CUNY Start defer matriculation for up to a semester to engage in time-intensive math and English developmental courses and attend advisory workshops. They pay only $75 to enroll in the program thereby saving their financial aid dollars for when they are better prepared to succeed in college. The program provides:
The program requires a substantial time commitment from the student, with 26.5 hours of class time per week for the full-time program (focusing on both math AND English) or 13.5 hours of class time per week for the part-time program (focusing on either math OR English). Students enter as part of a cohort group and take all their classes together.
CUNY Start provides a well-defined next-step pathway to CUNY's Accelerated Study in Associate Programs (ASAP), which is certified as Model Plus by Blueprints. Weiss et al. (2021) evaluated CUNY Start within the context of the availability of ASAP to the students as they matriculated into college. It may be that CUNY Start works best for students who also participate in the ASAP program. CUNY Start operates at nine CUNY colleges.
Primary Evidence Base for Certification
Study 1
Weiss et al. (2021) found that, relative to the usual-services control students, the intervention students had significantly more
Primary Evidence Base for Certification
Of the three studies that Blueprints has reviewed, one study (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). In addition, Study 1 was conducted by independent evaluators.
Study 1
Weiss et al. (2021) evaluated the program with a randomized controlled trial. Students entering four City University of New York (CUNY) colleges with remedial academic needs were assigned to either the intervention condition (n = 2,997) or a usual-services control group (n = 838). Data on course readiness, credit completion, and certificate/degree received were collected over the following three years.
Study 1
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (2021). An on-ramp to student success: A randomized controlled trial evaluation of a developmental education reform at the City University of New York. MDRC.
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In subgroup analyses, Study 1 (Weiss et al., 2021) found few differences in program effects across race and ethnic groups.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
The analytic sample in Study 1 (Weiss et al., 2021) included more women than men. Black (26%) and Hispanic (36%) students made up most of the sample, and a substantial proportion (37%) of the sample had a native language other than English.
CUNY Start offers college administrators and faculty an opportunity to visit and explore ways to implement the program at their own institutions.
A TASTE OF CUNY START
A Taste of CUNY Start is a three-and-half-hour session that provides an overview of CUNY Start's unique program model, including the English, math and advisory curriculum. Participants are provided with an overview of CUNY Start and the program's role in CUNY's broader developmental education reform efforts, various program implementation models (i.e., Math Start and English Start), as well as opportunities to speak with a range of program staff including professional developers, advisors and program administrators. This session can be scheduled either in person or virtually.
DESIGN STUDIO
CUNY Start's Design Studio is a one-and-a-half-day event that invites attendees to observe various elements of the CUNY Start program in-person. From program administration to curriculum and instruction, participants have the opportunity to visit English and math classes, learn more about advisement sessions, and interact with members of the CUNY Start professional development team, central office and campus administrators, teachers, advisors, and students.
SUMMER INSTITUTE
The CUNY Start Summer Institute is a four-day intensive training for institutions preparing to implement elements of CUNY Start within six months to a year of their Design Studio visit. Participants spend time unpacking the curriculum and pedagogy of CUNY Start, learning more about how the program operates within a college, exploring how data is used to inform program management and evaluation, and meeting with a range of administrators and staff.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
The City University of New York (CUNY)New York, New York United States
A one-semester pre-enrollment program of noncredit developmental courses and support services to help entering community college students in need of academic remediation prepare for college-level coursework.
Students entering college with academic remediation needs and often from disadvantaged backgrounds.
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In subgroup analyses, Study 1 (Weiss et al., 2021) found few differences in program effects across race and ethnic groups.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
The analytic sample in Study 1 (Weiss et al., 2021) included more women than men. Black (26%) and Hispanic (36%) students made up most of the sample, and a substantial proportion (37%) of the sample had a native language other than English.
Gaps in academic preparation for college
*Risk/Protective Factor was significantly impacted by the program
CUNY Start is a one-semester program designed for associate degree-seeking students, assessed as having significant remedial needs in math and/or English, to help them meet the City University of New York's (CUNY) proficiency milestones and prepare for college credit courses. Students who enroll in CUNY Start defer matriculation for up to a semester to engage in time-intensive math and English developmental courses and attend advisory workshops. They pay only $75 to enroll in the program thereby saving their financial aid dollars for when they are better prepared to succeed in college.
CUNY Start is a one-semester program designed for associate degree-seeking students, assessed as having significant remedial needs in math and/or English, to help them meet the City University of New York's (CUNY) proficiency milestones and prepare for college credit courses. Students who enroll in CUNY Start defer matriculation for up to a semester to engage in time-intensive math and English developmental courses and attend advisory workshops. They pay only $75 to enroll in the program thereby saving their financial aid dollars for when they are better prepared to succeed in college. The program provides:
The program requires a substantial time commitment from the student, with 26.5 hours of class time per week for the full-time program (focusing on both math AND English) or 13.5 hours of class time per week for the part-time program (focusing on either math OR English). Students enter as part of a cohort group and take all their classes together.
CUNY Start provides a well-defined next-step pathway to CUNY's Accelerated Study in Associate Programs (ASAP), which is certified as Model Plus by Blueprints. Weiss et al. (2021) evaluated CUNY Start within the context of the availability of ASAP to the students as they matriculated into college. It may be that CUNY Start works best for students who also participate in the ASAP program. CUNY Start operates at nine CUNY colleges.
CUNY Start's theory of change posits that students with substantial developmental education requirements are best served through a time-intensive effort to build academic preparedness and college skills before entering college. It is designed to build academic preparedness and college skills before matriculation. The program is intended to make students more engaged in their coursework, help them view themselves as learners, help them make greater connections with their peers, give them the support they need to succeed, and prepare them for college-level work. The program's low cost to students is expected to make it easier for them to participate, and the fact that the program is provided before students' matriculate allows them to preserve their financial aid for future courses.
Primary Evidence Base for Certification
Of the three studies that Blueprints has reviewed, one study (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). In addition, Study 1 was conducted by independent evaluators.
Study 1
Weiss et al. (2021) evaluated the program with a randomized controlled trial. Students entering four City University of New York (CUNY) colleges with remedial academic needs were assigned to either the intervention condition (n = 2,997) or a usual-services control group (n = 838). Data on course readiness, credit completion, and certificate/degree received were collected over the following three years.
Primary Evidence Base for Certification
Study 1
Weiss et al. (2021) found that the intervention group students had significantly more college-ready subject areas, rates of certificate completion, and college enrollment than the usual-services control students.
Primary Evidence Base for Certification
Study 1
Weiss et al. (2021) found that, relative to the usual-services control students, the intervention students had significantly more
A partial analysis of mediation in Study 1 (Weiss et al., 2021) found that higher participation of the intervention group than the control group in another program, ASAP, accounted for most of the relationship between CUNY Start and the key outcomes.
Study 1 did not present effect sizes.
One study (Study 1) meets Blueprints standards or high-quality methods with strong evidence of program impact (i.e., "certified" by Blueprints): Study 1 (Weiss et al., 2021). The study sample included college students with remedial educational needs.
Study 1 took place at the City University of New York and compared the treatment group with a services-as-usual control group.
Additional Studies (not certified by Blueprints)
Study 2 (Webber, 2018)
Webber, A. J. (2018). Starting to succeed: The impact of CUNY Start on academic momentum. New York: City University of New York, Office of Research, Evaluation, and Program Support. Available online: https://www.cuny.edu/wp-content/uploads/sites/4/media-assets/gateway_brief_final.pdf
Study 3 (Polon & Haralampoudis, 2021)
Polon, I., & Haralampoudis, A. (2021). Academic momentum after CUNY Start a quasi-experimental study of post matriculation outcomes. New York: City University of New York, Office of Research, Evaluation, and Program Support. Available online: https://www1.cuny.edu/sites/cunystart/wp-content/uploads/sites/51/2022/01/CS_PSM_20211105_FINAL.pdf
Blueprints: Promising
Mia Simon
University Director, CUNY Start and Math Start
Mia.Simon@cuny.edu
Website: www.cuny.edu/cunystart
Certified
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (2021). An on-ramp to student success: A randomized controlled trial evaluation of a developmental education reform at the City University of New York. MDRC.
Scrivener, S., Gupta, H., Weiss, M. J., Cohen, B., Cormier, M. S., & Brathwaite, J. (2018). Becoming college-ready: Early findings from a CUNY Start evaluation. New York: MDRC. Available online: https://www.mdrc.org/sites/default/files/CUNY_START_Interim_Report_FINAL_0.pdf.
Webber, A. J. (2018). Starting to succeed: The impact of CUNY Start on academic momentum. New York: City University of New York, Office of Research, Evaluation, and Program Support. Available online: https://www.cuny.edu/wp-content/uploads/sites/4/media-assets/gateway_brief_final.pdf.
Polon, I., & Haralampoudis, A. (2021). Academic momentum after CUNY Start a quasi-experimental study of post matriculation outcomes. New York: City University of New York, Office of Research, Evaluation, and Program Support. Available online: https://www1.cuny.edu/sites/cunystart/wp-content/uploads/sites/51/2022/01/CS_PSM_20211105_FINAL.pdf
Scrivener et al. (2018) report results only for the semester in which the program is being delivered. Unless otherwise mentioned, the description of the study refers to the full period of evaluation examined in Weiss et al. (2021)
Summary
Weiss et al. (2021) evaluated the program with a randomized controlled trial. Students entering four City University of New York (CUNY) colleges with remedial academic needs were assigned to either the intervention condition (n = 2,997) or a usual-services control group (n = 838). Data on course readiness, credit completion, and certificate/degree received were collected over the following three years.
Weiss et al. (2021) found that, relative to the usual-services control students, the intervention students had significantly more
Evaluation Methodology
Design:
Recruitment: The sample came from four large CUNY colleges in New York City that had at least two years experience with the program and sufficient program infrastructure to continue its operation. Eligible student participants had remedial educational needs for college and volunteered for the program. A total of 4,434 students applied, and 3,873 students agreed to participate and could be randomly assigned (excluding the 13% of applicants who, for a variety of reasons, were guaranteed a spot in program). Three cohorts of students took part in the study, starting in spring 2015, fall 2015, or spring 2016 semesters.
Assignment: The study randomly assigned the 3,873 students to the program or a usual-services control group within blocks defined by cohort, intended college, and intended program (full- or part-time). Assignment was done in real time via a web portal controlled by the research team. The probability of being assigned to the program group was 78%, on average. The uneven random assignment ratio was, by design, chosen to help staff fill all program slots. After assignment, 38 participants were dropped (0.8% of program group students and 1.6% of control group students) because they asked to withdraw or did not submit signed informed consent forms. The analytic sample thus consisted of 3,835 students, including 2,997 program group students and 838 control group students.
However, the authors emphasized that CUNY had another program, Accelerated Study in Associate Programs, or ASAP (a model plus Blueprints program), that made it hard to isolate the effects of CUNY Start and may be seen as both a confound and a mediator.
Assessments/Attrition: Data were available for up to three years or six semesters, from spring 2015 (the program semester for the first cohort) through winter 2019 (three years after the third cohort's program semester). After the end of the one semester program, the follow-up period lasted three years. The analysis sample of 3,835 students meant a completion rate of 99%. However, Table A-2 shows that up to 35% of the sample had missing data on baseline measures.
Sample:
The analytic sample included more women than men and was divided almost equally between students aged 19 or younger and students aged 20 or older. Black (26%) and Hispanic (36%) students made up most of the sample, and a substantial proportion (37%) of the sample had a native language other than English. The proportion of students in the sample who were not the first in their family to attend college was almost double the proportion of students who were.
Measures:
The outcome measures came from administrative records of CUNY and the National Student Clearinghouse. The three confirmatory outcomes were measured for the full period and, when appropriate, for each semester: 1) college readiness as the number of college-ready subject areas based on standardized test scores or passing courses (available only for CUNY students and missing for about 3.7% of the sample), 2) academic progress as the cumulative college-level credits earned (available only for CUNY students), and 3) completion of a degree or certificate credential (available for all students from CUNY or the National Student Clearinghouse). Other measures included enrollment, developmental and college-level courses passed, and completion of gateway courses. The single-item measures lacked details on reliability and validity but appear suitable for the study and were independently gathered.
Analysis:
The analysis used regression models with controls for assignment blocks, baseline measures, and missing data indicators. Significance tests used robust (Huber-White) standard errors.
Intent-to-Treat: The analysis included nearly all participants and examined the effect of receiving a program offer rather than participating in the program.
Outcomes
Implementation Fidelity:
The authors reported without quantitative figures that the program was implemented with fidelity. Around 81% of program group members participated in CUNY Start and less than 1% of the control group participated.
Baseline Equivalence:
The tests for baseline equivalence used the analysis sample, which differed from the randomized sample by only 1%. There were no significant differences between program and control group members on any of the 13 baseline characteristics shown in Table 1 or any of the 17 baseline characteristics shown in Table A-2. Furthermore, an omnibus F-test showed that students' baseline characteristics were not jointly predictive of the condition (p = .885).
Differential Attrition:
Attrition was minimal (1%). Data were missing for baseline measures but, given nearly complete data for the outcomes, did not affect the final sample size.
Posttest and Long-Term:
Preliminary results at the end of the program semester reported in Scrivener et al. (2018) showed that the intervention group had passed significantly more college-ready subject areas and had higher CUNY assessment test scores than the control group.
The key results in Weiss et al. (2021) covering six semesters span both the posttest and long-term. The confirmatory results in Table 3 indicated significant program effects on two of three outcomes. The intervention group had significantly more college-ready subject areas and rates of certificate completion (14.5% versus 11.4%). Cumulative credits earned did not differ across conditions. Results presented separately by semester showed that the intervention group had significantly greater enrollment rates in the first three semesters but not the last three. The number of college-ready subject areas differed significantly in all six semesters, while credential completion differed only in semester six. Subgroup analyses suggested that the program most benefitted triple-remedial students but gave little evidence of any other moderation.
However, the authors stated that the one-semester CUNY Start model by itself did not appear to "meaningfully increase college-level credit accumulation or graduation rates in the absence of ASAP or something similar." In support of this point, a mediation analysis found that participation in ASAP accounted for most of the effect of CUNY Start on college-ready subject areas and credential completion. The authors note that CUNY Start enabled more treatment group students to become eligible for and participate in CUNY's Accelerated Study in Associate Programs (ASAP), a separate and highly effective three-year program. Since the control group had lower participation in ASAP, the effects of the two programs appear to be confounded.
Summary
Webber (2018) conducted a quasi-experimental study of 11,198 entering students at CUNY. Students who self-selected into the CUNY Start Program (n = 5,599) were matched and compared with the same number of students who did not participate in the program. Outcomes measured at one year and two years after entrance focused on passing remedial courses in English and math.
Webber (2018) found that, relative to the matched comparison group, the CUNY Start students passed significantly more
Evaluation Methodology
Design:
Recruitment: The study included a sample of students who entered either CUNY Start or a CUNY college as first-time freshmen with at least one remedial need between Fall 2013 and Spring 2015 (total n = 45,594). The students were required on entrance to take gateway courses to fulfill remediation requirements.
Assignment: The first-time students self-selected into the CUNY Start intervention group (n = 5,760) or to regular enrollment (n = 39,834). The intervention group postponed regular coursework to complete the one-semester program, while others began regular coursework immediately.
The study used propensity score matching to minimize condition differences in observed measures. The propensity score matching used a one-to-one matching algorithm without replacement (i.e., a "greedy" match) within cohort and college. The caliper restriction was tweaked by school and semester in order to ensure balance and maximize sample retention, and it ranged from 0.05 to 0.1. Propensity scores were based on demographics (gender, race and ethnicity, age, native language, country of origin, borough of residence) and academic background (initial remedial status, placement exam scores).
The final matched sample of 11,198 included 5,599 in the CUNY Start intervention group and 5,599 non-CUNY Start students in the comparison group. The authors stated on page 9 that "The overall match included 97 percent of students in the final sample."
Assessments/Attrition: Assessments occurred one year and two years after entry into the university, either as a CUNY Start participant for the intervention group or as a matriculated first-time freshman for the comparison group. The study did not report on attrition but likely excluded dropouts before doing the matching.
Sample:
According to Figure 3, the sample was 10% Asian American, 35% Black, 46% Hispanic, and 9% white.
Measures:
The study examined three primary outcomes based on fulfilling remediation requirements: (1) passed the English gateway course, (2) passed the math gateway course, and (3) passed both gateway courses. The measures came from student records maintained by CUNY.
The outcomes were measured from the point of enrollment in CUNY Start for the intervention group or the freshman year for the comparison group. Since CUNY Start students did not enroll in regular coursework until after the one-semester program, the intervention group had one fewer semester as regular students to take and pass courses than their peers in the comparison group.
Analysis:
The analysis used the matched students with logistic regression models and linear probability models that included fixed effects for college and cohort and controls for demographic characteristics and academic background. It was not possible to control for the outcomes of passing gateway courses, but the academic background controls served as proxies.
Missing Data Method: The study did not appear to have any missing data, but any missing data would have been excluded from the propensity score matching.
Intent-to-Treat: The analysis used all matched participants.
Outcomes
Implementation Fidelity:
Not examined.
Baseline Equivalence:
Not presented
Differential Attrition:
No missing data after matching.
Posttest:
Table A1 shows significant program effects for the matched sample on the three outcomes after one year (or one semester after the program end). The CUNY Start students passed significantly more English, math, and both English and math gateway courses than the matched comparison group. Odds ratios ranged from 1.435 to 2.259.
Moderation tests found similar effects after one year across racial and ethnic group but stronger effects among students with more remedial needs at baseline.
Long-Term:
Table A1 also shows significant program effects for the matched sample on the three outcomes after two years (or three semesters after the program end). The CUNY Start students passed significantly more English, math, and both English and math gateway courses than the matched comparison group. Odds ratios ranged from 1.435 to 1.743.
Summary
Polon & Haralampoudis (2021) conducted a quasi-experimental study involving 18,030 students at CUNY who enrolled with remedial coursework needs and self-selected into the intervention or regular coursework. The study used propensity score matching to equalize the conditions on baseline measures and assessed enrollment retention, credit accrual, and degree attainment over the next three to four years.
Polon & Haralampoudis (2021) found that the intervention group, relative to the comparison group, had significantly higher
Evaluation Methodology
Design:
Recruitment: The study sample began with all students who entered CUNY with remedial coursework needs from the Fall 2014 through Spring 2018 cohorts. Eight cohorts of students started in each of eight semesters over the study period. However, the sample was then limited to those students who were retained from semester one to semester two. The authors did not report the total number of eligible students, but the final sample selected for study was 18,030.
Assignment: The intervention group consisted of students who self-selected into CUNY Start, completed the one-semester program, and did not begin remedial coursework until the second semester. The comparison group consisted of CUNY students who did not participate in CUNY Start, began remedial coursework in the first semester, and continued on to the second semester. Both groups thus completed the first semester and enrolled in the second semester, but the groups differed in that the intervention group completed the program while the comparison group completed remedial coursework.
To help make the groups equivalent, the study used one-to-one propensity score matching without replacement. The student characteristics used to calculate the propensity score were age, race, sex, initial semester of enrollment at CUNY, CUNY institution of enrollment, new student code, College Admissions Average, TAP and Pell receipt, high school type, and remedial need in math, reading and/or writing prior to matriculation at CUNY or enrollment in CUNY Start. The matching produced 9,015 students in each condition.
The authors noted considerable overlap of the program with another program ongoing at CUNY, the Accelerated Study in Associate Program (ASAP). About 46% of the CUNY Start participants enrolled in ASAP compared to just 12% of the comparison group students
Assessments/Attrition: Assessments occurred in each of six semesters after the start of the program. The end of semesters three to six came one year or more after the program ended. The use of university records meant that there was no attrition for the eligible sample used in the matching.
Sample:
The matched sample consisted of 58% women, 33% African Americans, 45% Hispanics, and 11% whites. The sample averaged 23 years of age.
Measures:
The study examined three outcomes: 1) the rate of semester-to-semester retention, 2) the number of credits attempted and earned, and 3) the rate of degree attainment. The measures came from university records and appear to be cumulative (i.e., credits and degrees earned in early semesters are counted for later semesters).
Analysis:
The main analyses compared proportions across groups and included significance tests. One mention was made of multivariable regression analyses that controlled for the characteristics used to construct the propensity scores. The outcomes did not exist at baseline, but the model controlled for several baseline measures of past academic performance.
Missing Data Methods. The matched sample had no missing data.
Intent-to-Treat: All matched participants were included in the analysis, but the intervention group included only those students who finished the program and enrolled for the following semester. The comparison group was similarly limited to those who finished the first semester coursework and enrolled for the following semester.
Outcomes
Implementation Fidelity:
Not examined.
Baseline Equivalence:
The balance table in the Appendix reports the standardized mean difference between the matched conditions for the baseline measures. The differences were small, with only one of 34 being greater than .10. Significance tests for differences were not reported.
Differential Attrition:
No attrition
Posttest:
The study could not test for condition differences in persistence at posttest, as only those persisting from semester 1 to semester 2 were included in the sample. Persistence at posttest was therefore 100% for both conditions. For the other two outcomes, the intervention group had fewer credits and degrees than the comparison group.
Long-Term:
Table 2 shows that the persistence rate was significantly higher for the intervention group than the comparison group in semesters 3, 4, 5, and 6. The multivariable estimates indicated that CUNY Start students were 4.8 percentage points more likely to reenroll in their third consecutive semester at CUNY than the matched comparison students.
For the other two outcomes, CUNY Start participants had one less semester to enroll in credit-bearing courses and complete degrees than the comparison group during the span of the study. For both outcomes, CUNY Start students began at lower levels but increased at faster rates and closed the credit accrual gap and degree attainment gaps. For degree attainment in year four, the intervention group had a significantly higher percentage of students with degrees.