Cooperative Learning

Study 1

Evaluation Methodology

Design:

Recruitment: All seventh graders in 15 rural middle schools in the Pacific Northwest were invited to participate in the study. In the initial reports (Van Ryzin & Roseth, 2017, 2018), of the 1,750 eligible seventh-grade students, 1460 (83%) received parental consent to participate in the study. The subsequent reports (Van Ryzin & Roseth, 2019a, 2019b; Van Ryzin et al., 2020) included additional eighth graders who were not part of the original sample. The analysis sample rose to 1,890 students.

Assignment: Schools (n=15) were matched based on size and demographics (e.g., free and reduced-price lunch percentage) and randomly assigned to treatment (n = 7 schools, 668 students) or control (n = 8 schools, 792 students) conditions. [Note that numbers are reversed in Figure 1 of Van Ryzin and Roseth (2017).] The additional students included in Van Ryzin and Roseth (2019a, 2019b) and Van Ryzin et al. (2020) increased the numbers in both treatment and control schools, but the study did not provide specifics.  Van Ryzin and Roseth (2021) reported having 875 students in the intervention schools and 1,015 students in the control schools.

Assessments/Attrition: The initial reports (Van Ryzin & Roseth, 2017, 2018) examined baseline data from the fall of grade seven and follow-up data from the spring of grade seven. At the follow-up, data were collected on a total of 1,325 students (attrition rate of 9%).

The subsequent reports (Van Ryzin & Roseth, 2019a, 2019b, 2021; Van Ryzin et al., 2020) examined two more waves of data collection, one in the fall of grade eight and one in the spring of grade eight. The authors reported an analysis sample of 1,890 and more than 80% student participation in the data gathering. Van Ryzin and Roseth (2021) reported attrition of 10% at wave 2, 15% at wave 3, and 9% at wave 4. Given the length of the program (i.e., two years), the last follow-up corresponded to a posttest and the earlier follow-ups to interim assessments.

Sample: 

Overall, the initial sample was 48.2% female and 76.4% White. Other racial-ethnic groups included Hispanic-Latino (14.3%), multiracial (4.2%), and American Indian-Alaska Native (3.5%). Asian, African American, and Native Hawaiian-Pacific Islander were less than 1%. Approximately 14% were reported as having special education status. Income data were not provided at the student level. The proportion of low-income students served at the school level ranged from 33% to 95%.

Measures:

Bullying and victimization: Van Ryzin and Roseth (2018, 2019a)

Subscales from the University of Illinois Bully Scale were used. Bullying was assessed using five items, including "I teased other students while we were in a group" and "I spread rumors about other students." Alpha reliability was .74-.83 across waves 1-4. Victimization was assessed using three items, including "Other students picked on me" and "Other students made fun of me." Alpha reliability was .93 and .94 at baseline and follow-up, respectively. For both subscales, students responded on a 5-point scale ranging from 0 (Never) to 4 (7 or more times), and items were averaged to arrive at the subscale scores.

Perceived stress: Van Ryzin and Roseth (2018)

Four items from the Perceived Stress Scale were used. Items included "In the last month, how often have you been upset because of something that happened unexpectedly?" and "In the last month, how often have you felt that things were going your way?" (reverse-scored). Students responded on a 5-point scale ranging from 0 (Never) to 4 (Very often). Alpha reliability was .59 at baseline and .63 at follow-up. Items were averaged to arrive at the scale score.

Emotional problems: Van Ryzin and Roseth (2018, 2021)

Three items from the Emotional Problems subscale of the Strengths and Difficulties Questionnaire were used. Items included "I worry a lot" and "I am often unhappy, depressed or tearful." Students responded on a 3-point scale ranging from 1 (Not true) to 3 (Certainly true). Alpha reliability was .71 at baseline and .75 at follow-up. Items were averaged to arrive at the scale score.

Peer Relatedness (or close relationships with peers): Van Ryzin and Roseth (2018, 2019a, 2019b) and Van Ryzin et al. (2020)

Four items from the Relatedness Scale were used. Items included "When I'm with my classmates, I feel accepted" and "When I'm with my classmates, I feel unimportant" (reverse-scored). Students responded on a 4-point scale ranging from 1 (Not at all true) to 4 (Very true). Alpha reliability was .71 at baseline and .79 at the first follow-up. Items were averaged to arrive at the scale score.

Engagement (or low school commitment and attachment): Van Ryzin and Roseth (2018)

Four items from the Behavioral Engagement subscale of the Engagement versus Disaffection with Learning Scale were used. Items included "I try hard to focus in class" and "In class, I do just enough to get by" (reverse-scored). Students responded on a 4-point scale ranging from 1 (Not at all true) to 4 (Very true). Alpha reliability was .75 at baseline. Items were averaged to arrive at the scale score.

Alcohol use: Van Ryzin and Roseth (2017, 2019b)

Students reported on their use of alcohol in the last month using the following scale: 1 = no use, 2 = occasionally (1-3 times), 3 = fairly often (4-6 times), 4= regularly (7-9 times), and 5 = all the time (10+ times).

Willingness to use alcohol: Van Ryzin and Roseth (2017)

Students reported on their willingness to use alcohol in the company of friends (e.g., "Suppose you were with a group of friends and there was some alcohol that you could have if you wanted. How willing would you be to have one drink?"). Students responded using the following scale: 1= not at all willing, 2= a little willing, 3= pretty willing, and 4= very willing.

Deviant peer affiliation: Van Ryzin and Roseth (2017, 2019b)

Students reported on the frequency in the last month with which they associated with other youth who engaged in delinquent activities, including "get in trouble a lot," "fight a lot," "take things that don't belong to them," and "skip school" (four items overall). Alpha reliability was .76-.84 across waves 1-3.

Empathy: Van Ryzin and Roseth (2019a)

A cognitive empathy scale used three items from the Basic Empathy Scale (e.g., "I can usually realize quickly when a friend is angry" and "I can often understand how people are feeling even before they tell me"). An affective empathy scale also used three items (e.g., "After being with a friend who is sad about something, I usually feel sad" and "I can often understand how people are feeling even before they tell me"). Alpha reliabilities were between 0.69 and 0.78.

Prosocial Behavior: Van Ryzin et al. (2020)

Three items from the Prosocial Behavior subscale of the Strengths and Difficulties Questionnaire were used. Items included "I usually share with others" and "I often offer to help others (parents, teachers, other students)." Alpha reliabilities ranged from .70 to .81 across waves 1-4.

Analysis: 

Van Ryzin and Roseth (2017, 2018) used linear mixed multi-level models. In these models, student data (pretests, entered uncentered, and student demographics) were modeled at level 1 while school data (assignment) was modeled at level 2. All individual-level effects were fixed because the authors reported the cluster-level sample size was too small to model school-level random effects. Van Ryzin and Roseth (2017) reported that all models were fit using robust maximum likelihood (RML), which provides Huber-White standard errors. Willingness to use alcohol, which was different between treatment and control at baseline, was included as a covariate. Meanwhile, Van Ryzin and Roseth (2018) reported that interaction terms to represent moderation of intervention effects by baseline levels of marginalization (e.g., low engagement) were included at level 1. Baseline outcome measures were adjusted in each model.

For the analysis of all four waves of data, Van Ryzin and Roseth (2019a, 2019b) and Van Ryzin et al. (2020) used linear growth curve models. The Structural Equation Models estimated robust standard errors but given the small number of schools, did not use random effects for schools to adjust for clustering. The sample of 15 schools may not be large enough to accurately estimate robust standard errors, and the result may be to overstate the significance of the tests.

Intent-to-Treat: It appears that all reports included participants in their originally assigned condition regardless of participation in the treatment. Later reports gave only the final analysis sample size of 1,890 without details on completion rates for each assessment. However, it appears from the addition of new students in eighth grade and references to estimation "in the presence of missing data" that the maximum likelihood estimation included participants with partial data.

Outcomes

Implementation Fidelity: 

Research staff blind to assignment observed teaching practices in intervention and control schools. Results indicated significantly higher levels of observed positive interdependence in intervention schools compared to control schools.

Baseline Equivalence: 

Baseline equivalence was tested only on outcome measures, and there was a significant difference between treatment and control in 1 of the 5 outcomes reported in Van Ryzin and Roseth (2018) (emotional problems) and 1 of the 3 outcomes reported in Van Ryzin and Roseth (2017) (willingness to use alcohol), both in favor of treatment.

Later reports examined only the outcome measures used in the analysis. Van Ryzin and Roseth (2019a) found one significant baseline difference between conditions in four tests. Control students reported higher affective empathy, but the effect was very small (R-squared < .01). Van Ryzin and Roseth (2019b) found one significant baseline difference between conditions in three tests. Control students reported higher deviant peer affiliation, but the effect was small (R-squared < .01). Van Ryzin et al. (2020) found no significant baseline differences between conditions in two tests.

Differential Attrition: 

Not examined.

Posttest: 

Van Ryzin and Roseth (2017, 2018) tested 5 behavioral outcomes (bullying, victimization, emotional problems, close relationship with peers, and alcohol use) and found a significant main effect in favor of treatment on 3 of these outcomes. In addition, the authors assessed 4 risk and protective factors (perceived stress, deviant peer affiliation, engagement, and willingness to use alcohol) and found a significant main effect on 2 of these outcomes. Thus, the studies reported a main effect on 3 of 8 outcomes tested.

At the posttest, Van Ryzin and Roseth (2017) found that treatment students showed lower rates of alcohol use (beta = -.61) and deviant peer affiliation (beta = -.68) than control students. In addition, Van Ryzin and Roseth (2017) reported marginally significant results (p = .05) in favor of treatment for willingness to use alcohol (beta = -.69).

Van Ryzin and Roseth (2018) found that at the posttest, students in the treatment group reported lower rates of emotional problems (ES = .55) and higher rates of relatedness (or close relationship with peers; ES = .43) than students in the control group. In terms of moderation analyses, results showed that marginalized students in the treatment group reported lower rates of bullying, victimization, and perceived stress than marginalized students in the control group.

Two-Year Posttest: Van Ryzin and Roseth (2019a) found that the intervention group reported significantly better scores than the control group on all four outcomes: higher peer relatedness (wave 2), higher cognitive empathy and higher affective empathy (waves 2 and 3), and lower bullying (waves 2, 3, and 4). In addition, mediation tests confirmed the logic model by showing significant indirect effects of the program on bullying through affective empathy and peer relatedness.

Van Ryzin and Roseth (2019b) found that the intervention group reported significantly better scores than the control group on all three outcomes: higher peer relatedness (wave 2), lower deviant peer affiliation (waves 2 and 3), and lower alcohol use (waves 2, 3, and 4). In addition, mediation tests confirmed the logic model by showing significant indirect effects of the program on alcohol use through peer relatedness and deviant peer affiliation.

Van Ryzin et al. (2020) found that the intervention group reported significantly better scores than the control group on both outcomes: higher peer relatedness (wave 2) and more prosocial behavior (waves 2, 3, and 4). In addition, mediation tests confirmed the logic model by showing significant indirect effects of the program on prosocial behavior through peer relatedness.

Mediation: Van Ryzin and Roseth (2021) examined mediation using structural equation models with linear growth curves, fixed effects for schools, and robust standard errors to account for clustering. They first found that the program significantly increased the two outcomes of emotional problems and academic engagement. They further found that the program increased peer relatedness and reduced student stress across the first three waves, which in turn reduced emotional problems and promoted academic engagement in the last wave. The indirect effects of the program via increased peer relatedness and reduced student stress were small but significant for both emotional problems and academic engagement. The mediation was partial, as cooperative learning still had significant effects on emotional problems and academic engagement after controlling for the mediators.

Additional Results. Van Ryzin and Roseth (2022) examined the bidirectional relationship between peer relations and affective empathy but did not compare the intervention and control groups. They found that improved peer relations increased empathy at the same time increased empathy improved peer relations - meaning the two were mutually reinforcing. Together, changes in peer relations and empathy reduced bullying.

Supplemental Runs. In response to requests made by the Blueprints Advisory Board, Van Ryzin provided additional sets of results (May 25, 2023) that: 1) included only the students initially randomized to condition (and not those who entered the schools after randomization), 2) examined program effects on outcomes at wave 4, the end of eighth grade (and excluding data from interim waves at the end of seventh grade and the start of eighth grade), and 3) used binary measures for the highly skewed outcomes of alcohol, tobacco use, and marijuana use. In addition, Van Ryzin offered new details on ICCs for school-level clusters, which ranged from .02-.11, and on the statistical model (robust standard errors were not used).

To summarize the supplemental results, the sample based on originally assigned students showed baseline equivalence for the full randomized sample (one significant condition difference in 15 tests) and for the analysis sample (no significant condition differences in 15 tests). The latter tests indicated little in the way of differential attrition. Additional tests showed no relationship of gender or race/ethnicity to attrition. The statistical models for the sample of originally assigned students showed a significant intervention effect on outcomes at the end of eighth grade in 14 of 16 tests. The results thus demonstrated program efficacy for several risk and protective factors and for several key outcomes such as prosocial behavior (d = .52), anti-social behavior (d = .48), bullying (d = .89), emotional problems (d = .83), academic engagement (d = .52), tobacco use (OR = .44), alcohol use (OR = .56), and marijuana use (OR = .45).

Long-Term: 

Not examined.

Study 2

Study 2 (Low & Van Ryzin, 2023) leveraged an existing web-based software (https://PeerLearning.net) to support teachers in implementing Cooperative Learning with fidelity to essential design components. When considering lesson design, https://PeerLearning.net contains several lesson templates that possess built-in fidelity to key Cooperative Learning design principles. The lesson design process is implemented as a series of wizard-like screens that ask users to (a) enter/select lesson details (title, lesson type, subject, grade, and topic/domain), (b) upload the lesson content (i.e., curricular materials), (c) create prompts for student note-taking through https://PeerLearning.net, (d) add a follow-up activity to the lesson (if desired), (e) insert student info (i.e., a description of the learning goal for students, a plan for assessment, and any group-level bonuses that are offered to enhance positive interdependence), and (f) specify the length of each phase of the lesson to ensure that the lesson can finish on time. The ultimate goal of the software platform is to support teachers in adhering to best practices in Cooperative Learning while also reducing the implementation burden, allowing teachers to provide these learning experiences more frequently and with less time and effort. Training in https://PeerLearning.net for teachers began in the fall and consisted of two independent two-hour sessions roughly two months apart. Teachers in intervention schools were asked to use https://PeerLearning.net to deliver a lesson at least twice per month.

Summary

Study 2 (Low & Van Ryzin, 2023) used a cluster randomized controlled trial to compare a sample of 813 students in 12 middle and high schools in the Pacific Northwest assigned to one of two conditions: (1) Cooperative Learning; or (2) control. Assessments at baseline and posttest measured social-emotional skills, prosocial behavior, and supportive peer relations.

Study 2 (Low & Van Ryzin, 2023) found at posttest that students in intervention schools, relative to those in control schools, showed significantly greater improvements in:

• Prosocial behavior
• Supportive peer relations
• Self-awareness (risk & protective factor)
• Self-management (risk & protective factor)
• Social awareness (risk & protective factor)
• Relationships skills (risk & protective factor)
• Decision making (risk & protective factor)

Evaluation Methodology

Design:

Recruitment: The study took place during the 2021-2022 school year. Twelve middle and high schools in the Pacific Northwest volunteered to participate in the study. The authors noted that they intended to enroll eighth graders in middle schools and ninth graders in high schools, but two middle schools requested that they enroll sixth graders instead. Between 79% and 97% of eligible students in each school were successfully recruited, for an initial sample of 813 students. The authors did not provide information on consent rates by condition, however.

Assignment: Participating schools were matched based upon size, grade level, and demographics (i.e., free/reduced lunch percentage, race/ethnicity) into pairs. Using an online random number generator, one school from each pair was randomized to the intervention condition, with the remaining school being allocated to the control condition. Table 1 indicates that 389 and 424 students were allocated to the intervention and control groups, respectively. Control schools received no training and proceeded with their standard instructional practices.

Assessments/Attrition: Assessments were conducted at baseline and posttest (i.e., six months after baseline). Of the 813 students who responded to the baseline survey, 720 (88.6%) completed the posttest.

Sample:

The analytic sample was 50.2% female and 70.7% White. Other racial/ethnic groups included Hispanic/Latino (18.0%), multiracial (7.5%), and Asian, African American, American Indian/Alaska Native, and Native Hawaiian/Pacific Islander (all <3%).

Measures:

Outcome data came from student self-reports using online surveys. Two behavioral outcomes were computed for analysis: (1) prosocial behavior; and (2) supportive peer relations. The additional five outcomes of self-awareness, self-management, social awareness, relationship skills, and decision making represent risk and protective factors. Alpha reliabilities ranged from .59 to .90. 

Analysis:

Analyses used two-level hierarchical linear models with maximum likelihood estimation to account for nesting of students within schools. However, the level-2 sample size of 12 schools may not be large enough to accurately estimate the standard errors, and the result may be to overstate the significance of the tests. Baseline outcomes were included as fixed effects in all models. Cross-level interaction terms were used to evaluate moderation by student characteristics.

Missing Data Methodology: The hierarchical linear models used FIML estimation to include participants with incomplete data.

Intent-to-Treat: All participants with available data (i.e., who completed at least one survey) were used in analyses.

Outcomes

Implementation Fidelity:

For schools in the intervention group, the number of lessons taught per school ranged from 31 to 108 (M = 39.3). Teachers in control schools indicated no use of Cooperative Learning.

Baseline Equivalence:

Tests for baseline equivalence indicated no significant differences between conditions on the seven outcome measures. Whether the tests used the initial or analytic sample is unclear, and no tests were reported for baseline sociodemographic characteristics. 

Differential Attrition:

The authors reported that attritors "were not significantly different from the students who participated in Wave 2 in terms of sex, race/ethnicity, or intervention condition" (p. 5). No other information was provided.

Posttest:

Relative to students in schools assigned to the control group, those in intervention schools showed significantly greater improvements across all seven outcomes (ds = .34-.82). Results were unchanged when correcting for the family-wise error rate among the social-emotional learning subscales.

There was no evidence for moderation by sex, race/ethnicity, or grade level.

Long-Term:

Not examined.