Red Light, Purple Light!: A Self-Regulation Intervention

Study 1

Evaluation Methodology

Design:

Recruitment: The researchers mailed letters to the parents of all 4-year-olds in nine classrooms at the two participating preschools inviting them to participate in the study. A total of 74 children were selected to participate based on their kindergarten eligibility the following year.

Assignment: Half of the children in each classroom were randomly assigned to receive the intervention program (with the other half serving in the control condition).

Assessments/Attrition: Baseline data were collected in the fall (November and December) of the final year of preschool before kindergarten, and posttest data were collected in the spring (April and May) of that year. Of the initial recruited sample of 74 participants, data were obtained from all children at baseline, and data were obtained from 65 children at posttest, resulting in an overall attrition rate of 12%. Of the nine non-completers, four moved away, one left school for a family vacation, three declined to participate in the posttest, and one withdrew from the study due to newly diagnosed developmental delays.

Sample:

The majority of the children in the study attended preschool in a university child development center and laboratory school. Placement in the center was available either through tuition or at no cost to children enrolled in Head Start. Approximately half of the children in each classroom paid tuition, and half received care at no cost because of enrollment in Head Start. Thus, approximately half the sample was low-income as defined by Head Start status. Mothers of children enrolled in Head Start had an average education level of 12.2 years with a range of 6-16 years. Mothers of children who were not enrolled in the Head Start program had an average education level of 17 years with a range of 12-21 years. The average age of children was 4.6 years, with 39 females (60%) and 26 males (40%).

Measures:

The primary outcomes of interest were behavioral self-regulation and early academic achievement. Behavioral self-regulation was measured with the Head-Toes-Knees-Shoulders task, a validated and well-established measure for children. Twenty test items are scored as 0, 1, or 2, resulting in a total possible range of scores from 0 to 40. The measure had an interrater reliability of kappa = 0.92 for the present study sample, and demonstrated external validity based on correlations with teacher-rated behavioral self-regulation at each time point (fall r=.20, p=.09; spring r=.24, p=.06).

Academic outcomes were measured using three subtests of the Woodcock Johnson Psycho-Educational Battery-III Tests of Achievement or the Batería III Woodcock-Muñoz: the Applied Problems subtest, Letter-Word Identification subtest, and Picture Vocabulary subtest. This is a well-established standardized measure of early childhood academic competence.

Analysis:

The researchers used t-tests to examine baseline equivalence and then used multiple regression analyses to estimate intervention effects. The regression model for behavioral self-regulation predicted self-regulation change scores (spring minus fall) from group while controlling for Head Start status, age, gender, baseline outcome scores, and the interaction between group and baseline outcomes. The regression models for academic achievement (applied problems, word-letter identification, and picture vocabulary subtests) predicted difference scores on each subtest (spring minus fall) from group while controlling for age, gender, and Head Start status. No basis was provided for inclusion of these covariates aside from previous research documenting the importance of these variables for predicting the outcomes of interest. The researchers stated that the intraclass correlation for classrooms on baseline self-regulation scores was 0.06 so multilevel modeling was not used despite the nesting of children within classrooms and schools.

Intent-to-Treat: The researchers appear to have followed intent-to-treat. Only participants missing data from one or both assessment periods were dropped from the analysis.

Outcomes

Implementation Fidelity:

Children attended an average of 11.3 of the total 16 sessions in the program (range = 5-16). Most of the missed sessions were due to illness or vacation.

Baseline Equivalence:

The researchers used the analysis sample rather than the randomized sample to test for baseline equivalence on nine demographics and outcomes and did not find any significant differences, though the intervention group scored higher than the control group on all measures evaluated.

Differential Attrition:

The researchers reported that the children who left the study did not differ significantly from those who remained in the study on age or gender. A higher percentage of children who left the study were enrolled in Head Start (67%) as compared to the overall sample (43%), although this difference was not statistically significant (p > .05). The lack of significant differences between conditions in tests for baseline equivalence using the analysis sample suggested no problem with differential attrition.

Posttest:

Children in the intervention group showed significantly greater improvements in letter-word identification relative to children in the control group, controlling for age, gender, and whether they were enrolled in Head Start. No group effects were found for the other two subtests of the academic achievement test (applied problems and picture vocabulary) or for behavioral self-regulation.

Post-hoc subgroup analyses also revealed that the program significantly improved self-regulation scores for children in the bottom half of scores at baseline. The number of sessions attended also significantly predicted gains in self-regulation for children in the bottom half of scores at baseline.

Long-Term:

Not examined.

Study 2

Evaluation Methodology

Design:

Recruitment: Children and families were recruited through letters in an enrollment packet about the study sent during the summer prior to the school year. Children were enrolled in one of 14 Head Start classrooms across nine sites in a Pacific Northwest city. The recruited sample included 276 children. Signed consent was obtained from 24 teachers and from a parent of all children in the study.

Assignment: Classrooms (N = 14) were randomly assigned to receive the intervention program or no program. A total of 126 children were randomized to the treatment condition and 150 children to the control group, with seven classes in each condition.

Assessments/Attrition: Participants completed assessments in the fall and spring of their final year of preschool. From baseline to posttest, the sample decreased 275 to 241 (12.7% attrition). Missing data on individual tasks at pretest and posttest, either due to attrition or other extraneous circumstances (e.g., child was sick during testing session), ranged from 6% to 17%.

Sample:

The sample included preschoolers enrolled in the Head Start program from nine schools in a city in the Pacific Northwest.

Measures:

The primary outcomes included self-regulation and academic performance. Self-regulation was measured using a teacher-report survey and two child behavioral tasks. Teachers rated self-regulation using the Child Behavior Rating Scale, which had an alpha of 0.96 in the present sample. The child behavioral tasks included the Head-Toes-Knees-Shoulders task, a well-established and validated measure, and the Dimensional Change Card Sort task, a well-established measure of cognitive and attentional flexibility, which had an alpha of 0.84 in the fall and 0.77 in the spring in the present sample.

Academic outcomes were measured with the Applied Problems subtest, Letter-Word Identification subtest, and Picture Vocabulary subtest of the Woodcock Johnson Psycho-Educational Battery-III Tests of Achievement or the Batería III Woodcock-Muñoz.

Analysis:

The researchers used multilevel regression models clustered by classroom to analyze differences in self-regulation between groups, controlling for baseline outcome scores, age, and English language learner status. These covariates were included based on previous research suggesting their relation to self-regulation and academic achievement. Data were assumed to be missing at random because analyses showed that missing data was not related to ethnicity, parent education marital status, maternal age, parent employment, or residential mobility. Full information maximum likelihood was used to estimate models. Additionally, path models were used to test for mediation of self-regulation on the relationship between the intervention and academic scores.

Intent-to-Treat: The researchers stated they used intent-to-treat. Missing data were handled using full information maximum likelihood estimation under a missing at random assumption.

Outcomes

Implementation Fidelity:

Children in the intervention condition attended an average of 13 out of the 16 sessions, and 66% of the participants attended 80% of the sessions or more. Fidelity of the session content was also monitored and addressed in regular discussions.

Baseline Equivalence:

At baseline, the randomized conditions did not differ significantly by age, gender, parent education, parent marital status, maternal age, English language learner status, or outcome measures.

Differential Attrition:

The control group had a 19% attrition rate and the intervention group had a 5% attrition rate though the authors did not report whether this difference was significant. Missing data was not significantly related to ethnicity, parent education, marital status, maternal age, parent employment status, or residential mobility.

Posttest:

Children in the intervention condition demonstrated stronger improvements than children in the control condition on two of six outcomes tested. Specifically, children in the intervention had stronger self-regulation at posttest on two of three self-regulation measures (the two behavioral tasks, but not the teacher-report survey) relative to children in the control condition. Cohen's d = 0.32 and 0.16 for the two different self-regulation tasks. No group differences were observed for changes in the three subtests of academic achievement.

Subgroup analyses also revealed that English language learners showed significantly greater improvements in math scores in the intervention condition than English speakers in the intervention.

Long-Term:

Not examined.

Study 3

Evaluation Methodology

Design:

Recruitment: Twenty ECE centers were invited to take part in the study. These centers provided full-day early-learning and childcare and were in a range of socioeconomic areas (low SES = 7; mid SES = 8) across the Auckland (New Zealand) region. Centers belonged to a large national early care and education provider, who expressed interest in trialing the program. The study was confined to 4-year-old children for two reasons: (1) In New Zealand children typically start school at age 5-years, (2) the possibility of floor effects on outcome measures used in the study in children under 4 years of age. Signed consent was obtained from 61 teachers; however 15 teachers were included in the study.

Assignment: Sixteen early childhood education centers were randomized to either treatment (8 weeks of twice weekly 20- to 30-min sessions of the intervention; n = 8 centers) or a waitlist control group (business-as-usual, i.e., the same daily activities, routines, and curricula that preceded study participation; n = 8 centers). The intervention took place in the children's regular classroom (on the same two days each week) and the curriculum was taught by ECE qualified teachers. The article states that paired randomization was used to balance center size across the intervention and control groups, which involved pairing centers based on their similarity in the number of consented children. However, 16 centers were assigned in pairs and one of the centers (from the control group) did not participate, but the paired cluster from the treatment group was not dropped. This seems to be a violation of the randomization procedure.

Assessments/Attrition: One center dropped out after randomization, for an attrition rate of 6% at the cluster level. At the child-level, the authors did not report how many children were in the center that dropped out. Based on the 15 centers and 15 teachers included in the analysis, the sample included 212 children (107 assigned to treatment and 105 assigned to control). However, as noted above, this number did not include the count of children from the center that dropped out. At the posttest, 187 children of the initial reported sample of 212 children remained in the study (for an overall attrition rate of 12%). At the follow-up (4 months post-intervention), 124 children remained (for an attrition rate of 42%). In addition, the authors reported missing data for individual outcome measures that resulted from either attrition or other circumstances (e.g., absences on the day of assessments or child refusal to be assessed).

Sample: The children ranged in age from 47 to 58 months (M = 52.52 months, SD = 2.93), at the date of the pre-intervention assessments. Fifty two percent of the children were female. Thirty-one different ethnic groups were represented within the sample. Most parents nominated one ethnic group (n = 154, 73%) for their child, with Asian descent the most frequent, followed by New Zealand European and Pacific Island origins. Fifty-seven parents (27%) reported their child had two or more ethnicities. The majority of parents reported one main language spoken at home (n = 163, 77.1%), with English the most common (n = 114, 69.9%), followed by Indian (n = 19, 11.7%), and Chinese (n = 13, 8%). Forty-eight parents (22.9%) nominated two or more languages spoken at home.

Most of the teachers were female, with an average of 5.9 years of ECE teaching experience.

Measures:

Executive Functioning:

• The HTKS task (Cameron Ponitz et al. 2009) was used to assess children's integration of working memory, attentional flexibility, and inhibitory control. In the present study, 60 HTKS assessments were jointly observed and scored by pairs of raters, with an interrater agreement of Kw = 0.96.
• The DCCS task from the National Institute of Health (NIH) Toolbox Cognitive Function Battery (Zelazo et al. 2013), which was administered via an app on an iPad, was used to assess cognitive flexibility and attention. No reliability or validity was reported.

Behavioral Self-regulation

• Teacher ratings of self-regulation were assessed with the 10-item subscale of the Child Behavior Rating Scale (CBRS). In the current sample, internal consistency for this subscale ranged between α = .95 at pre- and post-test and α = .96 at the follow-up. This measure was not independent (as teachers administered the intervention).

Analysis: Hierarchical linear mixed models were used to estimate the effects of the intervention. Separate models were fitted for each outcome and at each time point. Covariates included baseline outcome scores, baseline receptive English vocabulary (BPVS) scores, children's age, and gender.

Intent-to-Treat: Authors stated they conducted an intent-to-treat analysis; however, one center in the control group did not participate and data were not collected from this site and the paired treatment center that remained in the study was not dropped - which could be a violation of intent-to-treat protocol. Missing data were imputed separately for each output measure at each time point. Thus, for example, for HTKS at the second time point, separate imputations were carried out for the intervention and waitlist control groups and the two sets of imputed complete data were recombined before analysis. Imputation respecting the hierarchical structure of the data was carried out using the R package pan.

Outcomes

Implementation Fidelity: On average, children were enrolled at their center for 32.23 h (SD = 11.78) per week and had spent 100.55 weeks (SD = 57.31) at their current center. Five RLPL sessions, spaced over 8 weeks, were observed by the researchers at each intervention center to assess quality and fidelity of program implementation. Across the eight intervention-group centers the average attendance rate was 12 RLPL sessions (SD = 4.12; range 1 to 16) and 72 children (70%) attended 12 or more intervention sessions.

Baseline Equivalence: Since the authors did not report the cluster of children in the center that withdrew after randomization, baseline equivalence for the randomized sample is unknown. Table 2 (p. 627) reports baseline equivalence for the 212 students within the 15 centers (i.e., the analysis sample). No significant differences were detected between conditions in baseline outcome scores or in baseline receptive English vocabulary (BPVS) scores. Socio-demographic characteristics were not assessed for baseline equivalence.

Differential Attrition: Logistic regressions were run to determine whether demographic variables (i.e., child age, child gender, weekly enrolment hours at ECE, parent employment status) were related to missingness. Child age predicted missingness on the DCCS at T2 and DCCS, HTKS and CBRS at T3. Data on these measures were more likely to be missing for older children in the sample who had started school and could not be contacted for data collection. However, baseline measures were not included in the logistic regression analyses. In addition, attrition by condition was not assessed.

Posttest: Compared to control, children who participated in the intervention demonstrated significantly stronger executive functioning skills at post-intervention, as measured by the DCCS task. These effects, however, were no longer significant at the 4-month follow-up. There were no posttest or follow-up effects for executive functioning (as measured by the HTKS) or teacher-rated self-regulation (as measured by the CBRS).

Long-Term: Not reported.

Study 4

Evaluation Methodology

Design:

Recruitment: Children from low-income families were recruited from 13 Head Start classrooms across nine sites in the Pacific Northwest of the United States. Children and families were recruited through consent forms distributed in enrollment packets during the summer prior to the start of preschool. Children were eligible to participate in the study if they were ages of 3-5 and attending, or planning to attend, one of the 13 target classrooms.

Assignment: A total of 188 children nested within 13 classrooms, 8 teachers and 7 Head Start sites were assigned to condition. The present study evaluated the core curriculum that focuses on self-regulation (attentional flexibility, working memory, and inhibitory control) compared to an enhanced version of the curriculum with embedded early math (counting, cardinality, and numeral knowledge) and literacy (phonological awareness and print knowledge) content. In both versions of the intervention, teachers were trained to administer the self-regulation intervention in preschool classrooms with coaching support. Both treatments were compared to a business-as-usual control group, in which children engaged in the daily routines and curricula activities that came before study participation. Block randomization occurred at the teacher level (n=8) in the winter so that teachers leading more than one classroom (i.e., teachers with a morning and an afternoon class) delivered the same condition in each classroom. In total, five classrooms were assigned to each of the two intervention conditions and three were randomly assigned to the control.

Assessments/Attrition: The study assessed the children in the fall (baseline) and the spring (posttest) of the school year. At the posttest, 157 completed an assessment for an overall attrition rate of 17%. In addition, of these 157 children, data were missing by assessment at either the pretest or posttest and this missing ranged from 3-11%. If children did not pass the preLAS (see "Measures" section), and their home language was not Spanish, they were not administered any assessments at that time point (n = 2).

Sample: At pretest, children had an average age of 51 months (range = 38-62 months, n = 41 3-year-olds, 99 4-year-olds, 17 5-year-olds). More than half of the sample of children and families identified as Latino (58%), 26% identified as White, 7% Pacific Islander, 6% African American, and 2% reported other for ethnicity. Information from the consent form (child's home language) identified 62 children (33%) as ELLs. All teachers were female, and five taught across a full day (one morning session; one afternoon session) while three taught half-day only (one morning session or one afternoon session). Parents' education level ranged from 2 to 17 years (M = 11.27, SD = 2.30).

Measures: Direct pre- and post-test assessments included well-known and commercially available self-regulation and inhibitory control skills (Cronbach's a = 0.90-.96 at pre-test and 0.91-0.97 at post-test for the current sample), emergent literacy (reliability for English-speaking preschool children ranged between 0.98-0.99 and 0.84-0.98 for Spanish-speaking children), and early math skills (Cronbach's a = 0.91 at pre-test and 0.92 at post-test). They were administered using trained research assistants blind to research condition. Spanish-speaking research assistants administered the Pre Language Assessment System (preLAS) at pre-test and post-test to determine whether a child should receive direct assessments in English or Spanish. Children identified as ELL's were assessed by Spanish-speaking research assistants at pre and post-test, whether or not children passed the preLAS.

Analysis: Authors utilized clustered robust standard errors to adjust for the nested structure of the data. The level-2 sample size of eight may not be large enough to accurately estimate the standard errors, and the result may be to overstate the significance of the tests. Separate, but parallel analyses for each of the self-regulation and academic outcomes were run, and all models included children's performance at pre-test on the outcome variable, as well as their age, gender, ELL status, and parent level of education.

Missing Data Method: All analyses excluded those without any posttest data and were conducted using the data from the 157 children who contributed at least partial data at both pre-test and post-test using a full information maximum likelihood (FIML) estimator in Stata 15.1.

Intent-to-Treat: The authors stated they conducted an ITT analysis where children's scores were analyzed as part of their assigned intervention group regardless of whether they were present for all aspects of their assigned intervention group.

Outcomes

Implementation Fidelity: On average, participating children attended 14 sessions across both intervention groups and 95% of participating children attended at least 10 intervention sessions. All participating teachers delivered 100% of the intervention sessions, in order, and on the dates scheduled (two times per week for 8 weeks). Videos of intervention classrooms indicated that teachers implemented the intervention effectively (e.g., played the correct games, modeled appropriate behaviors) and adhered to the condition of the intervention in which they were trained. Coders did not observe any deviations from the session guides and learning objectives included in the training manual. All participating classrooms (BAU and intervention classrooms) used Creative Curriculum. A review of the curricula and lesson plans used by BAU classrooms along with video observations also confirmed that teachers in BAU classrooms were not playing self-regulation games of a similar nature to those in either version of the intervention as part of their typical practice.

Baseline Equivalence: Baseline equivalence was not tested for the randomized sample. For the analysis sample, it was tested using seven variables, four demographic variables (age, gender, ELL status, maternal education) and four baseline measures (self-regulation, inhibitory control, early math, early literacy). Significant baseline differences were detected for four of these eight variables. Children in classrooms with teachers randomly assigned to the control group had higher baseline scores on self-regulation, and on early math skills, than children in classrooms with teachers randomly assigned to treatment groups. There were also significant differences in the proportion of ELL children and maternal education in the BAU control group and the two intervention groups.

Differential Attrition: Completers significantly differed from attritors in one of seven demographic and pretest measures: Children who did not participate in the post-test session were significantly more likely to be younger than children who did participate. Authors stated they assessed whether missingness within the 157 cases included in the analysis sample on any of the variables was due to any auxiliary variables available in the dataset using logistic regression and no significant predictors emerged. 

Posttest: Results comparing each treatment contrast to the control condition are presented in Table 3. For the regular version of the treatment, there were significant posttest differences in favor of the treatment group in one of three outcome measures: Children in the treatment group earned significant higher math skills at posttest compared to children in the control group (d = .34). This outcome was one that differed significantly at baseline. Similarly, children in the enhanced version of the treatment earned significantly higher math skills at posttest compared to children in the control group (d = .38). (Note, p-values reported in Table 3 are for one-tailed tests but remain statistically significant when calculated using two-tailed tests). There were no statistically significant differences across groups for literacy skills or in self-regulation.

Long-Term: Not tested.