Promoting First Relationships

Study 1

Evaluation Methodology

Design:

Recruitment: Researchers in the Spieker et al. (2012) study contacted caregivers of infants in one county between the ages of 10 and 24 months who had experienced a court-ordered placement that resulted in a change in primary caregiver within the prior 7 weeks, based on Department of Social and Health Services records. Initially, 427 caregivers were contacted and 280 cases were deemed eligible for the study (i.e., caregivers had to speak English and could be foster parents, biological parents or adult kin), of which 219 enrolled and 61 declined. After enrollment, an additional 9 dyads were declared ineligible, so the sample size at randomization was 210 caregiver/children dyads.

Assignment: Spieker et al. (2012) randomly assigned participating parent-child dyads, blocked by caregiver type (foster parents, biological parents or adult kin), to either the treatment or a control group. Control group participants received The Early Education Support program, which consisted of three monthly 90-minute home visits to promote growth and development and information on available resources. Of the 210 dyads, 105 were each randomly assigned to the treatment or control conditions.

Attrition: Of the 210 dyads randomized in Spieker et al. (2012), 175 (or 83%) completed the posttest with their original caregiver and thus were included in the analysis. At 6 months after the intervention, the sample included 129 dyads who completed the assessment with their original caregiver (70% of those randomized).

Spieker et al. (2014) also examined permanency and stability at 2-year follow-up. Presumably, although the authors did not discuss it, there was no attrition as the measures employed were obtained through state records. Pasalich et al. (2016) used the same sample and the same measures as Spieker et al. (2012).

Sample: Over half of infants in the Spieker et al. (2012) sample were male (55%) and approximately 55% of infants were white, 20% were mixed race, 15% were black, 10% were Hispanic, 7% were Native American or Alaskan native, and 4% were other races or were not identified. The average age of infants at the beginning of the study was 36 months and on average infants had experienced over 2 caregiver changes since birth. Approximately 27% of households in the study earned an income of less than $20,000 per year. On average, caregivers were around 36 years of age and had 13 years of education.

Measures: Spieker at al. (2012) and Pasalich et al. (2016) included 2 pre- and posttest measures of child behavioral outcomes and 3 child behavioral outcome measures administered at baseline and 6 months after the intervention (without posttest assessment). In addition, the authors used 5 measures of risk and protective factors and 6 assessing caregiver outcomes.

Child behavioral outcomes included:

• Competence (sample α = .69-.70) and problem behavior (sample α = .77-.79), completed by caregivers at all three time points, were both assessed using the Brief Infant Toddler Social and Emotional Assessment.
• The caregiver-reported Child Behavior Checklist scales of internalizing (α=.80) and externalizing (α=.90) behaviors, conducted at baseline and 6-month follow up.
• Emotional regulation from the Bayley Behavior Rating Scale (sample α ranged from .79 to .87) completed by researchers blind to condition at baseline and 6-month follow up.

Risk & protective factor child outcome measures included:

• Infant attachment security, scored by an observer at each time point (baseline, posttest and 6-month follow-up), was measured using the Toddler Attachment Sort-45 (interrater reliability conducted on 16% of the sample, r = .92).
• Engagement was assessed by researchers at each time point using the Indicator of Parent-Child Interaction by coders (sample α = .79-.82).
• Sleep problems (sample α=.70) and "other problems" (sample α=.70) assessed at baseline and 6-month follow-up using the Child Behavior Checklist completed by caregivers.
• Orientation/engagement from the Bayley Behavior Rating Scale (sample α ranged from .79 to .87) completed by researchers blind to condition at baseline and 6-month follow up.

For caregiver outcomes assessing reciprocal parent-child warmth, Spieker et al. (2012) and Pasalich et al. (2016) included the Nursing Child Assessment Teaching Scale to report parent sensitivity, which was independently coded by researchers using a videotaped session (sample α ranged from .71 to .79). Parent sensitivity items represented aspects of positive interaction and indicators of mutuality (e.g., contingency, gaze, and positive affect), caregiver verbal and nonverbal support of child, and sensitive instruction during the teaching task. The Indicator of Parent-Child Interaction, also scored by observers, reported support (sample α ranged from .76 to .84), which included items such as "acceptance/warmth," "descriptive language," and "follows child's lead." Commitment to child, which assessed the caregiver's desire to parent the child as long as the child remains in care or is benefitting from care and the caregiver's ability to fully attach to the child without withholding feeling or putting up barriers to limit the extent of the attachment, was rated based on interviews with caregivers using the This Is My Baby questionnaire (interrater agreement was r = .89). Caregivers reported understanding of toddlers (which measured caregivers' knowledge of infant and toddler social-emotional needs and developmentally appropriate expectations) with the Raising a Baby survey (sample α ranged from .73 to .77). Finally, the caregivers rated their perception of having a difficult child (Stress - Difficult Child) and having a dysfunctional parent-child relationship (Stress - Dysfunctional Interaction) with the Parenting Stress Index (sample α ranged from .87 to .89). All six of these measures were administered at baseline, posttest and 6-month follow-up.

Two years after initial enrollment in the Spieker et al. (2012) study, Spieker et al. (2014) used state child welfare administrative data to measure whether a child had remained with the study caregiver since randomization (stability) and whether the child remained with the same caregiver with a legal discharge to that caregiver (permanency).

Analysis: Spieker et al. (2012) used analysis of covariance models to assess differences by experimental condition in caregiver and child outcomes at pretest, posttest, and follow-up. For posttest and 6-month follow-up, the baseline score on the given measure (when available) was included as a covariate, in addition to other demographic variables including whether the child experienced multiple removals from the birth home, caregiver type, and age of child. Pasalich et al. (2016) conducted an additional moderation analysis to analyze whether there was a differential effect of the treatment on a cascade from placement instability to insecure attachment to externalizing problems in toddlers.

At the two-year follow-up, Spieker et al. (2014) used logistic regression to predict stability and permanency, with intervention group and various demographic variables (i.e., age of child at randomization, time in child welfare at time of randomization, number of placement changes prior to randomization, multiple removals - whether the child experienced one or more failed reunifications with the birth parent prior to randomization, and commitment to child) included as covariates. Baseline outcomes for the Spieker et al. (2012) study were not used in the Spieker et al. (2014) analysis.

Intent-to-Treat: Spieker et al. (2012) reported using an intent-to-treat protocol and treated infant-caregiver dyads that did not remain intact at follow-up time points as missing or attrition. To analyze intervention effects on outcomes they used both ANCOVA models with a listwise deletion approach to missing data and growth models that used maximum likelihood estimates and included data on all participants, including those with missing data at follow-up time points. Pasalich et al. (2016) reported they used FIML to handle missing data. Using the same sample as Spieker et al. (2012), Spieker et al. (2014) included all 210 children, consistent with an intent-to-treat model.

Outcomes

Implementation Fidelity: Spieker et al. (2012) reported that three video feedback sessions per family were conducted during the study. Independent observers assigned a global rating on a scale of 1-5 assessing level of fidelity to each treatment segment observed, and reported an average score of 4.04 (SD = .76). In addition, 97% of activities were completed.

Baseline Equivalence: Spieker et al. (2012) reported no differences between conditions in outcome measures at baseline. Infants in the treatment condition were significantly more likely to have experienced multiple removals from their birth families, but it appears there were no other demographic differences at baseline.

Differential Attrition: Spieker et al. (2012) reported no differences between completers and attritors for condition, gender, caregiver type, multiple removals, or number of caregiver changes before enrollment. At posttest, treatment group participants who did not have a change in caregiver included younger children and were more likely to have completed all sessions of the intervention. However, differential attrition for dyads in the comparison was unclear, and the study did not analyze condition by outcome for attrition.

Posttest: At posttest, Spieker et al. (2012) found a significant improvement in caretakers' perception of child competence among treatment group infants and toddlers as compared to control group infants and toddlers. However, this significant difference did not remain at the 6-month follow-up. Spieker et al. (2012) also found a significant improvement in parent sensitivity and understanding of infants and toddlers among treatment caregivers as compared to infants and toddlers among control caregivers at posttest, but these differences also did not remain at 6-month follow-up.

Pasalich et al. (2016) found several moderator effects (no main effects were tested in this study). That is, among children with more than 4 placement changes (27% of the sample), those in the control group exhibited lower levels of attachment security as compared to children in the treatment group at posttest. In addition, the study found that placement changes were significantly and inversely associated with attachment security and lower levels of attachment security were associated with higher scores on externalizing problems at a later assessment.

Long-Term: Two years after initial enrollment in the study, Spieker et al. (2014) found no overall effect of intervention condition on stability (i.e., placement with no interruptions or disruptions) or permanency (i.e., a stable placement ending with a legal discharge to the study caregiver); a significant interaction effect between type of caregiver and condition on permanency was detected. Compared to controls, receiving treatment increased the likelihood of permanency among children with foster/kin caregivers than among children with birth parents, and the size of this effect was large (OR = 9.67).

Study 1 Cont'd

Evaluation Methodology

Design:

Recruitment: The Nelson & Spieker (2013) study invited 57 of the 210 caregiver-child dyads randomized in the Spieker et al. (2012) study to participate in additional analyses of child cortisol levels. It is not clear how participants were selected from the full sample and if the randomization achieved in the full sample was maintained. Drawing from the same overall sample, researchers identified 56 biological parents and their recently reunified toddlers to conduct additional analyses comparing differences by condition in sleep problems (Oxford et al. 2013) and emotion regulation (Oxford, Marcenko et al., 2016).

Assignment: Of the 57 dyads invited to participate in the Nelson & Spieker (2013) study, 54 agreed and provided baseline saliva samples. The study states that 21 dyads were assigned to the treatment group and 25 were assigned to the control group. However, this only accounts for 46 participants, not the 48 who provided data at both time points or the 54 originally selected. For the Oxford et al. (2013) and Oxford, Marcenko et al. (2016) studies, 43 (treatment = 18 and control = 25) of the 56 dyads remained intact from enrollment to 6-month follow up and were therefore included in the analysis.

Attrition: Of the 54 participants who participated in the Nelson & Spieker (2013) study and provided baseline measurements, 2 caregivers dropped out and 4 children experienced a placement change by the postintervention measure. In addition, baseline morning cortisol samples were collected on 43 children and this data point was used as a covariate in the overall analysis. Of the 56 dyads identified for the Oxford et al. (2013) and Oxford, Marcenko et al. (2016) studies, 43 dyads remained intact and were included in the analysis.

Sample: Ages in the subsample reported in Nelson & Spieker (2013) ranged from 10 to 25 months, at baseline. In addition, 38% of children were living with a foster parent, 38% had been returned to a birth parent, and 25% were living with a family member. A majority of participants were white (68.8%), 17% were black, 13% were American Indian, and 2% were Hawaiian Native. The majority of the infants in the subsample in Oxford et al. (2013) and Oxford, Marcenko et al. (2016) were white (67%), female (53%), had a female caregiver (88%) and came from a household that earned less than $20,000 a year (60%).

Measures: In the Nelson & Spieker (2013) study, at both time points, authors collected five saliva samples: on arrival, just before the brief separation from caregiver, 30 minutes after the return of the caregiver, 45 minutes after the return of the caregiver, and the following morning. Cortisol levels were analyzed in each sample. Researchers established a conservative cut point of .05 ig/dL based on average variability within the sample to allow for capture of meaningful changes in cortisol level across collection points. The average ¼ SD for collection points ranged from .051 to .018. In this study, increasing cortisol levels in theoretically stressful conditions were considered a more adaptive behavior, while flat cortisol levels were considered potentially deleterious.

Oxford et al. (2013) pulled items from the Child Behavior Checklist and the Brief Infant Toddler Social and Emotional Assessment to create a 6-item measure of sleep problems that was completed by caregivers in treatment and control at baseline and 6-month follow up (sample α = .87). Separation distress was measured using the Toddler Attachment Sort-45 (inter-rater reliability, r = .92).

Measures used in the Oxford, Marcenko et al. (2016) study were the same as those used in the Spieker et al. (2012) study.

Analysis: Nelson & Spieker (2013) conducted multinomial logistic regression to analyze the association between posttest cortisol pattern and intervention group. The categorical outcome variable was increasing, decreasing, or flat cortisol levels during the course of the research visit. Analyses controlled for condition, time of day, child's age, baseline morning cortisol level, and flat or not flat cortisol pattern observed at baseline. OLS regression was used in the Oxford et al. (2013) study, which included sleep problems at baseline, age in months at enrollment, whether the child had experienced multiple removals from the biological parent's home, and time between baseline and 6-month follow up as covariates.

Oxford, Marcenko at al. (2016) conducted the same analysis and used the same covariates as Spieker et al. (2012).

Intent-to-Treat: Nelson & Spieker (2013) excluded baseline data from the 2 participants who dropped out of the study and had missing data for morning cortisol for an additional 5 participants. Meanwhile, Oxford et al. (2013) and Oxford, Marcenko et al. (2016) both dropped 13 dyads that did not remain intact during the study period. As no information was provided on handling of missing data, it is not clear if these studies followed intent-to-treat protocol.

Outcomes

Baseline Equivalence: Nelson & Spieker (2013) found no significant differences between the treatment and control groups for morning cortisol levels, however there were no statistical tests conducted on the demographic variables of this subsample. Oxford et al. (2013) found no demographic differences between groups at baseline, but did not test for baseline outcome differences. Oxford, Marcenko et al. (2016) reported no significant group differences in scores on measures at baseline, but did not clarify whether demographic and outcome measures were both tested.

Differential Attrition: None of the studies discussed differential attrition among the subsample.

Posttest: At posttest, Nelson & Spieker (2013) found that children in the treatment group were significantly more likely to have increasing postintervention cortisol levels. In addition, moderator analyses found that older children were more likely to have increasing postintervention cortisol levels. The study was not found to affect morning cortisol levels. Oxford et al. (2013) found that treatment predicted fewer sleep problems. Meanwhile, Oxford, Marcenko et al. (2016) found no program effects at posttest but a significant effect on observed parent support at the 6-month follow-up in favor of the treatment group.

Study 2

Evaluation Methodology

Design:

Recruitment: Participants were eligible if they were conversant in English, had housing, lived in a certain region within Washington state, and had a child between the ages of 10-24 months and an open child welfare case with an allegation of maltreatment. To recruit participants, a Department of Child and Family Services volunteer contacted families who had recently been reported to Child Protective Services (N=1,070). Of the total 1,070 families contacted, 504 could not be reached, 172 were not eligible, and 133 declined to participate. A total of 251 families agreed to a home visitation, during which an additional 4 were deemed ineligible.

Assignment: A total of 247 families were randomly assigned to the treatment (n=124) or the control (n=123) group in racial/ethnic blocks. The control condition received 3 phone consultations providing a needs assessment, mailed packet of personalized information, and two 10-minute follow-up check-in calls.

Attrition: Participants were assessed at four time points: 1) pretest (n = 247); 2) posttest (n = 225, 91% of the randomized sample); 3) 3-month follow-up (n = 215, 87%); and, 4) 6-month follow-up (n = 211, 86%).

Sample: A majority of parents in the study were mothers (91%). In terms of race, most parents were white (77%), while 3% of the sample parents were American Indian or Alaska Native, 4% were Asian, 5% were African American, and 11% were of a mixed race or "other." Around 20% of parents were Hispanic. Parent education included high school graduation (45%), GED (31%), and neither high school graduation nor GED (24%). Less than one third (31%) were employed full or part time, 28% were unemployed and looking, and 41% were not employed (i.e., were homemakers, students, retired, on disability, other). Over half of parents were never married (56%), 24% were married at the time of the study, and 22% were separated or divorced. Most of the sample received food stamps (79%) and average household income in the past year was $21,883, indicating lower socioeconomic status. Just over half of children in the study were male (54%) and were, on average, 16.37 months at baseline.

Measures: The study included 2 pre- and posttest measures of child behavioral outcomes and 1 child behavioral outcome measure administered at baseline and 3 months after the intervention (without posttest assessment). In addition, the authors used 3 measures of risk and protective factors for children, and 4 assessing caregiver outcomes.

Child behavioral outcomes included:

• Competence (sample α = .69-.70) and problem behavior (sample α = .77-.79), completed by caregivers at all four time points, were both assessed using the Brief Infant Toddler Social and Emotional Assessment.

Measures of risk & protective child outcomes included:

• The observer-rated Toddler Attachment Sort-45 scales of secure base behavior and atypical, affective communication conducted at all 3 time points (interrater reliability based on videotaped sessions for this measure was .75-.79)
• Emotional regulation and engagement/exploration from the Bayley Behavior Rating Scale (sample α ranged from .75 to .83) completed by researchers blind to condition at baseline and 3-month follow up.

Measures of caregiver outcomes assessing reciprocal parent-child warmth included:

• Understanding of toddlers (which measured caregivers' knowledge of infant and toddler social-emotional needs and developmentally appropriate expectations), assessed at posttest and 6-month follow-up using the Raising a Baby scale, a self-rated measure. Sample α = .73-.77.
• Parent sensitivity (α = .68-.72), assessed using the Nursing Child Assessment Teaching Scale, an observer-rated measure that was administered at posttest, 3-month follow-up, and 6-month follow-up. Parent sensitivity items represented aspects of positive interaction and indicators of mutuality (e.g., contingency, gaze, and positive affect), caregiver verbal and nonverbal support of child, and sensitive instruction during the teaching task.
• Dysfunctional interaction and parenting competence (α = .71-.94), assessed using the Parenting Stress Index and the Parenting Stress Index-Short Form, a self-reported measure, that was administered at 3-month and 6-month follow-up.

In addition, the study gathered official records on additional allegations of child maltreatment and removals from the birth parent home. For each child in the study, records were obtained of new allegations and removals that occurred between baseline and 12 months after the child's parent completed the intervention. For cases where the parent did not start or did not fully complete the intervention, information was obtained on allegations and removals that occurred within 12 months of when the parent would have completed the intervention if the parent had done so on schedule.

Analysis: The study used mixed model regressions to predict outcome variables, with intervention condition, age of child at baseline, and months between baseline and post-intervention assessment (which varied between conditions) as covariates. For measures assessed at only one posttest time point, the study used regression models predicting the posttest outcome measure. For measures with multiple posttest scores, 2-level random effects models were conducted in which posttest scores were nested within dyads. Survival analysis was conducted for models involving child maltreatment outcomes (i.e., new allegations and removal from birth home, as measured by child welfare administrative data).

Intent-to-Treat: For the child welfare outcomes, all study children were included in the analysis, consistent with an intent-to-treat model. For the data collected in the home, the study used an intent-to-treat protocol and employed Full Information Maximum Likelihood (FIML) protocols to account for missing data; however, infants who experienced a change in caregiver during the course of the study were excluded (which could be a violation of intent-to-treat protocol).

Outcomes

Implementation Fidelity: To ensure fidelity, the study used video-taped sessions to review providers' interactions with families. However, the study did not provide quantitative measures of these reviews. In the treatment condition, 86% of participants received all 10 sessions and 89% of control participants received all 3 telephone consultations.

Baseline Equivalence: The study reported there were no differences between conditions on any demographic variables at baseline. There was one difference in caregiver outcome at baseline in which the control group had significantly higher scores on parenting sensitivity.

Differential Attrition: There were no differences by completion status on demographic variables, but tests of differential attrition by outcomes were unclear.

Posttest: The study found no significant effects on the three behavioral outcomes (social-emotional competence, behavior problems and emotion regulation). Findings, however, showed a significant effect in favor of the treatment group on 1 of 3 child risk and protective outcomes at posttest: child atypical affective communication. In addition, within one-year post intervention, chances of removal from the birth parent home were greater for children in the control condition than children in the treatment condition at any given time.

For caregiver outcomes, the study found a significant effect on parent understanding of toddlers, which was only measured at posttest and 6-month follow-up, and parent sensitivity, measured at all four time points.

Long-Term: The study did not report long-term findings.

Study 3

Evaluation Methodology

Design:

Recruitment: Researchers recruited American Indian and Alaska Native families from the tribal health clinic, but also posted flyers in the community, publicized the study on Facebook, and provided information to Head Start teachers. To be eligible, participants had to have at least one parent or guardian who 1) was at least 18 years old; 2) spoke English; 3) was the primary caregiver for an Native American child aged 10-30 months; 4) had telephone access; 5) was not in a treatment facility or shelter; 6) was not hospitalized or imprisoned; 7) was willing to have researchers come to their home; and 8) lived on or near the reservation. If the household included multiple children in the target age range, the child with whom the caregiver wished to work was selected to participate in the study.

Assignment: Participants were 34 toddlers (10-30 months old) and their primary caregivers. Families were randomized to a treatment (n = 17) or waitlist control (n = 17) group. Researchers used an algorithm to randomize participants, blocking on male or female caregiver, in equal numbers to the treatment and control group. Waitlisted families in the control group had the opportunity to receive the intervention after the study assessments.

A problem with the design stems from the reliance on a single provider to deliver the program to all intervention participants. Page 11 states, "We trained the PFR Provider, who was a tribal member living on the reservation." The characteristics of the single PFR Provider were thus confounded with the program.

Assessments/Attrition: Families in both groups completed the follow-up assessment after the intervention period concluded. In the treatment group, the median time between baseline and follow-up assessments was 15 weeks (M = 18.00, SD = 5.29). In the control group, the median time between assessments was 17 weeks (M = 19.14, SD = 9.45). Of the 34 parent-child dyads who completed baseline assessments, 23 dyads completed a posttest (see consort diagram in Figure 1), for an overall attrition rate of 32%.

Sample: Of the total sample, 73% of the primary caregivers identified as American Indian (AI) or Alaska Native (AN), 15% as white, and 12% as mixed. All were living on the reservation, all of the non-Native caregivers had AI/AN partners or spouses, and all had an AI/AN child between 10 and 30 months old.

Measures:

The primary outcome measure assessed caregiver-child interaction (i.e., Reciprocal Parent-Child Warmth) and included the total score for Caregiver/Child Interaction on the Nursing Child Assessment Teaching Scale (NCATS), coded based on video recordings of the teaching task. A certified NCATS coder viewed the video recordings and scored them for caregiver sensitivity, stimulation of the child, and emotional responsiveness. The coder was blind to treatment group and time of assessment (baseline versus follow-up) and reviewed the videos in a randomized order provided by the investigators. The coder evaluated the caregiver's verbal and non-verbal support and the child's clarity of cues and response to the caregiver by responding. Researchers state that the NCATS has excellent psychometric properties, though no information on reliability or validity was reported.

The following measures were completed by caregivers (and thus are considered non-independent outcome measures):

• Caregivers' understanding of their children (Cronbach's α was .64 at baseline and .70 at follow-up.
• Parental stress (Cronbach's α was .91 at baseline and .93 at follow-up).
• Children's social-emotional competences, including Internalizing (α = .70 at baseline and .88 at follow-up), Externalizing (α = .73 and .76), and Competence (α = .87 and .93).

Analysis: Researchers used multiple regression models, adjusting for caregiver race, time elapsed between assessments, and the baseline score on the given measure.

Missing Data Method: Missing data were addressed using multiple imputation in SPSS (version 24) and STATA (version 14).

Intent-to-Treat: The authors stated they conducted "intent-to-treat" analyses.

Outcomes

Implementation Fidelity: After initial certification, the intervention group leader submitted one video recording of an intervention session per family to enable a master trainer to monitor ongoing fidelity by rating the quality and completeness of intervention delivery on a 5-point scale (M = 4.21, SD = 0.70 in the present study). Nine of the 17 caregivers in the treatment group completed all 10 visits.

Baseline Equivalence: T-tests found no significant baseline differences on any variables except the ITSEA Competence score. For this measure, the treatment had higher baseline scores compared to the control group. In addition, Table 1 shows some large non-significant differences between conditions (e.g., 65% female child in the intervention group versus 41% in the control group).

Differential Attrition: No tests of differential attrition were conducted. However, according to the What Works Clearinghouse standard, based on the overall attrition rate and the condition difference in attrition rates, the potential for attrition bias exists under both the cautious and optimistic thresholds. Attrition rates were 18% for the control group and 47% for the treatment group for a differential attrition rate of 29%. The use of multiple imputation may partly counter concerns over attrition bias.

Posttest: Relative to the control group, families in the treatment group showed significantly greater improvements in 3 of 5 behavioral outcomes. At the posttest, compared to the control group, participants in the treatment group earned a higher Caregiver/Child total score (d = 1.02) - which measures caregiver's verbal and non-verbal support and the child's clarity of cues and response to the caregiver - and Caregiver/Child Contingency total score (d = 1.21), which assesses the contingent responsiveness of the caregiver to the child, or vice versa.

In addition, significant findings in favor of the treatment group were detected in 1 of 2 risk and protective factors. Caregivers in the treatment group reported higher scores for Raising a Baby (d = 0.58), which measured caregivers' knowledge of the social-emotional needs of infants and toddlers as well as caregivers' level of developmentally appropriate expectations of their children.

Long-Term: Not reported.

Study 4

Evaluation Methodology

Design:

Recruitment: Researchers recruited participants through participating health centers. To be eligible, mothers needed to have received mental health treatment during pregnancy, be conversant in English or Spanish, have an infant under three months of age, have access to a telephone, planning to stay in the area until the child's first birthday, and to not have received the intervention previously. Each month, these health centers created a list of eligible mothers who were then contacted about the study.

Assignment: Participants consisted of 252 pairs of mothers and infants. These pairs were randomly assigned, blocked with preferred language (English or Spanish), to either receive the intervention or to a control group. The control group received a packet of resources available in the area, not including the intervention.

Assessments/Attrition: Overall attrition was less than 5% at both time points. That is, of those who completed the baseline assessment, 97% completed the 6 month assessment (posttest) and 96% completed the 12-month assessment (6-month follow-up).

Sample: Mothers ages ranged from 18 to 42 years. They were 66% White, 18% Black, and 16% other races. Forty-seven percent identified as Hispanic, and 33% preferred to read and speak in Spanish. Median family annual income was less than $20,000. To be eligible, mothers needed to have received mental health treatment during pregnancy. At baseline, 51% had mild to severe depression symptoms, 54% had mild to severe anxiety, and 35% had PTSD.

Measures:

Measures consisted of observational assessments of mothers interacting with their infants as well as some self-report measures completed by mothers.

Observational Measures

Parenting sensitivity was measured using the Nursing Child Assessment Teaching Scale (NCATS). For this task, mothers are instructed to attempt an activity with their infant that they had not yet demonstrated (e.g., transferring an object from one hand to another). A coder, blind to condition, coded the interaction on 50 possible behaviors that demonstrate parental sensitivity. Behaviors were scored as having occurred (1) or not (0) and summed to create a sensitivity score. A single bilingual and bicultural coder, blind to intervention condition, was trained to reliability by a certified NCATS instructor and passed biannual reliability checks with a minimum of 85% inter-rater agreement with a master trainer.

The Infant CARE-Index assessed parent-child interactions during play. A 3-5 minute play session was recorded and assessed by coders. The index produces a score for parent-child synchrony (dyadic synchrony), three measures of parent behavior (sensitivity, control, unresponsiveness) and four measures of child behavior (cooperation, compulsivity, difficultness, passivity). Coders had to meet a standard for inter-rater reliability of r = .80 on the measure of dyadic synchrony and an average of r = .70 across all scales. As a further check of reliability, over the course of coding approximately 20% of the videos were randomly selected to be coded by a reliability coder. Inter-rater correlations for dyadic synchrony between pairs of coders averaged r = .67. Meanwhile, inter-rater correlations for difficultness on the 20% of videos selected for coding by an additional coder averaged r = .49. The researchers only reported on dyadic synchrony and child difficultness.

Self-Report Measures

Mothers completed the Raising a Baby Scale, a measure of caregiver knowledge and understanding of infant and toddler social-emotional needs and developmental expectations (αs ranged from .68 to .71 across the three time points), and the Maternal Confidence Questionnaire, which measures the perceptions mothers have of their ability to care for and understand their infants with higher scores indicating more confidence (αs ranged from .66 to .76 across the three time points). At the final visit (infant age 12 months), mothers also completed the Infant Toddler Social Emotional Assessment (ITSEA). This measure has mothers rate their infants on an extensive list of behaviors. These behaviors produce composite scores of infant externalizing, internalizing, and dysregulation. This measure was only given at the final visit because it is inappropriate for infants younger than 12 months. The ITSEA has been standardized and normed on a nationally representative sample that was stratified to match the 2002 United States Census. In the study sample, alpha reliabilities for the domains were α = .81 for externalizing, α = .68 for internalizing, and α =.80 for dysregulation.

Analysis: Treatment effects were assessed at each time point using multiple linear regression controlling for preferred language (English or Spanish) and the baseline assessment of that outcome (for all outcomes except those assessed by the ITSEA).

Intent-to-Treat: The researchers followed an intent to treat approach. Missing covariate data were modeled using maximum likelihood estimation.

Outcomes

Implementation Fidelity: Providers were certified in the program before conducting home visits. In addition, providers were required to submit a videotaped recording of one session with each family that was assessed for program fidelity. The majority of videos (94%) met standards for fidelity; providers who did not received additional training. In addition, the majority of mothers in the treatment condition received the full program (82%).

Baseline Equivalence: Table 1 provides descriptive information on demographics by condition at baseline and Table 2 provides descriptive information about outcomes by condition at baseline. However, the researchers do not present significance tests of differences between conditions on any of these measures.

Differential Attrition: The researchers state that there were no significant differences in attrition by condition nor significant baseline-by-condition attrition effects. The researchers also state that "Mothers with incomplete data were more likely to be separated, divorced, or widowed, had lower incomes, and had more children at baseline" (page 8). No statistics are presented with this statement, so it is unclear as to whether or not these are statistically significant differential attrition effects. Also, overall attrition was less than 5%.

Posttest: Results showed significant differences in favor of the treatment group on 1 of 8 child behavioral outcomes, and on 2 of 5 risk & protective factors. Specifically, mothers in the treatment condition, compared to the control condition, reported significantly less externalizing behavior by their infants at 12 months of age (d = .28). In terms of risk & protective factors, mothers in the treatment condition demonstrated significantly better parental sensitivity than mothers in the control condition, both when their infants were 6 months old (d = .25) and when their infants were 12 months old (d = .26). Finally, mothers in the treatment condition reported significantly greater understanding of toddlers than mothers in the control condition, both when their infants were 6 months old (d = .21) and when their infants were 12 months old (d = .45).

Long-Term: Not examined.

Study 5

Evaluation Methodology

Design:

Recruitment: The study was conducted from 2017 to 2019. Researchers recruited Native caregiver-child dyads living on a rural reservation in the Northern Plains of the United States. Families were recruited at community events, from the Special Supplemental Nutrition Program for Women, Infants and Children (WIC), and via flyers/brochures, word of mouth, Facebook, and community programs (e.g., Head Start). Interested individuals were contacted by a Native Research Visitor, who provided detailed information on the study, determined eligibility, and then scheduled a home visit to obtain informed consent and collect baseline data. Family eligibility criteria included at least one parent or guardian who (1) was at least 18 years old and lived on or near the reservation; (2) spoke English; (3) was the primary caregiver for a child aged 10-31 months reported to be Native by their caregiver; (4) had telephone access; (5) was not living in a treatment facility or shelter, nor hospitalized or imprisoned; and (6) was willing to have study staff come to their home. If multiple children in the target age range lived in the household, the caregiver was asked to select one child to participate. A total of 458 families were assessed for eligibility, and 162 agreed to participate.

Assignment: Caregiver-child dyads were randomly assigned to the intervention group (n = 81) or a Resource and Referral (RR) control group (n = 81) after baseline data collection. An algorithm was used to randomize participants, blocking on male or female caregiver, in equal numbers to study conditions. The intervention group received an adapted version of the Promoting First Relationships® program for Native families that, similar to the original version, was delivered across 10-weekly home visits (with an average completion time of 14 weeks). The authors noted that the adaptations (e.g., changing the study name and logo to align with the Tribe's culture and language) did not change the basic format, structure, or principles of the program. The control group service consisted of three phone contacts between a local Native staff member and primary caregivers, and staff also provided families with tailored follow-up information on local services for their needs.

Assessments/Attrition: Assessments were at baseline, immediate posttest, and three months after the intervention end. At both posttest and follow-up time points, overall attrition was 32.7% (n=53).

Sample: Participants were primary caregivers (93% female) and their toddlers (10-31 months old; M = 20.10; 50% female). The majority of the primary caregivers were biological mothers of the study children (75%). Other primary caregivers were a female relative (16%), the biological father (6%), a male relative (<1%), other female (2%), or other male (<1%). Almost all caregivers (96%) self-reported as American Indian or Alaska Native, 62% were married or partnered, and 45% had a high school education or below. All of the children were reported to be American Indian or Alaska Native.

Measures: Study outcomes included independent observational assessments of caregiver-child interaction quality (primary outcome) and non-independent survey measures completed by caregivers (secondary outcomes). Caregiver measures were reports of child externalizing behavior and two risk and protective factors (knowledge of children's social-emotional needs and caregiver depressive symptoms). The measures came from well-known sources, and the authors reported acceptable reliability coefficients for the current sample.

Analysis: Repeated-measures multivariate analyses of covariance (MANCOVAs) were used to test for intervention effects, controlling for baseline outcomes. The authors conducted three sets of outcome analyses to examine overall post-intervention effects, as well as separate effects at posttest and 3-month follow-up.

Missing Data Strategy: The study excluded participants without follow up data (i.e., listwise deletion or complete case analysis) and did not use missing data strategies.

Intent-to-Treat: The researchers conducted an intent-to-treat analysis.

Outcomes

Implementation Fidelity: The authors stated that intervention providers passed every fidelity test, with a mean global rating of 4.88 out of 5 points. All caregivers in the intervention group agreed or strongly agreed that they were satisfied with the program (M = 3.90), and all agreed or strongly agreed that they would recommend the intervention to other caregivers (M = 3.90). In the intervention group, 51% of the participants completed all 10 visits, 45% withdrew from the study before starting the program or receiving only one visit, and 4% dropped out of the study after three or more visits. In the control group, 88% completed all of the calls, and 12% withdrew from the study before or during the implementation period.

Baseline Equivalence: The authors stated that there were no statistically significant differences between the intervention and control groups on any demographic variables at baseline, though no significance tests were presented. There was no mention of tests for baseline outcomes but the means for the intervention and control groups in Table 2 (p. 45) appear similar.

Differential Attrition: The study did not test for differential attrition. However, according to the What Works Clearinghouse standard, based on the overall attrition rate and the condition difference in attrition rates, the potential for attrition bias exists under both the cautious and optimistic thresholds. Attrition rates by condition were 49% for the intervention group and 16% for the control group for a differential rate of 33%.

Posttest: After controlling for baseline outcomes, analyses indicated an overall post-intervention effect on observed caregiver-child interaction quality (η2 = .07) but not for child externalizing behaviors. At specific follow-up time points, caregiver-child interaction quality was significantly higher at 3-month follow-up (but not posttest) for the intervention group (η2 = .06) compared to the control group. There were no significant intervention effects on child externalizing behaviors at either posttest or 3-month follow-up.

For risk and protective factors, results indicated a significant overall post-intervention effect on caregiver knowledge of children's social-emotional needs but not for caregiver depressive symptoms. Findings at specific follow-up time points showed that caregivers in the intervention group, compared to those in the control group, had significantly higher knowledge of children's social-emotional needs and less severe depressive symptoms at both posttest and 3-month follow-up.

Exploratory moderation tests showed that the effect of the intervention on caregiver-child interaction quality at 3-month follow-up was stronger for caregivers with low levels of baseline depressive symptoms.

Long-Term: Not examined.