Europe PMC

This website requires cookies, and the limited processing of your personal data in order to function. By using the site you are agreeing to this as outlined in our privacy notice and cookie policy.

Early Life Stress: Effects on the Regulation of Anxiety Expression in Children and Adolescents.

Author information

Affiliations

  • 1. Department of Human Development and Quantitative Methodology, University of Maryland, College Park.
      Authors
    • Burkholder AR 1
    (1 author)
  • 2. Institute of Child Development, University of Minnesota, Twin Cities.
      Authors
    • Koss KJ 2
    • Gunnar MR 2
    (2 authors)
  • 3. Institute for Policy Research, Northwestern University.
      Authors
    • Hostinar CE 3
    (1 author)
  • 4. Department of Psychology, St. Olaf College.
      Authors
    • Johnson AE 4
    (1 author)

ORCIDs linked to this article

Social Development (Oxford, England), 06 Nov 2015, 25(4):777-793
https://doi.org/10.1111/sode.12170 PMID: 28584408 PMCID: PMC5454775

Free full text in Europe PMC

Abstract 

This study examined children's (N = 79; 9-10 years) and adolescents' (N = 82; 15-16 years) ability to regulate their emotion expressions of anxiety as they completed a modified version of the Trier Social Stress Test for Children (TSST-C). Approximately half in each age group were internationally adopted from institutional care (N = 79) and half were non-adopted, age-matched peers (N = 82). Institutional care was viewed as a form of early life stress. Coders who were reliable and blind to group status watched videos of the session to assess anxiety expressions using the Child and Adolescent Stress and Emotion Scale developed for this study. Children exhibited more expressions of anxiety than adolescents, and youth adopted from institutions showed more expressions of anxiety than their non-adopted counterparts. The role of early life stress on observed anxiety expressions remained significant after controlling for differences in age, physiological stress responses measured through salivary cortisol reactivity, and self-reports of stress during the TSST-C. This suggests possible deficits in the regulation of expressive behavior for youth with early life stress histories, which cannot be explained by experiencing the task as more stressful.

Free full text 

Logo of nihpaLink to Publisher's site
Soc Dev. Author manuscript; available in PMC 2017 Nov 1.
Published in final edited form as:
Soc Dev. 2016 Nov; 25(4): 777–793.
Published online 2015 Nov 6. https://doi.org/10.1111/sode.12170
PMCID: PMC5454775
NIHMSID: NIHMS856970
PMID: 28584408

Early Life Stress: Effects on the Regulation of Anxiety Expression in Children and Adolescents

Amanda R Burkholder, Kalsea J. Koss, Camelia E Hostinar, Anna E. Johnson, and Megan R Gunnar
Author information Copyright and License information Disclaimer

Abstract

This study examined children’s (N = 79; 9–10 years) and adolescents’ (N = 82; 15–16 years) ability to regulate their emotion expressions of anxiety as they completed a modified version of the Trier Social Stress Test for Children (TSST-C). Approximately half in each age group were internationally adopted from institutional care (N = 79) and half were non-adopted, age-matched peers (N = 82). Institutional care was viewed as a form of early life stress. Coders who were reliable and blind to group status watched videos of the session to assess anxiety expressions using the Child and Adolescent Stress and Emotion Scale developed for this study. Children exhibited more expressions of anxiety than adolescents, and youth adopted from institutions showed more expressions of anxiety than their non-adopted counterparts. The role of early life stress on observed anxiety expressions remained significant after controlling for differences in age, physiological stress responses measured through salivary cortisol reactivity, and self-reports of stress during the TSST-C. This suggests possible deficits in the regulation of expressive behavior for youth with early life stress histories, which cannot be explained by experiencing the task as more stressful.

Emotion regulation, or the ability to regulate and manage one’s emotional responses, is an important aspect of social development. Development of effective emotion regulation has implications for social competence and adjustment in childhood and adolescence in normative populations (Bakracevic Vukman, 2005; Blair & Diamond, 2008; Carlo, Crockett, Wolff, & Beal, 2012; Eisenberg, Spinrad, & Eggum, 2010). In fact, the inability to effectively regulate emotional behavior is related to increases in inattention and hyperactive behavior (Martel, Nigg, & Von Eye, 2009). These deficits may even have a significant impact on school success (Blair & Diamond, 2008). On the other hand, emotion regulation skills are associated with increases in prosocial behaviors, especially in late childhood and adolescence (Carlo et al., 2012). Successfully engaging with others requires effective regulation of the individual’s own emotions and emotional behavior, suggesting that positive social relationships may in part depend on the ability to adequately regulate emotional responses. Indeed, emotion regulation is predictive of successful social relationships throughout childhood and adolescence (Kim & Cicchetti, 2010).

Differences in the development of emotion regulation, in addition to internal emotion states, may be important to examine in order to make predictions about child outcomes. Typically, the ability to regulate emotions develops fastest during the first years of life then slows to more gradual increases in competence from middle childhood to early adulthood (Eisenberg, Spinrad et al., 2010; Kochanska, Murray, & Harlan, 2000). These improvements correlate with the development of the prefrontal cortex and executive functions such as inhibitory control and attention (Bakracevic Vukman & Licardo, 2009). These early developing emotion regulation skills lay the groundwork for later emotion regulation competencies.

Caregivers and other environmental influences play a large role in the development of emotion regulation (Blair et al., 2008; Kochanska et al., 2000; Kopp & Neufeld, 2003; Rothbart & Bates, 2006). Responsive caregiving has been shown to have a significant influence on the development and maintenance of regulatory behaviors (Blair et al., 2008; Kochanska et al., 2000; Kopp & Neufeld, 2003; Rothbart & Bates, 2006). In infancy, emotions are regulated almost entirely by an external source, usually a caregiver (Kopp & Neufeld, 2003). Because most infants are not yet able to effectively self-regulate emotions, this social regulation creates a zone of proximal development within which children slowly learn the process of regulating their own emotional behavior (Kochanska et al., 2000; Kopp & Neufeld, 2003). Parent socialization of emotion regulation is important both in external mediation of the child’s emotional responses and in effective modeling of socially acceptable behaviors (Rothbart & Bates, 2006).

Environments that do not provide enough care, attention, or support from parents or other caregivers early in life may be associated with deficits in emotion regulation later in childhood and adolescence (Maughan & Cicchetti, 2002; Shields, Cicchetti, & Ryan, 1994; Shipman & Zeman, 2001; Tottenham et al., 2010). Infants lacking stable, consistent caregivers may not receive the responsive social scaffolding necessary to regulate their emotions (McCall, 2013). Children reared in these environments may not develop adequate emotion regulatory skills. However, less is known about how emotion regulation skills may adjust following a change in caregiving. Do these skills rebound once children are removed from deprived care and given supportive parents? Although children adopted from conditions of deprivation show initial gains in emotional and social skills, many children continue to exhibit deficits in these domains that persist or increase over time (Bos et al., 2011; Hodges & Tizard, 1978; Kumsta et al., 2010). Therefore, emotional regulatory competencies may not fully rebound after children’s social conditions improve.

Additionally, individuals who have experienced early life stress show difficulties in other areas of cognitive functioning, such as executive functioning, that correlate with the development of emotion regulation (e.g., Pollak et al., 2010). Inhibitory control, or the ability to inhibit undesired responses, is an important executive function (Bruce, Tarullo, & Gunnar, 2009). Individual differences in inhibitory control are associated with children’s ability to regulate their emotions (Bakracevic Vukman & Licardo, 2009; Carlson & Wang, 2007). Indeed, emotion regulation has also been shown to correlate with neural activation patterns associated with exerting inhibitory control (Lewis, Lamm, Segalowitz, Stieben, & Zelazo, 2006). Therefore, if children adopted from conditions of early life stress exhibit less competent emotion regulation, this may in part be due to problems in inhibitory control.

Emotion regulation has been identified as a key mediator between early experience and normative functioning in adulthood (Fonagy & Target, 2002). However, we have little evidence that early life stress, without continued poor care, has a long-term effect on emotion regulation abilities into late childhood and adolescence. Although research suggests that emotion regulation becomes relatively stable by the second year of life (Eisenberg, Spinrad et al., 2010), the work supporting these conclusions has been done with children in fairly consistent living conditions. Thus it is possible that experiences of deprivation early in life could have significant ongoing effects on emotion regulation abilities, despite a drastic change in caregiving after infancy. In order to isolate the impact of early experience on the development of emotion regulation separate from continued adversity, this study focused on post-institutionalized children as a model for the impact of early life stress. Internationally adopted youth typically experience inadequate or adverse care in orphanages or institutions followed by removal and placement into responsive care post-adoption (Tarullo, Bruce, & Gunnar, 2007). Studies of parenting in families that adopt children internationally from orphanages generally show that parenting quality is in the good to excellent range (Garvin, Tarullo, Van Ryzin, & Gunnar, 2012), thus there is likely a vast shift in caregiving from orphanage care to adoptive family care. Therefore, children adopted internationally from institutions provide information regarding the long-term impact of early deprived care on emotion regulation several years after the transition to a supportive family environment.

This study used new measures and analyses to examine a sample of participants also included in an earlier study (see Hostinar, Johnson, & Gunnar, 2015). The goal of this investigation was to examine whether children and adolescents adopted from conditions of early life stress were less competent at regulating their expressions of anxiety during a stress-eliciting task. Children and adolescents’ anxiety expressions were scored as they executed the Trier Social Stress Test, a socio-evaluative laboratory stressor involving a public speaking task and a mental arithmetic task. Both tasks are filmed and the participants are told that the tasks will be scored by judges and shown to peers. The goal is to deliver the talk without displaying signs of stress and nervousness. Although participants were not explicitly told to regulate their emotions, doing well on the task requires them to manage their observable stress and anxiety to receive high ratings from the judges. Thus youth’s anxiety-related emotion expressions serve as evidence for their differing ability to regulate emotional expressions.

To help rule out the possibility that group differences were due to the task being more stressful for the early life stress group, we examined group differences in emotion expression while controlling for physiological (i.e. cortisol) and self-reported indices of perceived stress. Next we examined whether adolescents would be better at regulating their emotion expressions than children, and whether the difference between youth with early life stress backgrounds and non-adopted youth changed with age. Finally, we examined whether differences in inhibitory control mediated the effect of age and group status, suggesting that poor inhibitory control contributed to both children and adopted youths’ reduced ability to regulate their emotion expressions.

Method

Participants

Participants included 161 youth in two groups: children and adolescents adopted from orphanages or other institutions termed the early life stress group (ELS, N = 79) and children and adolescents reared in their birth families termed the non-adopted group (NA, N = 82). Individuals in the ELS group spent at least 6 months in institutional care prior to adoption (duration M = 21.33 months, SD = 12.89; M = 26.54 months of age at adoption, SD = 15.44). Participants were recruited to reflect two different age groups: 9 and 10 year olds (categorized as children; M = 9.83 years at testing, SD = .55) and 15 and 16 year olds (categorized as adolescents; M = 15.81 years at testing, SD = .59). The Ns in each group were: 39 ELS children (20 female), 40 ELS adolescents (22 female), 40 NA children (20 female), and 42 NA adolescents (20 female).

The ELS group was recruited from a registry of families who had adopted children internationally in the upper Midwest region. Regions of origin were: Indo-Europe (N = 50), Southeast Asia (N = 25), Latin America (N = 3), and Africa (N = 1). NA participants were recruited from a registry of parents in a major Midwest metropolitan region who were interested in having their children participate in research (region of origin for all NAs = United States). In the ELS group, about 80% of contacted families agreed to participate in the study. In the NA group, about 50% of the contacted families agreed to participate in the study. These percentages are similar to previous studies conducted within these populations. Strict exclusion criteria were followed to ensure that the recruited groups were representative of the ELS and NA populations. Participants were excluded if they had a diagnosis of fetal alcohol syndrome, any pervasive developmental disorder, or autism. Both groups were also screened for steroid use, as these medications may interfere with salivary cortisol levels.

Group differences in key demographic variables were examined, including household income, parental education, parental age, and marital status. There was no significant difference in household income level among ELS and NA children (F(1,157) = 0.81, p = 0.13) where both groups’ median annual household income averaged between $75,001 and $125,000. There was also no difference in parental education [education level of parent in attendance (χ2(5) = 5.33, p = .38), education level of parent not in attendance (χ2(5) = 4.81, p = .55)], where the median highest level of education attained in both groups was a bachelor’s degree. The groups did differ on parental age, such that ELS families had older parents than NA families [age of parent in attendance (F(1,157) = 46.73, p < .001; ELS M = 51.19 years, SD = 6.19; NA M = 44.55 years, SD = 6.08), age of parent not in attendance (F(1,135) = 24.18, p < .001; ELS M = 51.48 years, SD = 7.62; NA M = 45.30 years, SD = 7.17)]. The groups also differed on marital status, such that ELS families reported lower rates of married parents than NA families (74.4% and 95.0% respectively, χ2(3) = 16.66, p < .001). The latter is typical of families that adopt internationally and likely reflects marital status at the time of adoption.

Procedure

The participants were accompanied by their parents to a laboratory session scheduled between the hours of 3:30 and 4:30 PM in order to control for daily fluctuations in cortisol levels. The session consisted of a modified version of the Trier Social Stress Test for Children, saliva collection, self-report levels of emotional arousal, and parent questionnaires. The study was approved by the University’s Institutional Review Board and parents and youth provided informed consent and assent, respectively.

Trier Social Stress Test for Children

A modified version of the Trier Social Stress Test for Children (TSST-C)—a laboratory social and cognitive stressor—was administered to induce stress and emotional arousal in the participants (Buske-Kirschbaum et al., 1997; Yim, Quas, Cahill, & Hayakawa, 2010). The TSST-C consists of a public speaking and arithmetic task that is performed in front of a perceived audience of judges. After arrival, participants were given a 25-minute period of relaxation to minimize any stress caused by travel or the new environment. After the waiting period, youth were taken into another room to begin the task. The task consisted of a 5-minute speech preparation period, followed by a 5-minute public speaking task and a 5-minute oral subtraction task. Youth performed the speech and subtraction tasks in a room alone, facing a two-way mirror and a noticeable camera. They were informed that teachers were sitting behind the two-way mirror and would be judging the speech and mathematical portions for content and accuracy. Additionally, participants were told that the session was being recorded and would be shown to a classroom of their peers who would judge their performance against other youth of their same age. In reality, participants were played a digital recording of two teachers introducing themselves and giving instructions about the task. For the public speaking section, youth were instructed to prepare a speech that would be a sufficient introduction of themselves, including “one good thing and one bad thing about yourself.” If a participant did not freely fill the whole 5 minutes of the task, s/he was prompted to continue by the experimenter. After the conclusion of the speech, youth completed the subtraction portion of the TSST-C. Children were required to subtract by 3s serially from 307. Adolescents were required to subtract by 7s serially from 758. If the participant got an answer wrong, they were asked to start over from the beginning. After 5 minutes, the experimenter ended the arithmetic section. At the end of the session, youth were debriefed. In addition, they were given positive feedback about their work and compensated for their time.

Anxiety expression

The speech and arithmetic sections of the TSST-C were recorded and coded using the Child and Adolescent Stress and Emotion Scale (CASES) developed for this study (see appendix). Due to video failure, 9 participants (4 ELS, 5 NA) were excluded from this analysis, leaving 152 participants. The CASES captured observed emotional behaviors on four indices (positive emotions, sadness/worry, anger/frustration, and anxiety-related emotions), though anxiety was the most prominent negative emotion and the focus of this analysis. Although all emotions were measured, they were not analyzed nor reported, as these emotions were not the goal of the present study. Three channels of anxiety expression were measured including bodily expressions (e.g., fidgeting and self-touch), vocal expressions (e.g., vocal quaking), and facial expressions (e.g., wincing or furrowed brow). Each participant was scored on a 4-point Likert scale (0 = no observed expressions of anxiety, 1 = mild expressions, 2 = moderate expressions, 3 = severe expressions) for each of the three channels of expression. These scores were averaged to create an overall anxiety expression score. Two coders blind to the participant’s prior history scored the behaviors. The present study had good inter-rater reliability (28 overlapping tapes; ICC= 0.80).

Cortisol reactivity

Salivary cortisol was collected at 0, 20, 40, and 60 minutes after the TSST-C. In the present analysis, the samples collected at 0 and 20 minutes after the task were used, representing reactivity to the TSST-C (for raw means, see Table 1; for a timeline of events in relation to saliva sample collection, see Figure 1). Participants were instructed to refrain from eating large, protein-filled meals or consuming caffeine for at least 2 hours before each session (Slag, Ahmed, Gannon, & Nuttall, 1981). The samples were collected by having the participant expectorate through a straw into labeled vials that were kept frozen at −20° Celsius until assayed. After study completion, samples were sent to Trier, Germany and were assayed in duplicate using a time-resolved fluorescence immunoassay (DELPHIA); intra-assay CV < 7%, inter-assay CV< 10%. All samples were included in the same assay and batches were matched for age and group. Cortisol reactivity was defined as the difference between the initial sample and a sample collected 20 minutes after the TSST-C; given the delay for peak cortisol levels to appear in saliva.

An external file that holds a picture, illustration, etc. Object name is nihms856970f1.jpg

Timeline of study events.

Table 1

Descriptive Statistics for Study Variables by Age and Group Status

ELSNA
Child
M (SD)		Adolescent
M (SD)		Child
M (SD)		Adolescent
M (SD)		F
Self-Reported Stress-Speech3.17 (1.34)3.41 (1.27)3.85 (.75)3.45 (1.09)F(3, 152) = 2.35
Self-Reported Stress-Math3.61 (1.23)3.72 (1.23)3.87 (1.20)3.76 (1.01)F(3, 152) = .32
Cortisol Sample 13.06 (2.24)4.43 (3.47)3.31 (2.43)4.67 (2.37)F(3, 156) = 3.58*
Cortisol Sample 23.76 (3.41)4.62 (3.74)3.73 (3.29)5.92 (3.82)F(3, 156) = 3.38*
Cortisol Reactivity  .70 (3.70)  .19 (2.20)  .42 (2.70)1.25 (2.97)F(3, 153) = .98
Inhibitory Control3.43 (.69)3.58 (.68)3.66 (.62)4.08 (.58)F(3,153) = 7.53***
Observed Anxiety Expression1.63 (.50)1.33 (.38)1.40 (.49)  .92 (.41)F(3,148) = 17.48***

Note.

*= p < .05,
***= p < .001

Self-reported emotion

Participants were asked to rate their perceived levels of stress during the speech and arithmetic portions of the TSST-C on the arousal dimension of the Self-Assessment Manikin (SAM); the SAM is a five-point pictorial scale (Bradley & Lang, 1994; Lang, 1980). Specifically, they noted how “stressed” they felt while giving the speech and how “stressed” they felt while doing the math (1 calm; 5 high stress). Participants separately rated the public speaking and arithmetic portions of the TSST-C 20–30 minutes after the task was completed and before they were debriefed. The mean level of self-reported stress during the speech portion was 3.49 (SD = 1.14); the mean level of self-reported stress during the arithmetic portion was 3.79 (SD = 1.13). The two scores were significantly correlated (r = .37, p < .001).

Inhibitory control

Parents completed the Inhibitory Control subscale of the Early Adolescent Temperament Questionnaire-Revised (EATQ-R; Ellis & Rothbart, 2001). The inhibitory control scale assesses the ability to suppress undesirable responses and plan before acting. Sample items include “your son or daughter is able to stop him/herself from laughing at inappropriate times” and “your son or daughter has a hard time waiting his/her turn to speak when excited.” Parents rated 5-items (3 reverse-scored) on a 5-point Likert scale ranging from almost always untrue to almost always true. These responses were averaged to produce an inhibitory control measure, higher scores indicating a greater capacity to plan and suppress undesirable emotions. In the present study there was adequate internal reliability (Cronbach’s α = .74).

Results

Descriptive statistics for all study variables are displayed by group in Tables 1 and ​and2.2. Observed anxiety expression was significantly correlated with a parent-reported measure of inhibitory control, but not significantly correlated with self-reported measures of stress and physiological reactivity during the TSST-C. The self-reported stress levels during both the speech and math sections of the TSST-C were highly correlated. A one-way between subjects analysis of variance (ANOVA) was conducted examining sex differences in observed anxiety expression; no sex differences were found (F(1,151) = 1.76, p = .42). Thus, sex was not included in subsequent analyses.

Table 2

Zero-Order Correlations and Descriptive Statistics for Main Study Variables

12345
1. Self-Reported Stress-Speech1.00
2. Self-Reported Stress-Math  .39***1.00
3. Cortisol Reactivity  .07  .021.00
4. Inhibitory Control  .12−.03  .24**1.00
5. Observed Anxiety Expression−.00−.04−.04−.35***1.00
M3.493.79  .023.701.51
SD1.141.13  .11  .68  .50

Note.

**p < .01,
***p < .001.

To test the effect of group status (ELS vs NA) and age (children vs adolescents), a two-way between-subjects ANOVA was performed (see Table 3). A main effect was found for group status such that ELS youth had higher scores of observed anxiety expression than NA youth. Additionally, there was a main effect of age on observed anxiety expression such that children had higher scores than adolescents. Group status and age did not significantly interact to predict observed anxiety expression.

Table 3

Summary Table for Two-Way Analysis of Variance of the Effects of Group Status and Age on Observed Anxiety Expression

dfSSMSFpη2
Group Status14.004.0019.98<.001.12
Age15.715.7128.49<.001.16
Group Status × Age1.31.312.07.219.01
Within Cells14829.65.20
Total152301.56

A hierarchical multiple regression analysis was conducted to evaluate the impact of group status and age on observed anxiety expression while controlling for physiological (cortisol) and self-reported indices of stress experienced during the stressor task (see Table 4). Age group (0 = children, 1 = adolescents) was entered at the first step, self-reported stress during the speech and math tasks were entered at the second step, cortisol reactivity was entered at the third step, and group status (0 = NA, 1 = ELS) was entered at the final step. The linear combination of these predictors was significantly related to observed anxiety expression and accounted for a modest amount of variance in observed anxiety expression in the final step of the regression (see Table 4). Even after statistically controlling for physiological and self-reported indices of stress, group status significantly predicted observed anxiety expression levels. Additionally, age was a significant predictor of observed anxiety expression.

Table 4

Summary of Hierarchical Regression Results Predicting Observed Anxiety Expression

Model 1Model 2Model 3Model 4
VariableB (SE)βB (SE)βB (SE)ΒB (SE)β
Age Group.39 (.08).38***  .39 (.08)  .38***  .39 (.08)  .38***  .39 (.07)  .38***
Self-Reported Stress Speech−.01 (.04)−.01−.01 (.04)−.01  .02 (.04)  .03
Self-Reported Stress Math−.01 (.04)−.02−.01 (.04)−.02−.01 (.04)−.02
Cortisol Reactivity−.00 (.01)−.02  .00 (.01)  .00
Group Status  .34 (.08)−.33***
R2.15  .15  .15  .25
F for change in R2F(1,145)=25.11***F(3,143)=.08F(4,142)=.07F(5,141)=19.74***

Note.

***= p <.001

Inhibitory control was examined as a mediator between both group status and age and participants’ observed anxiety expression (see Figure 2). ELS youth (β = −.27, p < .001) exhibited less inhibitory control compared to the NA group. Across groups, adolescents (β = .21, p < .01) showed better inhibitory control compared to children. Better inhibitory control was in turn associated with lower levels of observable anxiety expression (β = −.21, p < .01). As a test of mediation, bootstrapping was conducted (10000 bootstrap samples) in MPLUS (v. 7; Múthen & Múthen, 1998–2012). Bootstrapping can yield more accurate estimates of the standard errors of the indirect effect (Shrout & Bolger, 2002). An effect size for the indirect effect was calculated using the proportion of the maximum indirect effect (Preacher & Kelley, 2011). The 95% confidence interval for group status did not contain zero (CI: .02, .13; K2=.06), indicating a significant indirect effect (see Figure 2). Additionally, the 95% confidence interval for age group did not contain zero (CI: −.01, −.11; K2 = .05) indicating a significant indirect effect.

An external file that holds a picture, illustration, etc. Object name is nihms856970f2.jpg

Inhibitory control as a mediator between both group status and age group and observed anxiety expressions. Standardized parameter estimates reported. Group status (0 = Non-Adopted, 1= Early Life Stress); age group (0 = children, 1 = adolescents). Estimates in parentheses depict the direct effect of institutionalization and age on observed anxiety expressions prior to the inclusion of the mediator.

Note. **p < .01, ***p < .001.

Duration of Institutional Care

Because both adoption status and duration of institutional care were of theoretical interest, we considered including both variables in the regression model by assigning scores of “0” to all non-adopted, comparison youth. However, this resulted in a very non-normal distribution of scores on the “duration of institutional care” variable and created problems of multicollinearity because the group (ELS vs. NA) and duration of institutional care were reflections of one another (r = .75, p > .001). Therefore, we focused on group status (ELS vs. NA) in analyses of the total sample. Note that when only the adopted youth were analyzed, duration of institutional care was not significantly correlated with the anxiety expression variable (r = .11, p = .37), nor with other variables of interest, including age group (r = .06, p = .60), self-report measures of stress during the speech task (r = .19, p = .10), self-report measures of stress during the math task (r = .06, p = .63), cortisol reactivity (r = −.04, p = .76), and inhibitory control (r = .01, p = .94). Duration of institutional care was highly correlated with age at adoption (r = .76, p < .001).

Discussion

These results revealed that children and adolescents who experienced early life stress in the form of institutional care had higher levels of observed anxiety expression than non-adopted youth in response to a socio-evaluative stressor. Furthermore, the results remained significant after controlling for cortisol reactivity and participants’ perceived level of stress. This suggests that the differences in anxiety expression may have more to do with difficulties in regulating behavior than reflecting differences in the intensity of stress experienced. Additionally, children expressed more anxiety during the stressor than did adolescents, consistent with expectations that children get better at regulating the expression of emotions with development (Eisenberg, Spinrad et al., 2010). Furthermore, because the ELS youth in this study likely lived in stable, well-resourced families for many years after infancy (reflecting a vast change in early caregiving), these results raise the possibility of a sensitive period early in life during which stressful conditions of care may produce lasting problems in the regulation of emotional expression. Finally, it is possible that this sensitive period affects the regulation of expressed emotion through effects on inhibitory control. We found evidence of an indirect effect of early life stress operating through inhibitory control on anxiety expression.

One potential mechanism for continued deficits in the regulation of emotion expressions is alteration in brain development in youth adopted internationally from institutional care. Structural (Tottenham et al., 2010) and functional (Tottenham et al., 2011) differences in the amygdala and other portions of the limbic system, a brain region associated with emotion, have been observed in children who have experienced early life stress. Further, the connection between the limbic system and the prefrontal cortex is weakened in children with early stressful experiences (Govindan, Behen, Helder, Makki, & Chugani, 2010), suggesting that difficulty in effectively regulating emotions could be accounted for by the inability of the prefrontal cortex to act as a “suppressor” to the limbic system. This decreased communication between the two brain regions may account for the inability to minimize the expression of inappropriate or undesirable emotions. However, if this were the basis for the current findings, we would expect to find that children and adolescents who had experienced early life stress would have also exhibited larger cortisol responses to the TSST-C and reported feeling more stressed. However, this is not what we found.

We did find evidence that a parent-reported measure of inhibitory control mediated these relations. Previous research shows that children who had experienced early life stress also showed less inhibitory control and delays in goal-directed behavior (Hostinar, Stellern, Schaefer, Carlson, & Gunnar, 2012). Attention-deficit/hyperactivity disorder (ADHD) has long been associated with deficits in inhibitory control (Schachar, Mota, Logan, Tannock, & Klim, 2000; Schachar, Tannock, Marriott, & Logan, 1995). Indeed, ELS children exhibit problems with inattention and over-activity at a greater rate than typically developing populations (Rutter, Kreppner, O’Connor, & the English and Romanian Adoptees Study Team, 2001), and children who have experienced early life stress are diagnosed with ADHD at higher rates than the general population (Lindbald, Weitoft, & Hjern, 2010). However, we should note that the present study used a parent-report measure of inhibitory control derived from the EATQ-R. Although such measures have been shown to correlate with laboratory tasks of inhibitory control, we might have obtained different results had we included neuropsychological tests of inhibitory control along with our parent-report measure.

It should be noted that there were no age group differences on duration of institutional care within the ELS group. Therefore, ELS adolescents have likely lived in their adoptive family homes longer than children, and are thus exposed to more overall instances of sensitive care. This could account for differences in emotion regulation abilities in ELS children and adolescents. However, although the interaction of group and age approached significance, at this point we have no conclusive evidence that age moderates the relation between early life stress and emotion expression regulatory competence. The failure to achieve traditional levels of significance might well be due to low power to assess this interaction. In the future, this study should be replicated with sufficient power to test for the interaction of age and group.

Deficits in emotion regulation may have repercussions for social functioning in late childhood and adolescence. Effective emotion regulation is associated with overall social competence. Specifically, emotion regulation abilities have been linked to success in peer relationships and generally positive social skills (Blair & Diamond, 2008; Eisenberg, Valiente, & Eggum, 2010). Emotion regulation is also positively correlated with prosocial behavior in late childhood and adolescence (Carlo et al., 2012). Furthermore, deficits in the ability to regulate emotions are related to many forms of maladjustment, including externalizing and internalizing problems (Eisenberg, Spinrad et al., 2010). Therefore, less regulated emotionality may account for many problems that early life stress may create during the late childhood and adolescent years.

Because deficits in emotion regulation may have negative implications for development and social functioning, it is necessary for future studies to focus on developing intervention programs to improve emotion regulatory skills in youth who have experienced early life stress. Current interventions exist that could be adapted to fit this population. Interventions targeting emotion regulation and other aspects of executive function have been effective in the Head Start preschool program (Bierman, Nix, Greenberg, Blair, & Domitrovich, 2008; Izard et al., 2008). However, many children growing up under conditions of high early life stress do not receive early intervention and therefore must rely on programs that target older age groups. A psychological intervention targeting 12–14 year-olds improved many different factors of emotional development, including the regulation of emotion (Garaigordobil, 2004). Interventions involving peers have also been successful in improving emotion regulation in both children and adolescents (Dishion & Tipsord, 2011). These studies show promising avenues to building emotion regulation skills in late childhood and adolescence. Although interventions exist for both early childhood and later periods of development, they are not specifically tailored to the ELS population. Therefore, the degree to which these interventions are effective in populations that have experienced early life stress and deprivation and the identification of optimal timing will be important topics to address in the future.

It is also important to note that whereas being able to inhibit emotional expression in the context of a speech is adaptive, high emotion regulation may not necessarily relate to positive functioning in all situations. Over-controlled emotional expression, such as expressive suppression, can lead to maladaptive outcomes like anxiety, depression, and diminished social competence (Butler et al., 2003). In the present study, we specifically looked at regulation in the context of a socio-evaluative task, where it was adaptive to dampen or hide expressions of anxiety or nervousness. Future studies should examine emotion regulation in other contexts to assess whether early life stress is associated with deficits in the regulation of emotion in those contexts. Youth who have experienced early life stress tend to develop behaviors more consistent with under-control of emotion (e.g., indiscriminate friendliness; Chisholm, 1998) rather than over-control, thus lower expressivity in the present population was viewed as more adaptive functioning given the context of the task and the type of emotion regulation problems that are typical of youth who experience early life stress.

In this specific study, we are unable to disentangle emotion intensity from emotion regulation. Therefore, low emotion expressivity could have indicated low levels of experience, rather than successful regulation of anxious, nervous emotions. However, neither our biological (cortisol) nor self-report (SAM figure) measures indicate that the ELS group was more anxious or nervous than the NA group. Thus, these data do not support this alternative interpretation. Nonetheless, it cannot be ruled out. Future research should explore these alternative possibilities using momentary measures of perceived stress.

There are a number of limitations to this study. First, as noted earlier, we were likely not sufficiently powered to confidently test the interaction of age and group. Second, although this study found a significant effect of early institutional rearing on anxiety expression, it cannot link specific aspects of early life stress to the deficit. Future studies should focus on aspects of early life stress (e.g., time spent in an institution, quality of care received, etc.) as potential moderators of emotion regulation abilities. The results of this study are also limited to one type of early life stress, that of early rearing in institutional care. In the future, other types of early life stress should be examined in relation to deficits in regulation. Future studies should also examine the effects of the life-course of family stress on regulation of emotion expression, as this study only addresses early life stress as a predictor of emotion regulation deficits. The combination of early life experiences, as well as current family stress and the quality of parenting received after adoption, may account for a more nuanced view on how ELS affects the ability to regulate emotional responses throughout childhood and adolescence. Additionally, due to the ELS group’s international origins, there were differences in the racial and ethnic backgrounds of the ELS and NA groups. Future research should include an ethnically diverse comparison sample whose parents are of the same education and income, to rule out racial/ethnic differences in the ability to regulate emotion expressions. Last, this study did not perform a manipulation check to test whether the task was believable to participants. Future research should incorporate this type of comprehension check in their research protocol.

Despite these limitations, the findings of the present study suggest that both age and early life stress play a role in children’s and adolescents’ abilities to regulate their expression of anxiety. Furthermore, the effects of both age and previous institutionalization on emotion expression were partially mediated by inhibitory control. Future work should focus on developing intervention programs specific to improving emotion regulatory abilities in this population.

Appendix Child and Adolescent Stress and Emotion Scale: Anxiety-Specific Ratings

Facial Signs of Anxiety:

  • 0, No Evidence: Child exhibits no visible signs of anxiety in the face.

  • 1, Low Intensity: Slightly furrowed brow or eyes widened. Or, brief winces or moments of facial fear.

  • 2, Moderate Intensity: Distinctive furrowed brow and eyes widened. Moderate winces.

  • 3, High Intensity: Brow definitely raised or furrowed. Eyes widened, may be tearful. Visible and distinctive frown.

Vocal Signs of Anxiety:

  • 0, No Evidence: Child exhibits no verbal signs of anxiety.

  • 1, Low Intensity: Slight quaking in voice during brief periods, no overly anxious content.

  • 2, Moderate Intensity: Consistent quaking in voice, or significant quaking only in small portions of assessment. Possible vocal content related to anxiety (“oh no, shoot, ooh”).

  • 3, High Intensity: Quaking in voice so prominent it interferes with speech or prevents child from talking. Or significant anxious speech (“I can’t do this, this is too hard’).

Bodily Signs of Anxiety:

  • 0, No Evidence: Child exhibits no visible signs of bodily anxiety.

  • 1, Low Intensity: cautious or wary gait, slight muscular tension, nervous fidgeting or self-touch, hand tapping, or somewhat diminished activity level.

  • 2, Moderate Intensity: slight defensive body posture or body swaying, or significant muscular tension, or definite diminished activity level, or pronounced fidgeting or self-touch.

  • 3, High Intensity: Aggressive shaking, pronounced defensive body posture. Child freezes, becomes stiff. Or, child leaves room due to anxiety, takes “break” from assessment.

Contributor Information

Amanda R Burkholder, Department of Human Development and Quantitative Methodology, University of Maryland, College Park.

Kalsea J. Koss, Institute of Child Development, University of Minnesota, Twin Cities.

Camelia E Hostinar, Institute for Policy Research, Northwestern University.

Anna E. Johnson, Department of Psychology, St. Olaf College.

Megan R Gunnar, Institute of Child Development, University of Minnesota, Twin Cities.

References

  • Bakracevic Vukman K. Developmental differences in metacognition and their connections with cognitive development in adulthood. Journal of Adult Development. 2005;12:211–221. 10.1007/s10804-005-7089-6. [CrossRef] [Google Scholar]
  • Bakracevic Vukman K, Licardo M. How cognitive, metacognitive, motivational, and emotion self-regulation influence school performance in adolescence and early adulthood. Educational Studies. 2009;36:259–268. 10.1080/03055690903180376. [CrossRef] [Google Scholar]
  • Bierman KL, Nix RL, Greenberg MT, Blair C, Domitrovich CE. Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program. Development and Psychopathology. 2008;20:821–843. 10.1017/S0954579408000394. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Blair C, Diamond A. Biological processes in prevention and intervention: The promotion of self-regulation as a means of preventing school failure. Development and Psychopathology. 2008;20:899–911. 10.1017/S0954579408000436. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Blair C, Granger DA, Kivilighan KT, Mills-Koonce R, Willoughby M, Greenberg MT, Fortunado CK. Maternal and child contributions to cortisol response to emotional arousal in young children from low-income rural communities. Developmental Psychology. 2008;44:1095–1109. 10.1037/0012-1649.44.4.1095. [Abstract] [CrossRef] [Google Scholar]
  • Bos K, Zeanah CH, Fox NA, Drury SS, McLaughlin KA, Nelson CA. Psychiatric outcomes in young children with a history of institutionalization. Harvard Review of Psychiatry. 2011;19:15–24. 10.3109/10673229.2011.549773. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Bradley MM, Lang PJ. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavioral Therapy & Experimental Psychiatry. 1994;25:49–59. 10.1016/0005-7916(94)90063-9. [Abstract] [CrossRef] [Google Scholar]
  • Bruce J, Tarullo AR, Gunnar MR. Disinhibited social behavior among internationally adopted children. Development and Psychopathology. 2009;21:157–171. 10.1017/S0954579409000108. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Buske-Kirschbaum A, Jobst S, Wustmans A, Kirschbaum C, Rauh W, Hellhmammer D. Attentuated free cortisol response to psychosocial stress in children with atopic dermatitis. Psychosomatic Medicine. 1997;59:419–426. 10.1097/00006842-199707000-00012. [Abstract] [CrossRef] [Google Scholar]
  • Butler EA, Egloff B, Wlhelm FH, Smith NC, Erickson EA, Gross JJ. The social consequences of expressive suppression. Emotion. 2003;3:48–67. 10.1037/1528-3542.3.1.48. [Abstract] [CrossRef] [Google Scholar]
  • Carlo G, Crockett LJ, Wolff JM, Beal SJ. The role of emotional reactivity, self-regulation, and puberty in adolescents’ prosocial behaviors. Social Development. 2012;21:667–685. 10.1111/j.1467-9507.2012.00660.x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Carlson SM, Wang TS. Inhibitory control and emotion regulation in preschool children. Cognitive Development. 2007;22:489–510. 10.1016/j.cogdev.2007.08.002. [CrossRef] [Google Scholar]
  • Chisholm K. A three year follow-up of attachment and indiscriminate friendliness in children adopted from Romanian orphanages. Child Development. 1998;69:1092–1106. 10.1111/j.1467-8624.1998.tb06162.x. [Abstract] [CrossRef] [Google Scholar]
  • Dishion TJ, Tipsord JM. Peer contagion in child and adolescent social and emotional development. Annual Review of Psychology. 2011;62:189–214. 10.1146/annurev.psych.093008.100412. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Eisenberg N, Spinrad TL, Eggum ND. Emotion-related self-regulation and its relation to children’s maladjustment. Annual Review of Clinical Psychology. 2010;6:495–525. 10.1146/annurev.clinpsy.121208.131208. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Eisenberg N, Valiente C, Eggum ND. Self-regulation and school readiness. Early Education and Development. 2010;21:681–698. 10.1080/10409289.2010.497451. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Ellis LK, Rothbart MK. Revision of the Early Adolescent Temperament Questionnaire. Poster presented at the biennial meeting of the Society for Research in Child Development; Minneapolis, MN. 2001. Apr, [Google Scholar]
  • Fonagy P, Target M. Early intervention and the development of self-regulation. Psychoanalytic Inquiry. 2002;22:307–335. 10.1080/07351692209348990. [CrossRef] [Google Scholar]
  • Garaigordobil M. Effects of a psychological intervention on factors of emotional development during adolescence. European Journal of Psychological Assessment. 2004;20:66–80. 10.1027/1015-5759.20.1.66. [CrossRef] [Google Scholar]
  • Garvin MC, Tarullo AR, Van Ryzin M, Gunnar MR. Post-adoption parenting and socioemotional development in post-institutionalized children. Development and Psychopathology. 2012;24:35–48. 10.1017/S0954579411000642. [Abstract] [CrossRef] [Google Scholar]
  • Govindan RM, Behen ME, Helder D, Makki MI, Chugani HT. Altered water diffusivity in cortisol association tracts in children with early deprivation identified with tract-based spatial statistics. Cerebral Cortex. 2010;20:561–569. 10.1093/cercor/bhp122. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Hodges J, Tizard B. The effect of early institutional rearing on the development of eight your old children. The Journal of Child Psychology Psychiatry. 1978;19:99–118. 10.1111/j.1469-7610.1978.tb00453.x. [Abstract] [CrossRef] [Google Scholar]
  • Hostinar CE, Johnson AE, Gunnar MR. Parent support is less effective in buffering cortisol stress reactivity for adolescents compared to children. Developmental Science. 2015;18:281–97. 10.1111/desc.12195. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Hostinar CE, Stellern SA, Schaefer C, Carlson SM, Gunnar MR. Associations between early life adversity and executive function in children adopted internationally from orphanages. Proceedings of the National Academy of Sciences of the United States of America. 2012;109:17208–17212. 10.1007/s10803-005-0005-x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Izard CE, King KA, Trentacosta CJ, Morgan JK, Laurenceau JP, Krauthamer-Ewing ES, Finlon KJ. Accelerating the development of emotion competence in Head Start children: Effects on adaptive and maladaptive behavior. Development and Psychopathology. 2008;20:369–397. 10.1017/S0954579408000175. [Abstract] [CrossRef] [Google Scholar]
  • Kim J, Cicchetti D. Longitudinal pathways linking child maltreatment, emotion regulation, peer relations, and psychopathology. The Journal of Child Psychology and Psychiatry. 2010;51:706–716. 10.1111/j.1469-7610.2009.02202.x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Kochanska G, Murray LT, Harlan ET. Effortful control in early childhood: Continuity and change antecedents and implications for social development. Developmental Psychology. 2000;36:220–232. 10.1037/0012-1649.36.2.220. [Abstract] [CrossRef] [Google Scholar]
  • Kopp CB, Neufeld SJ. Emotional development during infancy. In: Davidson RJ, Sherer KR, Goldsmith HH, editors. Handbook of affective sciences. New York, NY: Oxford University Press; 2003. pp. 347–274. [Google Scholar]
  • Kumsta R, Steven S, Brookes K, Schlotz W, Castle J, Beckett C, Sonuga-Barke E. 5HTT genotype moderates in influence of early institutional deprivation on emotional problems in adolescence: Evidence form the English and Romanian Adoptee (ERA) study. Journal of Child Psychology and Psychiatry. 2010;51:755–762. 10.1111/j.1469-7610.2010.02249.x. [Abstract] [CrossRef] [Google Scholar]
  • Lang PJ. Behavioral treatment and bio-behavioral assessment: Computer applications. In: Sidowski JB, Johnson JH, Williams TA, editors. Technology in mental health care delivery systems. Norwood, NJ: Ablex; 1980. pp. 119–137. [Google Scholar]
  • Lewis MD, Lamm C, Segalowitz SJ, Stieben J, Zelazo PD. Neurophysiological correlates of emotion regulation in children and adolescents. Journal of Cognitive Neuroscience. 2006;18:430–443. 10.1162/jocn.2006.18.3.430. [Abstract] [CrossRef] [Google Scholar]
  • Lindbald F, Weitoft GR, Hjern A. ADHD in international adoptees: A national cohort study. European Journal of Child and Adolescent Psychiatry. 2010;19:37–44. 10.1007/s00787-009-0038-3. [Abstract] [CrossRef] [Google Scholar]
  • Martel MM, Nigg JT, Von Eye A. How do trait dimensions map onto ADHD symptom domains? Journal of Abnormal Child Psychology. 2009;37:337–366. 10.1007/s10802-008-9255-3. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Maughan A, Cicchetti D. Impact of child maltreatment and interadult violence on children’s emotion regulation abilities and socioemotional adjustment. Child Development. 2002;73:1525–1542. 10.1111/1467-8624.00488. [Abstract] [CrossRef] [Google Scholar]
  • McCall RB. Review: The consequences of early institutionalization: Can institutions be improved? – should they? Child and Adolescent Mental Health. 2013;18:193–201. 10.1111/camh.12025. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Muthén LK, Muthén BO. Mplus User’s Guide Seventh Edition. Los Angeles, CA: Muthén & Muthén; 1998–2012. [Google Scholar]
  • Pollak SD, Nelson CA, Schlaak MF, Roeber BJ, Weweka SS, Wiik KL, Gunnar MR. Neurodevelopmental effects of early deprivation in post-institutionalized children. Child Development. 2010;81:224–236. 10.1111/j.1467-8624.2009.01391.x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Preacher KJ, Kelley K. Effect sizes measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods. 2011;16:93–115. 10.1037/a0022658. [Abstract] [CrossRef] [Google Scholar]
  • Rothbart MK, Bates JE. Temperament. In: Damon W, Lerner RM, Eisenberg N, editors. Handbook of child psychology, Volume 3: Social emotional and personality development. New York, NY: John Wiley & Sons; 2006. pp. 105–176. [Google Scholar]
  • Rutter M, Kreppner J, O’Connor T, the English and Romanian Adoptees (ERA) Study Team Risk and resilience following profound early privation. British Journal of Psychiatry. 2001;179:97–103. 10.1192/bjp.179.2.97. [Abstract] [CrossRef] [Google Scholar]
  • Schachar R, Mota VL, Logan GD, Tannock R, Klim P. Confirmation of an inhibitory control deficit in attention-deficit/hyperactivity disorder. Journal of Abnormal Child Psychology. 2000;28:227–235. 10.1023/A:1005140103162. [Abstract] [CrossRef] [Google Scholar]
  • Schachar R, Tannock R, Marriott M, Logan G. Deficient inhibitory control in attention deficit hyperactivity disorder. Journal of Abnormal Child Psychology. 1995;23:411–437. 10.1007/BF01447206. [Abstract] [CrossRef] [Google Scholar]
  • Shields AM, Cicchetti D, Ryan RM. The development of emotional and behavioral self-regulation and social competence among maltreated school-age children. Development and Psychopathology. 1994;6:57–75. 10.1017/S0954579400005885. [Abstract] [CrossRef] [Google Scholar]
  • Shipman KL, Zeman J. Socialization of children’s emotion regulation in mother- child dyads: A developmental psychopathology perspective. Development and Psychopathology. 2001;13:317–336. 10.1017/S0954579401002073. [Abstract] [CrossRef] [Google Scholar]
  • Shrout PE, Bolger N. Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods. 2002;7:422–445. 10.1037/1082-989X.7.4.422. [Abstract] [CrossRef] [Google Scholar]
  • Slag MF, Ahmed M, Gannon MC, Nuttall FQ. Meal stimulation of cortisol secretion: A protein induced effect. Metabolism. 1981;30:1104–1108. 10.1016/0026-0495(81)90055-X. [Abstract] [CrossRef] [Google Scholar]
  • Tarullo AR, Bruce J, Gunnar MR. False belief and emotion understanding in post-institutionalized children. Social Development. 2007;16:57–78. 10.1111/j.1467-9507.2007.00372.x. [CrossRef] [Google Scholar]
  • Tottenham N, Hare TA, Millner A, Gilhooly T, Zevin JD, Casey BJ. Elevated amygdala response to faces following early deprivation. Developmental Science. 2011;14:190–204. 10.1111/j.1467-7687.2010.00971.x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Tottenham N, Hare TA, Quinn BT, McCarry TW, Nurse M, Gilhooly T, Casey BJ. Prolonged institutional rearing is associated with atypically large amygdala volume and difficulties in emotion regulation. Developmental Science. 2010;13:46–61. 10.1111/j.1467-7687.2009.00852.x. [Europe PMC free article] [Abstract] [CrossRef] [Google Scholar]
  • Yim IS, Quas JA, Cahill L, Hayakawa CM. Children’s and adults’ salivary cortisol responses to an identical psychosocial laboratory stressor. Psychoneuroendocrinology. 2010;35:241–248. 10.1016/j.psyneuen.2009.06.014. [Abstract] [CrossRef] [Google Scholar]

Full text links 

Read article at publisher's site: https://doi.org/10.1111/sode.12170

Read article for free, from open access legal sources, via Unpaywall: https://europepmc.org/articles/pmc5454775?pdf=renderUnpaywall

Citations & impact 

Impact metrics

17
Citations
Jump to Citations

Citations of article over time

Alternative metrics

Altmetric item for https://www.altmetric.com/details/4732684
Altmetric
Discover the attention surrounding your research
https://www.altmetric.com/details/4732684

Smart citations by scite.ai
Smart citations by scite.ai include citation statements extracted from the full text of the citing article. The number of the statements may be higher than the number of citations provided by EuropePMC if one paper cites another multiple times or lower if scite has not yet processed some of the citing articles.
Explore citation contexts and check if this article has been supported or disputed.
https://scite.ai/reports/10.1111/sode.12170

Supporting
Mentioning
Contrasting
1
38
1

Article citations

Go to all (17) article citations

Similar Articles 

To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.

Funding 

Funders who supported this work.

NIMH NIH HHS (1)