Auditory stimulation protocol

EEG recordings were conducted at the patients’ bedside, within 24 h after cardiac arrest while all patients were in a comatose state. Patients were presented with a series of pure tones as previously described.^15–17 Tones consisted of 16-bit stereo sounds, sampled at 44.1 kHz, with a 10 ms linear amplitude envelope applied at stimulus onset and offset to avoid clicks. Between each sound, there was a 700 ms interstimulus interval. Standard sounds were presented in 70% of the trials and had a pitch of 1000 Hz and duration of 100 ms. Deviant sounds differed from the standards in duration (150 ms), interaural time difference, with left ear leading with 700 μs, or pitch at 1200 Hz. Stimuli were presented in a pseudo-randomized order, in a way that at least one standard sound was presented between two deviant ones. The auditory stimulation protocol consisted in total of 1500 tones, split into three blocks, each lasting ~7 min. Similar to a previous study investigating neural properties of auditory processing in coma,¹⁸ in the present study we focused on responses to standard and duration deviant sounds, as they have been previously shown to be highly informative of coma outcome.^15,18 A detailed evaluation of the predictive value of EEG responses to all sound types is provided in the Supplementary material.

Recording setup and preprocessing

As the data of this study were multicentre and multiprotocol, EEG was recorded at the patient's bedside with 19 or 62 electrodes, depending on the original study design,^15–17 positioned according to the international 10–20 system. EEG data were collected with a sampling frequency of either 1000 or 1200 Hz. For consistency, data recorded with 1200 Hz were downsampled to 1000 Hz and data recorded with 62 electrodes were downsampled to the overlapping set of 19 electrodes. Across all channels the impedance was kept below 10 kΩ. The online reference for the electrodes was set as Fpz. In the course of preprocessing electrodes were re-referenced to a common average reference. Epochs were extracted from 50 ms before stimulus onset to 500 ms after. Artefacts were rejected with a criterion of ±100 μV on all electrodes. Noisy electrodes were interpolated using three-dimensional splines.³³ After re-referencing, the data were filtered from 0.1 to 40 Hz, and in a control analysis, to ensure that our findings are not driven by muscle activity, from 0.1 to 20 Hz. Additionally, the EEG epochs were visually inspected and noisy epochs were manually removed. After preprocessing, we obtained 347.22 ± 9.23 trials (mean ± SE) per patient, 204.76 ± 8.66 in response to standard sounds and 142.46 ± 1.42 in response to duration deviants.

Training of convolutional neural network

For training the neural network we adopted a 10-fold cross-validation procedure, i.e. we split the patients 10 times into 3 different sets of train, validation and test patients, in a way that data from each patient were included in only one of the three sets. The train set had 60% of patients (80 patients), the validation set 20% (27 patients) and the test set 20% (27 patients). The single trials of all patients from the train set were used to train the neural network and the trials from the validation set were used to evaluate any hyperparameters (e.g. learning rate, early stopping, etc.). The test set was used only after training and optimizing the network, for an objective evaluation of the model’s performance on unseen patients.

We trained a CNN called EEGNet²¹ to predict patients’ outcomes, which has been designed specifically for EEG data. The overall network size is moderate, giving smaller training times while still achieving robust results. The original network architecture was changed slightly for the purpose of our study. First, according to the recommendations as in the original study²¹ the filter length of the first convolutional layer was changed to 512, instead of the original 64. Consequently, the number of temporal, spatial and pointwise filters were increased to 16, 4 and 64, respectively, while the activation function was set to ReLu. These changes were made as preliminary analysis on a small subset of patients showed a more stable training.

For optimizing the model we used the binary cross-entropy loss function and the Adaptive Moment Estimation (Adam),³⁴ with a learning rate of 5 × 10⁻⁶. All other parameters were unchanged from the default suggestions. We trained the network for a maximum of 100 epochs, but employed early stopping after 30 epochs if the validation loss of two consecutive epochs was smaller than a threshold, as is standard practice in the field.³⁵ To evaluate the network performance we additionally used the area under the curve (AUC).³⁶

Outcome prediction based on network’s output

As the neural network was trained on single-trial EEG responses to sounds (Fig. 1), it classified single-trial EEG responses as belonging to a survivor versus a non-survivor. To link the network’s output to coma outcome at the single patient level, for each patient we computed the mean classification performance across all trials, which we term ‘confidence of predicting survival’. The confidence score ranged between zero and one and indicated the confidence of the model’s prediction of coma outcome, as the mean label that the model assigned across single-trial EEG responses to sounds of a given patient. If that score was above 0.5, then a patient was classified as a survivor, while below 0.5 as a non-survivor, as per convention based on a sigmoid function (Fig. 1).

Open in a separate window

Figure 1

Schematic description of the deep learning algorithm for predicting outcome from coma. Auditory stimulation and EEG recordings are performed on the first day of coma, shortly after a cardiac arrest. Single-trial EEG responses are then given as input to a CNN which classifies them as belonging to a survivor versus a non-survivor. The average of all single-trial predictions provides the network’s confidence of predicting survival. Survival is defined at 3 months via the CPC score.

For evaluating the network’s performance, for all metrics we report mean ± SE values obtained across the 10 folds of cross-validation. We additionally plot the network’s output in relation to patients’ outcome for the best model, defined as the one with the highest AUC score of the validation set. The reported final results remain objective, as they concern the test set of patients which was not used to train, optimize or select the best model. We additionally evaluated the positive and negative predictive values (PPV and NPV, respectively), as the ratio of correctly predicted survivors among all predicted survivors (PPV) or correctly predicted non-survivors among all predicted non-survivors (NPV).

Outcome prediction in patients with indeterminate prognosis

To assess the additional utility of our method for the current clinical practice, we evaluated the network’s predictions for a subset of coma patients whose outcome prognosis was inconclusive based on existing clinical tests. Based on previous literature and recommendations, we considered the motor response, EEG reactivity, EEG continuity and brainstem reflexes.^1,2 If the above-mentioned variables showed a discrepancy (e.g. present motor response and brainstem reflexes, but a discontinuous and irritative EEG), a patient was defined as being in a ‘grey zone’. For this definition, we did not include patients where only brainstem reflex was present and all other variables predicted a negative outcome, due to the low positive predictive power of brainstem reflex for good outcome.¹ This resulted in 48 patients fulfilling the criteria for a clinical ‘grey zone’ (32 survivors, 16 non-survivors). Because of their relatively low number, for this analysis we merged patients from train, validation and test sets, and examined the overall distribution of the outcome prediction resulting from the CNN. Moreover, we performed two control analyses to further evaluate the generalizability of the network on ‘grey zone’ patients. First, we curated this split of patients to train, validation and test sets, to ensure that a fixed but high number of ‘grey zone’ patients are part of the test set (Supplementary material). Second, we trained one network using exclusively patients with determinate outcomes for training and validation, and kept all ‘grey zone’ patients as a test set, to evaluate generalization of outcome prediction results (Supplementary material).

Exploring links between network’s output and electrophysiological features of EEG responses

To explore the features of EEG responses related to the network’s decisions, we explored (i) measures of neural synchrony and complexity, previously shown to relate to patients’ outcome and presence of consciousness, respectively¹⁸; and (ii) well established clinical variables currently used for outcome prognosis.^1,2

We first computed the phase-locking value (PLV), which quantifies the synchrony of EEG responses to auditory stimuli.³⁷ The PLV was computed for electrode pairs in the alpha range, which has been recently shown to be predictive of patients’ outcome in two different cohorts of patients undergoing HT.¹⁸ Here, we calculated the mean PLV per patient across electrode pairs, applying the same procedure as previously reported, based on a subset of patients included in the present study.¹⁸ The neural complexity was quantified via the Lempel–Ziv (LZ) complexity, which measures the number of unique patterns present in a signal.³⁸ Both PLV and LZ were calculated on single-trial EEG responses, similar to the neural network, to capture trial-by-trial characteristics of EEG responses to the auditory stimuli. Here, we expanded the analysis that was recently reported¹⁸ for both measures on a larger patient cohort. Our goal on the one hand was to link these measures to the network’s output (via Pearson correlation coefficient, P_corr < 0.01, Bonferoni corrected) and on the other hand, to validate these measures in patients treated with targeted temperature management at 36°C, as they were previously only reported for patients treated at 33°C. We excluded two patients that were part of a previous study,¹⁸ because they awoke in the hospital but subsequently died before the CPC assessment at 3 months (see the 'Patients and procedure' section).

Second, we compared predictions of the network in subgroups of patients defined according to the following binary markers: presence of brainstem reflexes (pupil and corneal reflexes present), presence of motor response (motor GCS ≥ 4), reactive EEG background (change in amplitude and/or frequency after stimulus, judged visually), discontinuous or suppressed EEG background³⁹ and irritative EEG (presence of electrographic seizures or status epilepticus, sporadic epileptiform discharges, spiky or sharp periodic discharges or rhythmic spike waves), using Mann–Whitney tests (P_corr < 0.01), and Bonferroni correction for multiple comparisons. We additionally explored correlations between the network’s confidence and time to return of spontaneous circulation (ROSC), with a Pearson correlation coefficient, as well as links between confidence and hospital site or CPC at 3 months with Kruskal–Wallis H-tests.

Prediction of outcome based on the neural network

The neural networks trained to discriminate auditory EEG responses of patients that later survived from those who did not reached a mean AUC score of 0.81 ± 0.00 on the train, 0.75 ± 0.03 on the validation and 0.70 ± 0.04 on the test sets (Fig. 2 and Table 1). On the test set we obtained a PPV of 0.83 ± 0.03 and an NPV of 0.57 ± 0.04 (Table 1). For patients treated with targeted temperature management at 33°C, the average PPV was 0.83 ± 0.04 and for those treated at 36°C 0.81 ± 0.06. The difference in PPV over the two treatments over the 10 cross-validation folds was not significant (P = 0.67, Wilcoxon signed rank test), implying that the networks performed at similar levels for patients treated with targeted temperature management at 33°C and 36°C. The AUC scores were also replicated with a control analysis where the neural networks were trained with EEG data filtered between 0.1 and 20 Hz, resulting in a mean AUC score of 0.74 ± 0.03, PPV of 0.86 ± 0.02 and NPV of 0.62 ± 0.03 on the test set.

Open in a separate window

Figure 2

Prediction of outcome based on the CNN. Confidence scores were computed for each patient by averaging the network’s outcome prediction for all single-trial EEG responses to sounds of a given patient. A patient was predicted to be a survivor if the confidence score was above 0.5, otherwise a non-survivor. Data from patients of the train set (empty circles) were used to train the network, while data from patients in the validation set (shaded circles) were used to evaluate hyperparameters. The predictive value of the network was evaluated on a separate test set of patients (full circles), whose data were never used to train or optimize the network. For each of the sets, we split patients into HT (hypothermia, targeted temperature management at 33°C) and NT (patients with targeted temperature management at 36°C). For the numerical performance scores see Table 1.

We next focused on one single fold of the cross-validation, and evaluated the confidence scores assigned by the network to individual patients (Fig. 2), as our goal was to investigate the neural properties of EEG signals that may mediate the network’s outcome prediction. In the validation and test sets, we obtained a sensitivity of 84% for survivors and a specificity of 82% for non-survivors. Out of all the patients classified as survivors in the train set (n = 42 patients), 36 awoke from coma, resulting in a PPV of 86%, while in the validation/test sets 27 of 31 patients that were predicted as survivors awoke, resulting in a PPV of 87%. These results were largely similar for patients treated with different temperature treatments (Fig. 2 HT, NT and Table 1).

Outcome prediction for patients in a ‘grey zone’

We next focused on patients who, from a clinical viewpoint, were part of a ‘grey zone’, i.e. cases where currently used outcome predictors indicated indeterminate outcome. Forty-eight patients fulfilled these criteria (n = 32 survivors and n = 16 non-survivors, Fig. 3). The distribution of confidence values assigned by the network in this subset of patients followed the distribution of the full cohort (Figs 2 and ​and3).3). For this subset of patients we obtained a PPV of 0.86, an NPV of 0.60 and an AUC score of 0.75, based on the CNN. These scores were at similar levels as those obtained with the full cohort, suggesting that although this group of patients had indeterminate prognosis based on existing clinical tests, they were not ‘peculiar’ cases for the neural network (see the Supplementary material for additional analyses on 'grey zone' patients).

Open in a separate window

Figure 3

Confidence of survival assigned by the network for patients with uncertain outcome prognosis based on existing outcome predictors. Confidence scores for patients in this subset followed the distribution of confidence scores in the entire patient cohort (Fig. 2). The empty circles show patients in the train set, shaded circles patients in the validation and full circles patients in the test set. The left side of the figure represents survivors and the right side non-survivors.

Investigating the interpretability of the neural network’s outcome prediction

We next evaluated the electrophysiological properties of EEG signals that the network’s output may be reflecting while providing a prediction about patients’ outcome. We first evaluated the PLV of EEG responses to sounds, previously shown to be predictive of patients’ outcome when treated with hypothermia.¹⁸ Here, we first replicated these results for patients in NT. The mean PLV for survivors treated with hypothermia was 0.75 ± 0.01 and for survivors treated with normothermia 0.72 ± 0.02. For non-survivors, we found a mean PLV of 0.55 ± 0.02 and 0.57 ± 0.03 for patients treated with hypothermia and NT, respectively. When statistically tested, we found a significant main effect of outcome on PLV (F = 101.68; P_corr < 0.01), while neither the main effect of temperature treatment (F = 4.07; P_corr = 0.046) nor the outcome by temperature interaction were significant (F = 2.17; P = 0.14). Next, we explored the predictive power of the PLV in patients treated with NT, as done previously for the subset of patients treated with hypothermia (partially overlapping with those included in our previous study¹⁸). For patients in hypothermia, the PLV provided a PPV of 0.85 and an NPV of 0.83, as previously reported.¹⁸ For patients in NT, the PLV resulted in a PPV of 0.77 and an NPV of 0.80 (Fig. 4A).

Open in a separate window

Figure 4

Investigating the interpretability of the neural network’s output. (A) PLV for survivors (left) and non-survivors (right). The horizontal line corresponds to a threshold PLV of 0.69, identified and already evaluated in a subset of patients, previously reported in Alnes et al.¹⁸ The previously reported distribution of PLV values was replicated in a new cohort, predominantly treated with NT. (B) Correlation of network confidence to PLV. The network’s confidence of survival was strongly correlated to the average PLV (Pearson’s r = 0.76; P_corr < 0.01). Exemplar EEG responses to sounds have been plotted for two survivors and two non-survivors, one correctly and one incorrectly classified based on the neural network. (C) LZ complexity of auditory ERPs for each patient. As previously reported in a subset of patients,¹⁸ LZ complexity was not informative of patients’ outcome. (D) Correlation between LZ complexity and the network’s confidence of survival for the entire patient cohort (r = −0.53; P_corr < 0.01). The correlation values are also plotted separately for survivors (bottom plot) and non-survivors (top plot). The highlighted circles with borders mark the patients whose exemplar EEG responses are shown in B.

Importantly, the confidence of survival assigned by the network to each patient strongly correlated with the mean PLV across electrodes (r = 0.76; P_corr < 0.01; Fig. 4B). This correlation was not trivially driven by the fact that both measures predict outcome, as it remained significant when tested for survivors (r = 0.49; P_corr < 0.01) and non-survivors (r = 0.65; P_corr < 0.01) separately.

Next, we computed the LZ complexity of EEG responses to sounds, which by itself has been shown not to be predictive of outcome when only including patients treated at 33°C.¹⁸ We confirmed this previous finding for patients of the present study who were treated at 36°C, whose distribution of LZ values was similar to the distribution of LZ values for patients treated at 33°C (Fig. 4C).

Interestingly, although LZ complexity by itself was not predictive of outcome, it showed a significant negative correlation with the network’s confidence (r = −0.57; P_corr < 0.01; Fig. 4D), so that the higher the network’s confidence in predicting survival, the lower the complexity of EEG responses to sounds. A strong correlation between network’s confidence and EEG complexity was also observed for survivors (r = −0.48; P_corr < 0.01) and non-survivors (r = −0.76; P_corr < 0.01) separately. PLV and LZ complexity were also negatively correlated, albeit with a weaker correlation than each of these measures did with the confidence of the neural network (Pearson’s r = −0.39; P_corr < 0.01).

Overall, these results suggest that the network assigned higher confidence scores for awakening for patients with high PLV and low complexity, or in other words, with stronger neural synchrony and temporal structure in their EEG responses. Exemplar EEG traces of these responses for a correctly classified survivor and a misclassified non-survivor show rather smooth EEG responses to the sounds, compared to exemplar traces of a correctly classified non-survivor and misclassified survivor, where the EEG responses are more ‘stochastic’ (Fig. 4B).

Auditory stimulation for predicting outcome from coma

The main novelty and advantages of our approach of combining auditory stimulation with deep learning to predict coma outcome are 3-fold: (i) it is semi-automatic, based on a single EEG recording performed within 24 h after cardiac arrest and, if confirmed in other data sets, can be objectively used to assist in predicting patients’ chances of awakening from coma; (ii) it relies on the auditory pathway, which is currently not actively used in the clinical routine for outcome prediction, and can therefore provide additional clinical insights for patients in a ‘grey zone’, whose outcome is indeterminate based on existing techniques; and (iii) the output of the neural network is not a simple binary prediction of outcome, but exploits a continuum of confidence values, which are then directly linked and strongly correlated to interpretable features of EEG responses.

Our work follows a large body of literature showing links between neural responses to auditory stimulation in comatose or unresponsive patients and patients’ outcome.^4–7 Standardized auditory stimulation in particular has been proposed to be informative of patients’ chances of awakening.^9–13 However, to date it is not regularly used in the clinical routine, as the majority of existing techniques are not robust enough, rendering a clinical implementation challenging.^15–17 Here, we overcame this challenge by focusing on one single EEG recording of ~20 min, and analysed EEG responses to auditory stimulation with CNNs. These networks have the strong advantage that they can detect discriminative patterns even in heterogeneous data sets, with minimal a priori assumptions and preprocessing.^{19,20,22–24,26} Importantly, all steps of the presented method are automatic, apart from visual inspection of the data and manual rejection of artefacts, which were done to ensure high-quality data. Future studies can investigate whether this step can also be automated and replaced with existing algorithms for rejection of EEG artefacts.⁴⁰

Neural networks assisting prognostication of coma outcome

Neural networks have been used in several fields showing an astonishing potential to automate and improve prognostication of various neurological disorders,^19,20 but their use in neuro-critical care and coma outcome prognosis remains limited. The few existing studies using neural networks to predict coma outcome are based on EEG recordings in the absence of external stimulation. These have shown a remarkable performance (AUC = 0.89²⁴; AUC at 24 h = 0.88²³; best AUC at 24 h = 0.89²⁵), in predicting outcome. In our study, we obtained a mean AUC score of 0.70 ± 0.04 and 0.83 for the best fold (Table 1), and a mean PPV of 0.83 ± 0.03 and 0.92 for the best fold (Table 1). The AUC values we obtained are slightly lower than previous studies. However, it is worth noting that our study relies on a complementary piece of information to what is currently used in the clinical routine, that is, standardized auditory stimulation. Our approach is predictive of awakening and not of overall outcome, and therefore the most informative measure of performance is the PPV and not the AUC.

As such, we could show that neural networks have the potential to provide concrete diagnostic information for patients in a clinical ‘grey zone’, for whom currently available clinical tests are inconclusive. The PPV for these patients was at similar levels as for the full cohort. Moreover, control analyses confirmed that the network’s prediction generalizes to ‘grey zone’ patients, even when it has been trained with few or no ‘grey zone’ patients (Supplementary material). Future studies with larger patient cohorts can evaluate the clinical applicability of neural networks and confirm these results.

One main limitation of our approach, and also of most previous studies (with few exceptions, such as Juan et al.⁴¹), is that we only predict a binary outcome of survival versus non-survival. Survivors are defined as patients with a CPC 1–3, which corresponds to varying levels of autonomy and quality of life.^32,42 Although a CPC of 1–2 generally represents a satisfactory quality of life, a CPC of 3 can be a heterogeneous class. Because we trained the network to specifically discriminate between patients with CPC 1–3 versus patients with a CPC of 5 (death), we could not find a more fine-grained link between the network’s output and the state of survival at 3 months. Future studies, expanding on larger patient cohorts, could use the CPC score already during network training to evaluate whether auditory responses in the acute coma phase can be informative not only of survival, but also of its quality.