Different temporal and frequential properties of activity in the aSPPO region between speech production and perception

Once we had established that a robust overlap exists between expressive and receptive speech in the articulatory cortex during real-world conversations, we were interested in characterizing the neuronal activity in the aSPPO region in more detail. To this end, we estimated the differences in the onset of the earliest significant activation and in the maximum response magnitude between anatomical areas within each condition and within anatomical areas between conditions.

Timing of neural activity during speech production and perception

Our analysis of the earliest time points of significant activation in each cortical area and speech condition revealed temporal differences between areas (Figs 2, ​,4A,4A, Suppl. Table ²). These were particularly pronounced in speech production, whereas involvement of the different anatomical areas occurred more simultaneously during perception. Activity underlying speech production in the aSPPO region started significantly earlier than in a number of other anatomical areas (in the prefrontal cortex, in the temporal cortex and on the lateral sulcus, in Brodmann area (BA) 45, in the parietal operculum, and in the inferior parietal cortex, see left panel in Fig. 4A). It proceeded with responses in the premotor cortex accompanied by activation of BA44 and of the primary somatosensory cortex at the earlier stages of processing. This activity was followed by approximately simultaneous involvement of several other areas, such as the inferior parietal cortex, the parietal operculum, BA45, the temporal cortex, and ca. 200ms later by the prefrontal cortex. Interestingly, BA44 was activated significantly earlier than BA45, possibly reflecting functional differences between these compartments of Broca’s area⁴⁴.

Open in a separate window

Figure 4

A comparison of the earliest time points of significant activation (A,B) of the maximum high-gamma response magnitudes between the different anatomical areas within each condition and within each anatomical area between conditions. The height of the colored vertical bars show the earliest timing (A) and the maximum high-gamma response magnitude values (B) for all frequency bands together, averaged over all electrodes with significant RSMC (Wilcoxon sign test, FDR-corrected at q=0.005) in each anatomical area of each subject, then averaged in the respective anatomical area (color-coded, abbreviations for the anatomical areas are below the bars) over subjects (see Methods). The left panels of (A,B) show effects observed in speech production, and the right panels show effects related to speech perception. The number of subjects with significant RSMC in the respective area (those with significant effects in the given area/those with electrode coverage of the respective area) is indicated below the vertical colored bars. The whiskers at each colored bar show the standard errors of the median obtained using a bootstrapping procedure (see Methods). Note that the number of values per anatomical area was often small (2–8 subjects with effects in the given area), so one should not overestimate the importance of the bootstrapping-approximated whisker length with regard to the outcome of statistical testing. (i) Significant differences between anatomical areas within the same condition (Wilcoxon rank sum test, uncorrected) are indicated by black brackets, and (ii) significant differences within the same anatomical area between conditions (Wilcoxon rank sum test, uncorrected) are indicated by red brackets. The height of the p-values is coded by the number of (i) black and (ii) red stars, one star: p<0.05, two stars: p<0.01, three stars: p<0.001.

In speech perception, earliest activations in BA44 and in aSPPO occurred significantly later than those in the non-aSPPO premotor cortex and on the central sulcus. Activation in all other areas did not show a systematic sequential order (right panel of Fig. 4A, Suppl. Table ²). Significant activations in the inferior parietal cortex and/or in the non-aSPPO premotor cortex and the central sulcus were observed prior to the onset of speech perception. In an attempt to find an explanation to this surprising finding, we looked up, what functional roles the electrodes with this early activation showed in ESM. They lay in the eye motor cortex or in the immediate proximity of the eye motor cortex (not visualized). These effects thus possibly reflect eye movements, which are known to occur prior to speech for mediation of turn transitions in real-world conversations⁴⁵. The effects in the inferior parietal cortex may be related to the processing of communicative intentions⁴⁶.

A within-area statistical comparison between speech production and perception revealed that aSPPO and BA44 had most pronounced temporal differences between conditions. These areas may thus be good candidates to distinguish between speech production and perception in real-world brain-computer-interface applications⁴⁷. Note, however, that our analysis of differences between speech production and perception is based on values that could be obtained from a small population of subjects. Our observation that speech production may be more sequentially organized, whereas speech perception appears to involve a more simultaneous structure of activation of the different cortical areas, will thus need validation in a larger population of subjects.

Relation to previous experimental findings

Wilson and colleagues¹⁵ were the first to provide evidence of spatially overlapping activity between speech production and perception in the motor cortex. These researchers investigated brain activity related to overt production of meaningless syllables and listening to the same set of syllables using fMRI. They found that the bilateral BA6, located in the superior-most part of the ventral premotor cortex, was activated in both speech production and perception. The center of mass in the perception-related fMRI data was shifted a few mm anteriorly (see our Fig. 3). Interestingly, the center of mass of the left-hemispheric fMRI overlap found in this experimental study was very close to the aSPPO region delineated in the present study during real-life communication (3,5-mm distance). Wilson and Iacoboni⁴⁹ hypothesized that effects in this overlap area could be interpreted within the theoretical framework of forward modelling (e.g.^50,51) and concluded that speech perception is a sensorimotor process in which sensory and motor modalities communicate^49,52. Our study lends support to these earlier findings of shared activity in the svPMC between speech production and perception, showing that such activity takes place not only in neurolinguistic experiments but also in conditions of every-day, real-life communication.

These earlier studies presented novel evidence on the spatial organization of expressive and receptive language. Due to the limitations intrinsic to fMRI as a method for investigating human brain function⁵³, however, they were unable to provide a more detailed description of the temporal and functional properties of activity in this region. These studies could not tell if their identified overlaps between speech production and perception in the svPMC lay in the mouth motor cortex, nor could they draw conclusions as to when this region was activated in the time course of speech production³. Our findings show that this overlap region lies in the mouth motor cortex as evidenced by ESM (Fig. 3, Suppl. Table ¹), and that it shows early activity, reflected in increased ECoG-RSMC across several bands of high-gamma frequencies prior to the onset of speech production (Fig. 4A, Suppl. Table ²).

The aSPPO region in our study was situated between the hand and orofacial subregions of the sensorimotor cortex along the central sulcus (Fig. 3). This region matches well with the overlapping gamma-band activity between speaking and listening to speech under experimental conditions observed in ECoG by Towle et al.³³. While their results were reported without strict statistical correction for multiple comparisons, the overlap in our data persisted in all subjects even with a conservative statistical threshold after FDR correction (two-tailed Wilcoxon sign test, FDR-corrected at q=0.005). The early activation of the premotor cortex in speech production is also consistent with an ECoG study by Pei et al.⁵⁴, which showed that word repetition after visual and auditory presentation is associated with early premotor-cortical responses.

Mouth motor properties of the overlap region between speech production and perception in the svPMC

In the present study, mouth motor function of the overlap region between speech production and perception in the svPMC could be established with the help of ESM. Stimulation against different reference electrodes may cause different current distributions. We used both monopolar and bipolar stimulation to diminish the likelihood of false positive stimulation results. Our findings firmly establish that the aSPPO in all subjects lay in a mouth motor area (Fig. 3, Suppl. Table ¹). ESM also provided useful information for the functional characterization of the neighborhood of the aSPPO region. The functional significance of the surrounding areas was relatively constant: aSPPO bordered dorsally with hand motor and sensory areas, and dorsorostrally with the eye motor cortex in all subjects. Ventrally, the aSPPO region adjoined other mouth motor sites and sometimes also a second hand motor area located in the border region between the ventral premotor cortex and BA44 (Fig. 1, Suppl. Fig. ¹). This arrangement, summarized in a schematic functional map in Fig. 3, may provide useful topographical orientation for future studies.

The choice of signal components may be crucial to the detection of the aSPPO region

Since gamma activity may show task-related differences between subjects⁵⁵, we investigated different gamma frequency bands in order not to miss the overlap between speech production and perception by restricting our analyses to a single frequency band. This was indeed helpful, as differences between subjects could be observed (Suppl. Table ¹): While an overlap between speech production and perception in the anatomical motor cortex occurred in all (S2, S4, S7) or at least in two of the three investigated frequency bands (S3, S6, S8), such an overlap in two subjects was found in one frequency band only (100–200Hz in S1, 200–350Hz in S5). The variability in the dominant signal component in which overlapping activity between speech production and perception in the motor cortex occurred possibly explains the fact that some previous studies have found none¹² or non-reproducible patterns of premotor-cortical activation during speech perception^13,14. Approaches similar to those implemented in the present study may be helpful to account for such inter-individual variability.

Functional interpretations of aSPPO activity during speech perception

The functional significance of the motor-cortical activity during speech perception is highly controversial. Some researchers interpret it as reflecting evoked articulatory representations^15,56. Another candidate role is the processing of visemic information^21,57. Activation of the premotor cortex, however, can also be observed when speech perception takes place in the auditory-only modality^15,21, and the functional role of premotor activation therefore cannot be restricted to the processing of information from the visual domain.

Other interpretations of the functional significance of such activity during speech perception include generation of forward predictions^49,51, modulation of auditory speech processing in a top-down manner⁵⁸, sensorimotor integration of speech information within the dorsal pathway^50,59, recognition of articulatory gestures via the mirror-neuron system^60,61, or intra- and inter-personal perceptuomotor coupling in a conversation⁶², turn taking being one example of such a phenomenon⁶³.

It has also been suggested that activation of motor-cortical regions during speech perception may represent a compensatory mechanism for the retrieval of speech signals under difficult listening conditions, e.g., during speech perception in noise or when the speaker has a foreign accent^4,40. Experimental reports, however, are divided as to whether activation of the motor cortex only occurs when subjects listen to distorted speech⁴¹ or also when they attend to clear speech²⁴.

To address the possibility that activity in the aSPPO region may be modulated by acoustic background noise during natural communication, we compared the levels of activation during receptive speech in the presence of strong ambient noises with receptive speech without such noises. Enhanced activity in the speech-in-noise condition could be expected if this area were responsible for filtering out the speech-relevant information from the acoustic signal during adverse listening conditions⁴⁰. This was, however, not the case in our data, suggesting that activation in the articulatory premotor cortex during real-world conversations in our data may not be specialized for compensation of acoustic noise in the environment. Note that natural noises in the speech-in-noise condition were shorter than the speech perception periods embedding them (see Methods). Their shorter duration, however, does not explain the complete absence of stronger responses during the speech-in-noise condition. One would expect enhanced noise-related neuronal effects to occur on a shorter time scale. These were completely absent in both speech-in-noise and in noise perception (Fig. 5). We thus consider an explanation of absent enhanced noise-related activity in the motor cortex by the shorter duration of the noises to be unlikely.

Due to the fact that this study was conducted on data from non-experimental, real-life conditions of human behavior, our choice of acoustic phenomena was constrained by the subjects’ situation. They were in a hospital, and most of them had no ward neighbors. It is conceivable that activity in the aSPPO region could be modulated by acoustic phenomena that we could not study due to their seldom occurrence or absence in our materials. Interesting questions, e.g., would be whether the perception of such complex acoustic stimuli as music or of unattended speech during conversations in which the subject is not involved would elicit enhanced neuronal responses in the aSPPO region. These questions may be of relevance for future studies interested in its functional role.

Previous research indicates that activity in the motor cortex can be important for phoneme categorization²⁷. It is engaged in phonological discrimination⁵⁷ and correlates with successful categorical perception of consonants in healthy but not in dyslexic subjects⁶⁴. Our study shows that activity in the same articulatory motor area (Figs 1A, ​,3,3, Suppl. Fig. ¹) starts early in the time course of speech production and ca. 500ms later in speech perception (aSPPO region in Fig. 4A). In line with³, this leads us to tentatively suggest that activity in the aSPPO region might reflect the processing of intentions to produce a certain phoneme in speech production and the retrieval of this intention during speech comprehension. Further evidence to validate this possibility is needed.

ECoG acquisition and trial selection

Electrocorticography (ECoG) grids of 8×8 platinum/stainless-steel electrodes (4-mm diameter, 10-mm centre-to-centre inter-electrode distance) were implanted on the surface of the left fronto-temporo-parietal cortex. ECoG was recorded 24/7 over the entire period of pre-neurosurgical diagnostics of one to two weeks with a clinical AC EEG-System (IT-Med, Germany) at a 1024-Hz sampling rate using a high-pass filter with a cutoff frequency of 0.032Hz and an antialiasing, low-pass filter at 379Hz. 25-frame-per-second digital video and audio data were recorded synchronized with ECoG as a part of the diagnostic procedure. Each video frame corresponded to ca. 40 ECoG data points. All ECoG data points were assigned to the concurrent video frame. The neuronal recordings were not downsampled to this resolution, as this would have resulted in losing much of the temporal and hence frequency resolution. ECoG recordings were acquired and visualized together with the audiovisual materials using the Coherence PSG 299 System (Deltamed, Paris). This software allows identifying behavioral and/or neuronal events and their post-hoc tagging in the ECoG signal.

Audiovisual data were manually screened to select recordings during which the subjects were alert, conscious and engaged in real-world face-to-face conversations with visitors or medical staff (Table 1: right-most entries in column 7). Speech production and perception were defined from the point of view of the subject: ‘Production’ refers to the subject’s own overt speech and ‘perception’ refers to the overt speech of other individual discourse participants. Speech perception data were selected only from conversations in which the subjects were direct participants. Recordings during overlapping talk and during conversations in the background were not analyzed. Speech production and perception events were detected by audiovisual inspection of the audio-video data and tagged at their onset in Coherence based on their aural identification by the investigator frame by frame. Since psycholinguistic literature shows that pauses >200ms are perceived by humans as ‘pauses’ in a conversation⁶⁹, a single epoch of speech production or perception was defined by the absence of such pauses in it. Speech production and perception were annotated at their onsets if the duration of the respective events was at least 1000ms. The resulting speech production and perception trials were analyzed in the time periods of -1000 to 1000ms relative to their onset to make sure that speech production and perception in all trials were taking place through the second 1000ms of this time window. Speech production and perception events were discarded when accompanied by visible arm, leg, trunk, head movements or by loud non-speech background noises (e.g., objects falling/being moved, phone ringing, people coughing etc.) in the time period of at least 1000ms before and 1000ms after onset. Recordings lying ≤30min before and/or after epileptic seizures were excluded to minimize the impact of pathological activity on the ECoG signal. ECoG signals from all channels of the grid were re-referenced to a common average by subtracting the mean over all channels from each channel at each sampled data point.

Analysis of speech with background noise

One hypothesis to be tested was that the presence of noise during auditory speech perception might enhance activity in the motor cortex due to stronger recruitment of feed-forward prediction mechanisms used to reconstruct speech in noisy environments during natural communication^4,40. To address this possibility, we collected additional speech perception events which took place in noise-free conditions of speech perception (9^th column, entries 2 from the left in Table 1) or in the presence of loud non-speech background noises (9^th column, entries 3 from the left in Table 1). Trials in the selected noise-free condition were mostly a subset of the speech perception data described above, and they were identified by discarding trials with low-volume yet audible background noises (e.g., people sighing, quiet steps outside the ward, etc.) in the speech perception data used for the perception-production comparison. These subsets of speech perception data were gathered in S1–S7, and S8 was not evaluated due to the limited amount of suitable material. The speech-in-noise data were collected as follows. Based on a visualization of the auditory signal in ELAN 3.0 (available from http://tla.mpi.nl/tools/tla-tools/elan/elan-old-versions), we first identified noises with high amplitude of the auditory signal. Then, we inspected these for the presence or absence of simultaneous receptive speech. If the other aforementioned inclusion criteria applied (i.e., no movements or speech production of the subjects accompanied these auditory events 1000ms before and after their onset plus if the duration of the speech perception event was at least 1000ms), and if noise onsets took place around the onset of speech perception, the speech-in-noise epochs were selected and tagged in the ECoG data using the same procedure as described above. We originally intended to collect speech-in-noise-perception epochs with post-onset noises which were at least 1000ms long. Since naturally-occurring noises proved only rarely that long, however, the inclusion threshold had to be lowered to at least 200-ms noise duration to be able to collect a sufficient number of trials⁴ from the available data (6^th column in Table 1).

Analysis of non-speech noise

To find out if activity in the motor cortex also took place during perception of non-speech noises, we collected and analyzed ECoG data during the perception of loud background noises (last column in Table 1). The same selection criteria as for the noises in the speech-in-noise condition apply.

Assignment of electrodes to functional areas

To delineate the locations of the individual functional areas, ESM was performed using an INOMED NS 60 Stimulator (INOMED, Germany) that was temporally aligned with the ECoG recording system. The electrical stimulation artefacts were directly and clearly visible in the ongoing ECoG recordings. Trains of 10-s pulses at 50Hz with alternating-polarity square waves of 250 μs were applied to non-overlapping pairs of neighboring electrodes. Monopolar stimulation was performed against a functionally-neutral intracranial reference channel to spatially resolve the effect observed upon bipolar stimulation. Stimulation intensity was gradually increased until the induction of a motor effect up to 15mA or up to 18mA while testing speech functions. An electrode was attributed a motor, sensory, and/or speech function if the respective effect occurred at any of the tested stimulation intensities. These functional effects were observed within the several first seconds after stimulation or during it, and they were annotated regardless of the exact timing of the effect relative to stimulation onset. An electrode was labelled as functionally neutral if no functional effect was observed when the maximum current intensities were applied. Fig. 1A provides ESM results in S2, and Suppl. Fig. ¹ offers an overview of the data from S1, S3–S8. Additional details on the functional assignment of electrodes via ESM are available elsewhere³⁰.

Both motor effects of body parts relevant to articulation and transient speech production and/or perception effects can be considered as potentially important for language functions. The former effects were defined whenever ESM elicited a movement or impeded a movement of a body part that is involved in articulation (e.g., tongue, jaw, lips, or diaphragm). The latter effects were defined using a battery of six tasks (counting, execution of body commands, naming objects, reading, repetition of sentences, and a Token Test)⁷⁰ aimed to test whether the subject is capable of understanding and/or producing language upon ESM. As opposed to mouth motor effects, transient post-stimulation impairments in speech production and/or comprehension (e.g., if a person is no longer able to name days of the week in a correct sequence) are interpreted as indications that the underlying cortical area is essential for cognitive language functions⁷⁰.

Delineation of language-essential areas with ESM, however, can be challenging due to the following reasons: On the one hand, it is conceivable that not all ESM-identified mouth motor electrodes are relevant to articulation and that some of them may support other functions (e.g., smiling or eating). On the other hand, some electrodes without language or mouth motor functions in ESM may still record neuronal activity from areas that are important for language⁴. Keeping in mind these considerations, we defined potentially articulation-relevant areas as follows: they had to show both significant high-gamma responses during speech production and mouth motor effects upon ESM.

Assignment of electrodes to anatomical areas

As in our previous studies^28,30,32,71, we used a hierarchical probabilistic method which relies on structural post-implantation images to assign electrodes to anatomical areas. T1-weighted magnetic resonance images (MRIs) with full-head coverage were acquired in all subjects using a 1.5-T Vision MRI scanner (Siemens, Erlangen, Germany) with a magnetization-prepared rapid-acquisition gradient-echo sequence at a resolution of 1×1×1mm. These data were converted into a Matlab-compatible format with MRIcro⁷² and spatially normalized to the standard Montreal Neurological Institute (MNI) template in SPM5⁷³, anatomy toolbox v17. We visualized the MRI data in 3D with the help of custom Matlab-based software, identified and manually marked each electrode as well as the individual positions of the central and lateral sulci. The electrodes were first assigned to the frontal, parietal, and temporal lobes depending on their positions relative to the sulci in the respective subject. A probabilistic atlas system⁷⁴ was used to assign electrodes to most likely anatomical areas of the cortical surface within each lobe based on their MNI coordinates. Additional information on this procedure is available in³⁰. Note that our anatomical assignment results based on the MNI coordinates (Suppl. Table ¹) may deviate from those obtainable, e.g., using SPM alone due to the fact that we account for individual anatomical landmarks and make orthogonal projections of electrode positions outside of the standard brain onto the SPM-normalized cortical surface. The original MNI coordinates prior to this projection are reported in Suppl. Table ¹. Also, note that assignments to the parietal operculum refer to the subregions of this anatomical area that extend to the outer brain surface, e.g., to parts of regions OP 4 (PV) and OP 1(SII) as depicted in Figs 4–7 of⁷⁵. Finally, note that, following the conventions of the SPM anatomy toolbox (see also ref.⁷⁶), the primary motor cortex was defined as equal to BA4. In the vicinity of aSPPO, BA4 is situated within the depth of the central sulcus¹⁸. Since ECoG is recorded from the cortical surface, electrodes are not in direct contact with hand- and face-related BA4; the electrodes anterior to the central sulcus record from the premotor cortex (BA6), which adjoins the primary motor cortex in this direction. Whenever we refer to the anatomical motor cortex in the present study, we mean the premotor cortex including electrodes extending onto the central sulcus for these reasons.

Spectral analysis and statistics

We calculated time-resolved spectral magnitude values in each of the selected 2000-ms excerpts of the neuronal data using a time-resolved Fourier transformation with a sliding window of 200ms, a time step of 20ms and 5 Slepian tapers⁷⁷. The absolute spectral magnitude at each time-frequency bin at each electrode was baseline-corrected. The baseline period corresponded to the first 200ms of the 1000-ms epoch of silence preceding the onset of the respective trial. The absolute spectral magnitudes in this period were median-averaged over time and frequency, and relative spectral magnitude changes (RSMC) in each condition were calculated by dividing the time- and frequency-resolved absolute spectral magnitude of each trial by its baseline activity.

The RSMC in each subject were then median-averaged over trials in the respective condition. Gamma activity can behave differently in different frequency components of the ECoG signal⁷⁸, and inter-subject variability of gamma response properties can be observed in the same cortical region during the same task⁵⁵. We therefore investigated three high-gamma frequency bands (70–100Hz, 100–200Hz, and 200–350Hz) to ensure that the obtained results did not critically depend on our definition of the gamma-signal component. The statistical significance of RSMC in each condition was tested against the baseline using a two-tailed Wilcoxon sign test implemented in the signtest function in Matlab (Version R2012b). An example of time-resolved RSMC is provided in Fig. 1B,C, and band-averaged RSMC are illustrated for one frequency band in Fig. 2 (S2).

Timing of neuronal activity during speech production and perception

We exploited the high temporal resolution of ECoG data to test whether activity in the overlap region between speech production and perception in the anatomical motor cortex would occur early in the time course of speech production as predicted by some theoretical accounts³. We were also interested to find out, how the timing of activity in the overlap region between speech production and perception in the anatomical motor cortex would compare to that in the other anatomical areas in the production and perception conditions. We computed the earliest time points at which the high-gamma RSMC were significant (Wilcoxon sign test, FDR-corrected at q=0.005 for multiple comparisons over the number of tested electrodes and time points within each frequency band) in the analyzed time window of [−1000 1000] ms relative to speech production and perception onset. Within each condition, the earliest time point at which the high-gamma RSMC were significant was first identified for each electrode, subject, and frequency band separately. The results of this computation in each subject and frequency band for each electrode belonging to the given anatomical area (not visualized) were median-averaged across all electrodes belonging to this anatomical area (Suppl. Table ², not visualized). A median over subjects for each area was subsequently calculated (Suppl. Table ² and vertical colored bars in Fig. 4A). Since the number of significant values across subjects for some areas was small (Fig. 4A and Suppl. Table ²), a bootstrapping procedure was used (csboot function in Matlab, 100 repetitions) to estimate the standard error of the median (vertical black lines in Fig. 4A). Next, we compared the resulting earliest time points of significant activation using the Wilcoxon rank sum test at p=0.05 (i) between the nine anatomical areas with significant high-gamma RSMC effects within each condition (black brackets with stars indicating the level of significance next to them in Fig. 4A) and (ii) within each of these individual areas between speech production and perception (red brackets with stars indicating the level of significance next to them in Fig. 4A).

Response magnitudes during speech production and perception

We also estimated the strongest high-frequency response magnitudes (Suppl. Table ³) and compared them between anatomical areas within the two speech conditions and within the same anatomical area between conditions (Fig. 4B). Exactly the same procedure was applied as for the calculation of the earliest time points of significant activation described above. The highest high-gamma response magnitudes of the significant responses were used in this analysis instead of the earliest significant time points. The statistical differences in the maximum RSMC between the different conditions were also assessed using a two-tailed Wilcoxon rank sum test (ranksum function in Matlab). A FDR procedure was applied for each frequency band at q=0.05 to account for multiple testing over time bins and grid electrodes.

We thank DFG-GRK 1624 ‘Frequency effects in language’ (University of Freiburg), BrainLinks-BrainTools (DfG grant EXC1086, University of Freiburg) and the Baden-Württemberg Stiftung (grant NEU018) for financial support.