Stimuli

Stimulation consisted of visual noise presented on a gray (50% of maximum pixel intensity, luminance: 321cd/m²) background, which could contain an embedded horizontal or vertical grating (Fig. 1B). Noise patches consisted of white noise that was subsequently smoothed with a Gaussian kernel (SD=0.05°). Gratings were sine waves with a spatial frequency of 1cycle/° and random phase. The gratings were embedded in the noise by averaging the two images, weighted according to a desired noise level (0%: full-contrast noise-free grating; 100%: pure noise). The pixel values of the resulting image were rescaled such that the minimum and maximum values mapped onto 0% and 100% of maximum pixel intensity, respectively. Finally, to obtain an annulus, all stimuli were masked with a radially oriented Gaussian mask (SD=2°) centered at a radius of 6°, resulting in an overall diameter of approximately 24°. Stimuli were generated and presented using MATLAB (The Mathworks, Inc., Natick, Massachusetts, United States) and the Psychophysics Toolbox extensions²³.

Open in a separate window

Figure 1

Experimental paradigm and behavioral results.

(A) In the sensory processing blocks, the noise/grating stimuli were irrelevant and unattended. In the perceptual decision making blocks, a decision had to be made regarding the presence or absence of a grating. Grating visibility is enhanced for illustrative purposes. (B) Example stimuli. The noise level in perceptual decision making blocks was tailored to individual detection thresholds. (C) Average response proportions in perceptual decision making blocks, for grating present and grating absent trials, color-coded according to the four stimulus/response-categories. The numbers denote the average number of trials available in each of these categories. Error bars depict standard deviations.

Procedure and experimental design

Each subject participated in a behavioral practice session in order to become familiar with the experiment. The practice sessions were scheduled at most two days before the main experimental session. The main session involved three different types of blocks. After an initial staircase block (see below), subjects performed six sensory processing (localizer) blocks and six perceptual decision making blocks in alternating order, starting with a sensory processing block.

In the staircase block, we used the Quest staircase procedure²⁴ (as implemented in PsychToolbox) to estimate the individual noise level at which each subject correctly detects a grating embedded in noise in 70% of the cases. The procedure was similar to the perceptual decision making blocks (see below), except that subjects only had to indicate the presence or absence of a grating and not its orientation. In addition, subjects received feedback on every trial. Only the trials in which a grating was actually presented were used to update the staircase. For the last eighteen subjects, convergence of the staircase was visually inspected at the end of the block and, if convergence was not yet achieved, more staircase trials were administered.

Each sensory processing block comprised 120 trials during which subjects were presented with a brief stimulus for 50ms (Fig. 1A). In 50% of the trials, the stimulus was pure noise (referred to as noise trials) whereas the other 50% contained a grating embedded in the noise (referred to as grating trials). In half of the grating trials, the grating had a horizontal orientation whereas a vertical grating was present in the other half. The noise level of the grating trials was set to 90%. This value was chosen such that it was sufficiently high to be comparable to the stimuli presented during the perceptual decision making blocks, yet low enough to ensure clear visibility of the gratings (Fig. 1B). For the first seven subjects, the inter-trial interval was fixed to 950ms, whereas for the remaining fifteen subjects this interval was randomly drawn from a uniform distribution between 850 and 1050ms. As we were specifically interested in the neural signature of sensory processing in the absence of higher-level attentional and/or decision processes, subjects were required to perform a task at fixation to draw attention away from the stimuli. In 10% of the trials, and balanced across stimulus conditions, the fixation dot (diameter 0.2°) was absent during the 50ms stimulus presentation. Subjects were instructed to report such a “blink” by pressing a button as quickly as possible. These “oddball” trials, as well as the non-oddball trials on which subjects erroneously pressed a button, were excluded from further analyses.

A perceptual decision making block comprised 80 trials. Each trial began with a 1000ms fixation period, after which a stimulus was briefly presented for 50ms (Fig. 1A). Again, this stimulus was either pure noise (in 50% of trials), or contained a grating (vertical grating in 25% and horizontal grating in 25% of the trials). Unlike the sensory processing blocks, the noise level was set to a much higher level, namely the individual threshold of perception as determined by the staircase procedure. Then, after 600ms the letters ‘y’ and ‘n’ (as abbreviations for “yes” and “no”, respectively) were displayed, centered around the fixation dot. Subjects reported their decision as to whether they had perceived a grating or not by pressing a button with either the left or the right hand, corresponding to the position of the letter that matches their decision. The position of the letters (‘y’ left and ‘n’ right, or ‘n’ left and ‘y’ right) was randomized across trials to orthogonalize perceptual decision and motor response preparation. Trials in which no button press was made were discarded from further analysis. After button press, or after 2000ms in case of no button press, another fixation period of 300ms followed, after which a second display with a horizontal and a vertical line was presented to inquire the subject’s decision regarding the orientation of the grating. The position of the lines were also randomized across trials. Subjects were instructed to, in trials where they perceived only noise, indicate the orientation to which they thought the noise was most similar. Finally, a blank screen was presented with an inter-trial interval drawn randomly from a uniform distribution between 500 and 1500ms

The two different grating orientations were included in the design because we had hypotheses regarding not only the sensory processing of stimulus presence versus absence, but also regarding the processing of orientation-specific signals. However, since we were not able to successfully decode the orientation of stimuli from the MEG signal, stimulus orientation was left out of further consideration.

Behavioral analysis

The observer’s sensitivity d’ and criterion c were calculated as follows:

where Z(…) is the inverse standard normal distribution.

MEG recording and preprocessing

Whole-head neural recordings were obtained using a 275-channel MEG system with axial gradiometers (VSM/CTF Systems, Coquitlam, BC, Canada) located in a magnetically shielded room. Throughout the experiment, head position was monitored online, and corrected if necessary, using three fiducial coils that were placed on the nasion and on earplugs in both ears. Behavioral responses were made using two MEG-compatible button boxes, one for each hand. Visual stimulation was projected from outside the magnetically shielded room, via mirrors onto a screen in front of the subject. Furthermore, both horizontal and vertical electrooculograms (EOGs), as well as an electrocardiogram (ECG) were recorded to facilitate removal of eye- and heart-related artifacts. All signals were sampled at a rate of 1200Hz.

The data were preprocessed offline using FieldTrip²⁵ (www.fieldtriptoolbox.org). Notch filters were applied at 50, 100 and 150Hz to remove line noise and its harmonics. In order to identify artifacts, the variance (collapsed over channels and time) was calculated for each trial. Trials with large variances were subsequently selected for manual inspection and removed if they contained excessive and irregular artifacts. Independent component analysis was subsequently used to remove regular artifacts, such as heartbeats and eye blinks. Specifically, for each subject, the independent components were correlated to both EOGs and the ECG to identify potentially contaminating components, and these were subsequently inspected manually before removal. For covariance computation in the source localization, the remaining components were transformed back to sensor-space and subsequently low-pass filtered at a cut-off frequency of 30Hz. For the decoding analysis however, the components were kept in component-space to ensure that its covariance matrix is of full rank as required for this analysis (see below). Finally, the data were baseline corrected on the interval of −200 to 0ms relative to stimulus onset.

Decoding analysis

The general idea behind a neural decoding analysis is that it attempts to invert the encoding process. The encoding process determines the neural responses as a function of some parameter, for instance a physical stimulus or an experimental condition. A decoding analysis aims to invert this function in order to unveil the encoded parameter as a function of neural signals. The first step in achieving this is to estimate a forward model that describes the encoding process. For this, a data set is used in which the parameter is known and the corresponding neural signals are recorded empirically. Secondly, an inverse model is estimated on the basis of the forward model and subject to some criterion of optimality. We will refer to such an inverse model as a decoder, as it takes neural signals as input and produces an estimate of the encoded parameter as output.

Our method is largely based on linear discriminant analysis as described in ref. ²⁶. Linear discriminant analysis attempts to find a linear transformation of the data, such that the resulting signal is optimally discriminative between two classes. First, we demeaned the data such that for each time point and for each feature, the average over trials equals zero. Features here refer to independent components (see MEG recording and preprocessing). Let and be column vectors of length F, where F is the number of features, that contain the neural responses in the training set for class 1 and 2, respectively, at some time point and averaged across trials. Then, the weights vector w that optimally discriminates between classes on the basis of the features is given by²⁶:

where is the common regularized covariance matrix (see below). Next, let X be a matrix of size F×N, where N refers to the number of trials, that contains the data to be decoded. The decoded signal y is then obtained by:

where (…)^T denotes the matrix transpose. In linear discriminant analysis, this signal is transformed into a discrete class membership by specifying a cut-off value for y. Here, we deviated from standard linear discriminant analysis. Rather than assigning a discrete label to a trial, we were interested in a continuous measure of the degree to which a class is encoded in the neural signals. Thus, we did not apply a binary cut-off to the decoded signal. Furthermore, to make this signal comparable across time points, we added a normalization factor to the weights vector such that the mean difference in the decoded signal between classes equals a value of one. This is accomplished by modifying equation (3) into:

in which the denominator is the normalization factor. Thus, equation (4) and (5) constitute our final decoder. We term the output of the decoder the “discriminant channel”, as it optimally discriminates between the two classes that it was trained on. Specifically, if we denote the mean discriminant channel amplitude in class 1 and 2 by and , respectively, then we expect if there is information in the neural signals pertaining to the classes, whereas we expect if no such information is available. In other words, the difference in the discriminant channel between classes is a measure of the discriminability of these classes on the basis of the neural signals.

The interpretation of the decoder’s output as a discriminant channel is further motivated by drawing an analogy to linearly constrained minimum variance (LCMV) spatial filters²⁷, more commonly known as beamformers. A beamformer is an inverse model to estimate neural activity at the source level, given sensor-level activity. Beamformers are commonly used to extract time courses of activity of some source-level region of interest into so-called virtual channels, as if the activity in that region had been recorded directly. The forward model in this method, also known as the leadfield, describes how sensor-level activity varies as a function of source-level activity. The analogy is made by noting that equation (5) is equivalent to the calculation of beamformers, but whereas the forward models for beamformers are defined in a spatial sense, the forward model in our decoding method is defined in a discriminatory sense. It describes how activity at the sensor-level varies as a function of a discriminating parameter, namely the class, and is in fact the difference event-related field . If this difference event-related field is entered in the LCMV beamformer formula²⁷ as the forward model, then equation (5) is the result. Thus, the output of our decoder is analogous to a virtual channel, but defined in a discriminatory rather than a spatial sense, hence the name discriminant channel. Contrary to spatially defined virtual channels, the discriminant channel may stem from a wide array of neural sources, and its output is the collective activity of this array. Finally, we point out that the normalization in equation (5) corresponds directly to the unit-gain constraint in LCMV beamformers²⁷.

To facilitate comparison to other decoding methods, we included the results in the Supplementary Information normalized by their standard deviation at the individual level, yielding grand averaged Cohen’s d effect sizes. Moreover, we also carried out an additional analysis in which we did set a cut-off value in order to assign a discrete class to each trial. This value was chosen as , because this results in equal type I and type II error rates²⁸. By assigning discrete labels to trials, the results can be expressed as the proportion of trials that are assigned to either of two classes.

An important element in the decoding analysis is that it takes advantage of correlations between features in order to suppress noise. Let the column vector x_i of length F denote the data in trial i, and define the corresponding mean as:

then the estimated covariance matrix is obtained by:

For optimal noise suppression, we improved this estimation by means of regularization by shrinkage, using the analytically determined optimal shrinkage parameter (for details, see ref. ²⁶). This procedure was performed separately within each condition, and the resulting condition-specific regularized covariance matrices were subsequently averaged to obtain the common regularized covariance matrix .

The decoding analysis outlined above was performed in a time-resolved manner by applying it sequentially at each time point, in steps of 5ms, resulting in an array of decoders. To improve the signal-to-noise ratio, the data were first averaged within a window of 29.2ms centered around the time point of interest. The window length of 29.2ms is based on an a priori chosen length of 30ms, but minus one sample such that the window contained an odd number of samples for symmetric centering. Thus, the output of the time-resolved analysis is a one-dimensional time series of the amplitude of discriminant channel, for each individual trial.

It is imperative that the training data set is independent from the data set that is to be decoded in order to avoid “double dipping”²⁹. Therefore, we adopted a leave-one-out approach whenever a trial was decoded that belonged to one of the classes on which the decoder was trained. Note that no such procedure is required when the decoded data set stems from a class different from those used in training, for instance when generalizing across conditions or across blocks.

To facilitate a neurophysiological interpretation of the decoded signals, we derived a spatial projection of the discriminant channel. This spatial projection displays the coupling between the sensors and the discriminant channel and as such gives insight into which sensors contribute to the discrimination. The spatial projection a is given by³⁰:

and this was subsequently fed into the source localization (see Source localization).

The key aspect of the current study is that we used a functional localizer in order to extract the neural signature of sensory-specific processes, and subsequently traced this signature during perceptual decision making. This was done by training the decoders on the data from the sensory processing blocks and subsequently applying these decoders to the data from the perceptual decision making blocks. That is, the decoders were generalized across blocks. In addition to this generalization, we also trained and decoded within both the sensory processing and the perceptual decision making blocks separately. Within the sensory processing blocks, the decoders were trained to discriminate between noise trials and grating trials. Within the perceptual decision making blocks, the decoders were trained to discriminate between CRs and hits. This contrast was chosen, because these conditions correspond to accurate perceptual decision making and are therefore commonly used as a baseline to which inaccurate perceptual decisions (i.e. FAs and misses) are compared. These decoders were used to decode both CRs and hits themselves (using the leave-one-out approach), as well as the FAs and misses.

Finally, we implemented the temporal generalization method to elucidate the temporal organization of the neural processing stages that underlie sensory processing and perceptual decision making³¹. When training a decoder on any specific time point, we simultaneously applied this decoder to all other time points. After averaging the obtained discriminant channel amplitude over trials, this results in a (training time)×(decoding time) matrix per condition. Comparing these matrices for two specific conditions, by subtracting one from the other to obtain a difference matrix, provides insight into how discriminability between these conditions generalizes across time. For instance, a row in such a difference matrix, corresponding to some time point t_train, describes how well the two conditions can be discriminated over time on the basis of a decoder that is specifically trained, i.e. is optimally discriminative, at t_train. Conversely, a column, corresponding to some time point t_decode, gives insight into how well the two conditions can be discriminated at time point t_decode on the basis of the decoders trained on all other time points. As another example, consider the temporal generalization matrices obtained by generalizing from sensory processing to perceptual decision making, and consider in particular the average temporal generalization matrices for hits and CRs. Then, the difference between these matrices at the entry corresponding to some training time t_train and decoding time t_decode is a measure of how well hits can be discriminated from CRs at t_decode, on the basis of the weights that was maximally discriminative between noise and grating trials at t_train. The rationale behind the temporal generalization method is that the neural pattern identified by the decoder to be discriminative corresponds to some underlying neural process, and this pattern should be generated whenever that neural process is active. Thus, by testing for the presence of a particular neural pattern, one obtains an activity time course of the corresponding neural process. Note that the temporal generalization method described here is not to be confused with the between-block generalization described above.

Statistical testing

Statistical analyses were performed on subject-level temporal generalization matrices, averaged across trials. Contrasts between conditions were tested for statistical significance using permutation tests in conjunction with cluster-based correction for multiple comparisons³². Specifically, univariate paired t-tests were calculated for the entire matrix. Elements that passed a threshold value corresponding to a p-value of 0.05 (two-tailed) were marked, and neighboring marked elements were collected into separate negative and positive clusters. No specific constraint was set on the minimum number of marked elements in order to be considered a cluster - that is, the minimum number of required neighbors is 1. Elements were considered neighbors if they were directly adjacent, either cardinally or diagonally. Finally, the t-values within each cluster were summed and rectified, and these values were fed into the permutation framework as the test-statistic. Consequently, all tests were two-tailed. A cluster was considered significant when it’s p-value was below 0.05. The number of permutations per contrast was 10000.

Identifying sensory-specific neural processing

We first investigated whether we could reliably extract a neural signature that is specific to sensory processing. To this end, we focused on the neural responses during the sensory processing blocks. In these blocks, subjects were presented with noise stimuli, half of which contained an embedded grating. As these stimuli were not relevant for the task at hand,neural activity during these blocks is thus specific to sensory processing, uncontaminated by decision processes. Moreover, attention was drawn away from the stimuli towards a task at fixation, thereby minimizing potential effects of attentional modulation on the sensory processing. Although there is the theoretical possibility that the stimuli nevertheless drew attention in an automatic fashion, this bottom-up effect is presumably much weaker than the voluntary attentional engagement during perceptual decision making, and therefore most likely does not pose a problem for the present study.

We applied our decoding analysis to these data and extracted a discriminant channel that is maximally discriminative between noise and grating trials (Fig. 2A; see Supplementary Figures S1 and S2 for the same results expressed as classification accuracy and Cohen’s d, respectively). The results show that these two conditions can be reliably discriminated on the basis of the MEG recordings as evidenced by a significant cluster (p<0.001) that extends from 60ms post stimulus onset throughout the rest of the segment. The discriminability peaks at 80ms and 160ms, after which it gradually decays back toward baseline.

Open in a separate window

Figure 2

Decoding results for training and decoding within the functional localizer (A,D), for generalization from the functional localizer to correct perceptual decisions (B,E) and for training and decoding within correct perceptual decisions (C,F).

(A–C) Average discriminant channel activity for noise and grating trials (A) and CRs and hits (B,C), at matched training and decoding time. The horizontal black bars mark time points that belong to a significant cluster as outlined in (D–F). Shaded areas depict the SEM. (D–F) Temporal generalization matrices. Note that the diagonals are identical to the differences between the curves in (A–C) and note that the color scales are variable across figure for optimal visualization. Significant clusters are demarcated by the contours. (G,H) Source level contributions to the discriminant channel trained on the functional localizer (G) and on perceptual decision making (H), at four specific time points.

Next, we calculated the temporal generalization matrix of the discriminant channel to probe the temporal organization of the neural processing stages underlying sensory processing. This matrix contains the activity of the discriminant channel over time, while trained on all other time points. That means that the temporal generalization matrix provides insight into whether the conditions can be discriminated at some time point on the basis of the weights that are maximally discriminative at some other time point. The results show that, during sensory processing, the temporal generalization profile is largely located around the diagonal (Fig. 2D), indicating that a given discriminant channel only generalizes to temporally proximate neural signals. This is known as a “chain” profile³¹ and suggests that distinct, sequential neural processes are involved in the encoding of gratings versus noise. In order to interpret what these neural processes are, we assessed the source-level projection of the discriminant channel (Fig. 2G). This projection depicts the contribution of neural sources to the discriminant channel, and thus provides insight into the brain areas involved in the encoding of the stimuli. The results show that contributions stem primarily from occipital areas at both discrimination peaks, but especially prominently for the later one.

To summarize, we found that we could reliably extract a neural signature over time that is characteristic of sensory processing, in absence of decision making. This neural signature is evident throughout most of the trial and peaks at 80 and 160ms. It encapsulates distinct, sequential neural sources over time, and these are located primarily in occipital cortex, consistent with sensory processing.

Tracing sensory-specific processing during perceptual decision making

We proceeded to trace the sensory-specific neural signature while subjects were engaged in perceptual decision making. For this, we trained the decoders on the sensory processing blocks, serving as a functional localizer, to discriminate between noise and gratings and subsequently applied these decoders to the neural signals obtained during perceptual decision making.

We first focused on the trials in which subjects correctly reported the presence (hits) or absence (CRs) of a grating and asked whether these categories can be reliably distinguished on the basis of sensory processing. Figure 2B depicts the discriminant channel activity for these two conditions. We found a significant cluster (p=0.006), indicating that these categories are indeed different in terms of sensory processes. Interestingly, this cluster extended from 130 to 320ms, whereas no differences between CRs and hits were found for earlier or later time points. In addition, the cluster exhibited a relative late onset as compared to the early onset of discriminability in the sensory processing blocks. This inability to discriminate between CRs and hits before 130ms and after 320ms is likely due to the threshold-visibility of the gratings.

Sensory processing during perceptual decision making was found to be qualitatively different from during the functional localizer, as revealed by the temporal generalization matrix (Fig. 2E). While discriminability between CRs and hits is significant along the diagonal, i.e. when training and decoding times are matched, a substantial portion of the temporal generalization profile is found below the diagonal. This elongated shape, known as a “sustained” profile³¹, indicates that CRs and hits can be differentiated throughout an extended period of time, ranging from approximately 130 up to 400ms, using the weights that discriminated between gratings and noise trials during the shorter interval of approximately 130 to 250ms. In other words, the sensory representation of the stimulus, as defined during this relatively early period, is maintained over time.

Direct comparison of the between-task generalization (Fig. 2E) to the temporal generalization matrix depicting sensory processing only (Fig. 2D) indeed revealed a significant positive cluster (p=0.04) that extended from approximately 200 to 400ms decoding time and 130 to 200ms training time (Fig. 3). In addition, a significant negative cluster (p=0.003) was found around the diagonal that extended from approximately 100ms to 400ms, showing that the diagonal decoding performance is worse during perceptual decision making than during the functional localizer. This difference is most likely due to the increased stimulus noise level during the perceptual decision making task, as compared to the localizer. In summary, these results suggest that although the overall sensory representation was weaker during perceptual decision making, the early part of this representation was stabilized and kept online in the visual system, but only when attended and/or required for a subsequent decision.

Open in a separate window

Figure 3

Comparison of the temporal generalization matrices depicting sensory processing during perceptual decision making (Fig. 2E) and during unattended sensory processing (Fig. 2D).

Significant clusters are demarcated by the contours.

Tracing the neural signatures of correct perceptual decisions

We also examined whether we could discriminate between the different stimulus/response-categories within the perceptual decision-making task, by performing an analysis akin to more conventional approaches, in which we did not make use of the sensory processing blocks as a functional localizer but looked at the data obtained during perceptual decision making only.

We trained the decoders to discriminate between CRs and hits and calculated the average amplitude of the discriminant channel for these categories. We found that neural processes associated with these categories begin to diverge at 155ms and, contrary to the generalization analysis in which we decoded sensory-specific processing only, this difference remained significant (p<0.001) throughout the rest of the trial (Fig. 2C). As CRs differ from hits not only in terms of the presented stimulus, but also with respect to the behavioral decision, the later activity differences likely reflect differences in decision-related activity - activity to which the decoders trained on the sensory processing blocks were blind.

The interpretation of the later activity as representing processes distinct from sensory encoding is corroborated by the temporal generalization matrix. In addition to the rising discriminability along the diagonal, we observed long-lasting sustained activity throughout the interval of approximately 250ms post-stimulus until the end of the trial, meaning that CRs could be differentiated from hits during this interval using the weights obtained from any other time poi Direct comparison of the between-task generalization nt in this same interval. This “sustained” profile³¹, that can be observed in addition to a chain profile along the diagonal, shows that at least some of the neural sources remain active throughout the rest of the trial. Earlier work in monkeys demonstrated that neurons encoding for the monkey’s decision remain active in a sustained manner when the behavioral report of the decision is postponed over a delay period³⁵. Thus, our results are consistent with the interpretation that the later activity reflects decision-related processing. Conversely, earlier discriminant channel activity prior to this sustained activity, up to approximately 250ms, does not show this widespread temporal generalization, suggesting that different processes are at work in this time window, possibly transient sensory processes.

Finally, to substantiate these interpretations, we asked which brain areas contribute to the discrimination between CRs and hits. At both 160ms and 300ms, discriminant activity is observed selectively over occipital cortex. Interestingly, and in line with the interpretation that the later signal reflects a decision process, discriminant activity at 550ms is much more widespread and encompasses parietal and frontal cortices, which are often implicated in decision making^1,2. It is also noteworthy that the contribution in occipital regions to the discriminant channel increases from 160ms to 300ms when a perceptual decision is made, whereas it decreases between these two time points in the functional localizer blocks. Indeed, this is in agreement with the observation that the sensory representation in the brain is enhanced and maintained over time when required for the task at hand.

This study was supported by The Netherlands Organisation for Scientific Research (NWO Research Talent 406-13-001). We are grateful to Matthias Ekman for helpful comments on an earlier version of this manuscript.