2.2. Visual display

Stimuli were generated using the Psychophysics Toolbox version 3.0.11 (Brainard, 1997) in MATLAB 2014b (MathWorks, Natick, MA). The stimuli were presented on a 27-inch monitor (1920 × 1080, 144 Hz; default gamma correction of 2.2). The monitor was integrated in a noise-proof booth with a glass barrier right in front of the display. Viewing distance was 90 cm from the monitor where participants were stabilised at a fixed position using a chin rest. Using a photodiode we established a 24 ms delay between the stimulus triggers received on the acquisition software and the visual presentation of the stimulus on screen. We corrected for this in our analyses accordingly.

2.3. Stimuli & design

The critical stimuli for this experiment were presented within the context of a visual working-memory task that ensured attentive viewing of the visual stimuli at encoding. Task details are presented here for clarity, but the analyses only focus on decoding the memory stimuli at encoding (for a full description of the task and participants’ behavioural performance, see: Hajonides et al., 2020). At the start of each trial, two Gabor stimuli with a diameter of 4.3° of visual angle (one to the left, the other to the right of fixation, centred at ±4.3° each) were presented simultaneously for 300 ms (Fig. 1C) on a grey background (defined RGB: 0.196 0.196 0.196; measured CIE L*a*b*: 15.18, 3.146, −3.822). The stimuli consisted of bilateral luminance-defined sinusoidal Gabor gratings and each had a unique colour and orientation. Colours were drawn from a CIELAB colour space, which is often used to define perceptually uniform stimulus sets. Forty-eight evenly-spaced colours were drawn from a circle in CIELAB colour space with a fixed lightness of L = 54 (Fig. 1A; centre is at a = 18, b = −8, radius =59).

Open in a separate window

Fig. 1

Stimulus characteristics and presentation. A) Forty-eight colours - depicted in small dots - were sampled from CIELAB colour space with a fixed lightness (“L”) of 54. During analysis, colours were binned into 12 evenly-spaced bins, depicted in circles. B) Measured CIE chromaticity a* and b* coordinates on the monitor using the Colorimeter. C) Participants were presented with two Gabor gratings for 300 ms with both colour and orientation information. Colour and orientation were varied independently, using circular spaces with 48 possible feature-values (indicated in panel A and B). Participants were required to reproduce the exact colour or orientation of either stimulus after a brief memory delay, this ensured attentive encoding of the stimuli. The memory delay contained two events: a retro cue, and a visual distractor that consisted of a bilateral colour circle or two identical orientated Gabor gratings. We focus our analyses on the 1000 ms after onset of the gratings. The schematic shows an example trial prompting a report for the colour of the left grating. Stimulus durations are shown next to the tick marks along with in brackets the onset time after the start of the trial. See Hajonides et al. (2020) for the full details of the task design. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

To get a precise measurement of the obtained luminance of our stimuli on our monitor, forty-eight rendered colours were measured post-hoc using a CRS ColorCal KMII Colorimeter (Cambridge Research Systems, Kent, UK). Luminance and CIE chromaticity (in CIE xyY coordinates) were measured on a uniform colour patch containing only the average colour of the sinusoidal grating as an approximation of the grating colour with varying intensity. CIE xyY coordinates were converted to CIELAB coordinates using a d65 white point (Fig. 1B). Deviations of the rendered colours from the CIELAB colours can be caused through various ways. For example, CIELAB colours were converted using a linear colour conversion algorithm (Image Processing Toolbox, MATLAB, The MathWorks, Inc., Natick, MA) with an industry standard d65 white point. Without monitor calibration, the sampled RGB colours could deviate from originally defined CIELAB colours. Also, not all colours can be rendered by computer monitors. This is likely to impose small non-systematic imprecisions on the generation of individual colours. Finally, as previously described, there was a glass barrier between the monitor and the participant. The glass barrier could be introducing reflections, resulting in light loss. We added these deviations to our analysis to assess the relative contributions of stimulus rendering offsets and to map out (and rule out) potential alternative explanations for our colour decoding (such as decoding of luminance differences, rather than colour differences, between our stimuli).

In total, 48 different colours and 48 different orientations were presented, with feature values ranging from 3.75 to 180° in evenly spaced steps. Orientation and colour values were initially allocated randomly within each block. However, to minimise trials in which the two items had the same or very similar colour or orientation values, we redrew the colour and orientation values of all trials in a block when more than 2 trials were allocated two items with colour or orientation values within the same bin (out of 12 bins). To increase the amount of samples for feature, we binned the 48 colour and orientation features into 12 - equally spaced – bins (Fig. 1AB). To match the orientation space more closely, we opted for a symmetrical colour probe that included all colours in the 0 – π range of the probe. The colour wheel was not rotated from trial-to-trial but was always presented in the same orientation as displayed in Fig. 1C.

At the time of stimulus presentation, participants were instructed to attend to and remember both the colour and the orientation of both gratings. At the end of each trial, participants were probed to report a single feature from one of the two items. Which feature and item would be probed was unpredictable at encoding. During the 2400-ms memory delay, two events occurred: (1) 1000 ms after encoding onset a cue could provide information about the item or feature dimension that would be probed and (2) 1800 ms after encoding onset, a bilateral colour or orientation distractor was presented to distort visual processing. These trial details are not critical to the current investigation, which focuses almost exclusively on the encoding phase of the experiment. More detailed description of the experiment from which we here re-analysed the EEG data can be found in Hajonides et al. (2020).

2.4. EEG recording

The EEG signal was acquired using a 10–20 system using 61 Ag/AgCl sintered electrodes (EasyCap, Herrsching, Germany). The data were recorded using a Synamps amplifier and Curry 8 acquisition software (Compumedics NeuroScan, Charlotte, NC). An electrode placed behind the right mastoid was used as the active reference during acquisition. Offline, data were re-referenced to the average of the left and right mastoids. The ground was placed above the left elbow. Bipolar electrooculography (EOG) was recorded from electrodes above and below the left eye and lateral of both eyes. EEG data were digitised at 1000 Hz with an anti-aliasing filter with a cut-off frequency of 400 Hz. Impedances were kept below 7kΩ.

2.5. EEG pre-processing

EEG data were analysed in MATLAB 2017a using FieldTrip (Oostenveld et al., 2011) in conjunction with the OHBA Software Library (OSL; https://ohba-analysis.github.io/). After importing, the data were epoched from −300 ms to 1000 ms around presentation the presentation of the Gabor gratings (ft_redefinetrial) and re-referenced to the average of both mastoids (ft_preprocessing). Raw epoched data were downsampled to 100 Hz, and stored for subsequent analyses (analysis scripts and data is made available online at OSF https://osf.io/j289e/?view_only=b13407009b4245f7950960c34a5474a6).

Subsequently, we used the EOG to identify trials in which participants blinked and might have missed the visually presented stimuli. The vertical EOG was baselined between 300 ms to 100 ms before stimulus presentation. Trials that exceeded half of the maximum voltage elicited by blinks (half of 400 μV) within the timeframe of 100 ms prior to and 100 ms post stimulus presentation were marked and later discarded. Additional eye-related artefacts were identified using Independent Component Analysis (ICA; ft_componentalanalysis) by applying the FastICA algorithm (Hyvärinen, 1999) to the full EEG layout. Components were rejected if they had a correlation of r > 0.40 with the EOG electrodes (on average 2.7 components per participant, and a maximum of 4 components). In addition to trials where participants blinked at the moment of stimulus presentation, trials with high within-trial variance of the EEG broadband signal were removed using a generalised ESD test at a 0.05 significance threshold (Rosner, 1983; implemented in OSL), removing 2.48% ± 2.18% (mean ± standard deviation) of all trials.

2.6. Classification on EEG data

To focus our decoding on visual processing, we restricted our main analyses on the 17 most posterior electrodes (P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, O1, Oz and O2; as in Wolff et al., 2017).

To make class predictions we employed Linear Discriminant Analysis (LDA) using the Scikit-learn toolbox (Pedregosa et al., 2011) and estimated the likelihood for all classes. LDA is a multi-class decoding algorithm that performs well for neuroscientific time-course data (for a comparison of classifiers see Grootswagers et al., 2017).

To increase the sensitivity of our decoding analysis, we capitalised on both spatial and temporal patterns for decoding (Grootswagers et al., 2017; Wolff et al., 2020). To do so, we used a sliding-window approach and concatenated topographical patterns from t₀ up to t_-19 (20 steps of 10 ms, ranging over 200 ms) into a single vector. This increased the amount of features in the LDA classifier from 17 (17 posterior electrodes) to 340 (17 electrodes × 20 time steps). To avoid baselining issues and to utilise dynamic activity rather than stable brain states, we subtracted the mean activity across the 20 time points within each time window for each channel and trial separately. This analysis method thus exploits embedded informative temporal variability, reduces noise by combining data over time, and circumvents baselining issues (Grootswagers et al., 2017; Wolff et al., 2020). In a supplementary analysis, we confirmed that similar (be it less robust) colour and orientation decoding could be obtained when decoding based on purely spatial patterns, on a timepoint-by-timepoint basis.

Principal component analysis was applied to the data to capture 95% of the variance in the data (sklearn.decomposition.PCA). Trial feature labels were binned into 12 classes (each subtending 12/π for orientation and 6/π for colour; numpy.digitize). These data were split into train-and-test sets using 10-fold stratified cross-validation (sklearn.model_selection.RepeatedStratifiedKFold). Next, training data were standardised (sklearn.preprocessing.StandardScaler) and after the same standardisation was applied to the test data we fit the LDA to the training data and labels (sklearn.discriminant_analysis.LinearDisciminantAnalysis; using singular value decomposition and a default threshold for rank estimation of 0.001). Likelihoods for each class were estimated after training and testing the LDA classifier separately for each time point, features (colour and orientation from left and right sides), and participant. The resulting likelihoods were centred so that the predicted class was always in centre surrounded by neighbouring feature classes. These predicted-class likelihoods were convolved with a cosine ([-$\frac{1}{2}$π, $\frac{1}{2}$π] for orientation and [- π, π] for colour) to provide a single measure of evidence. This serves to convert two-dimensional tuning curves and obtain a single measure of evidence of each time point. For additional analyses, where we looked at dissimilarity matrices, no cosine convolution was applied. Training and testing were performed separately for left and right features and only at the final stage we averaged across both sides.

To confirm the posterior locus of decodability, we also ran a searchlight decoding analysis (Kriegeskorte et al., 2006), iteratively considering a small group of electrodes. To this end, we applied the same temporal classification on the voltages between 150 and 350 ms, but this time only considered the data from a given electrode together with the data from its immediately adjacent (left/right/above/below) neighbours.

Building on RSA (representational similarity analysis; Kriegeskorte et al., 2008) we additionally investigated how our measured differences in luminance (L*) and colour (a*b*) each contributed to the identified parametric decoding of colour. RSA allowed us to examine relative contributions of different neural codes. The preceding analyses assumed a perfectly circular organisation of colours (Fig. 1A) with identical tuning curves for each colour. In practice, rendered colours slightly violate this hypothesised organisation. To test the relative contribution of luminance and colour in our data, we therefore constructed ‘dissimilarity models’ for both features using the measured Colorimeter data. For measured luminance (L*), a one-dimensional variable, we characterised the difference in luminance of each of the 12 colour bins as the absolute difference of one colour bin with all other colour bins. Doing so for each colour, we obtained a 12-by-12 matrix containing differences of measured luminance of each colour with all other colours with a value of zero across the diagonal. Similarly, we generated a model characterising differences in measured a*b*. Here, we used Euclidean distance between measured a*- and b*-coordinates of each colour bin and all other colour bins as a proxy of colour dissimilarity. A third model acted to compare these two models to the circular model (that used the desired CIELAB colour space as its basis) described above. All models were z-scored. Twice is too much, all three models were multiplied - and summed - with LDA evidence of every colour bin for each of the 12 possible classifier predictions for every subject, using the spatial-temporal decoding method described above. Higher scores indicate higher similarity between the patterns in the data and those in the model.

3.2. Colour decoding is driven by visual signals with a contralateral-posterior topography

To understand what contributed to our ability to decode colour, we next ran a searchlight decoding analysis (Kriegeskorte et al., 2006) across all sensors (van Ede, Chekroud, Stokes, and Nobre, 2019; Wolff et al., 2020). This demonstrated that classifier evidence (in the 150 to 350 ms post-stimulus period in which we found robust decoding; grey-shaded windows in Fig. 2C) peaked in posterior electrodes, with primary contributions from electrodes contralateral to the decoded stimuli. This was the case both for decoded colour (Fig. 3A), and orientation (Fig. 3B). This provides compelling evidence for the “visual” nature of this decoding (as opposed to, for example, decoding distinct “verbal labels” associated with distinct colours).

Open in a separate window

Fig. 3

Colour and orientation decoding are primarily driven by posterior electrodes contralateral to the decoded stimulus. Top row topographies show which electrodes show highest mean cosine-convolved evidence for A) colours and B) orientations, of stimuli that were presented on the left or on the right of fixation. Posterior contralateral electrodes show highest evidence. Bottom panels show the difference between left-stimulus and right-stimulus decoding topographies, highlighting the lateralisation of the evidence, and the comparable lateralisation obtained for colour and orientation decoding.

3.3. Colour decoding is parametrically organised

To test how well each of the feature-value bins could be decoded (relative to all other feature-value bins within the colour and orientation spaces), we analysed mean cosine-convolved LDA likelihoods in the data pooled between 150 - 350ms (indicated in grey in Fig. 2C). Decoding, as a function of the decoded feature-values, is shown in Fig. 4A. For colour bins, all 12/12 colours could be decoded significantly against a p < .05 threshold (mean t₂₉ = 4.523; min = 2.440; max = 6.175), and for orientation 11/12 orientations could be decoded significantly against an uncorrected single-tailed threshold (mean t₂₉ = 4.096; min = 1.585; max = 7.390). After applying Bonferroni correction, 9/12 colour bins were significant and 8/12 orientations. Using multi-dimensional scaling we revealed the representational space in two dimensions, showing a circular, or oval, configuration for both colour and orientation bins (Fig. 4B). For orientation, the cardinal orientations (0, 90, and 180°) showed higher decoding than the off-cardinal decoding. Interestingly, due to some non-linearities in the circular space, we observe some colours (pink, red, orange) in one part of the representational space and other colours (purple, blue, green) on the other (see also Hermann et al., 2020).

Open in a separate window

Fig. 4

Decoding of individual features between 150 and 350 ms. A) This polar plot illustrates the decoding evidence of 12 individual colours and orientations. The distance from the centre depicts the t-value of testing the cosine-convolved tuning curve for a single feature, left and right combined, across participants, relative to zero. The size of the dots mirrors the magnitude of the evidence for that feature B) Two-dimensional visualisation of the similarity matrix in which we observed a circular configuration for colour features. Dissimilarity matrices show feature (colour or orientation) on the y-axis and distance between bins of the tuning curve on the x-axis.

3.4. Decoding is sensitive to luminance (L*) differences but primarily driven by difference in colour (a*b*)

Colours generated for this experiment were drawn from a plane in CIELAB with identical (desired) lightness, and with equal distances between the colours in the circular colour landscape (Fig. 1A). Nevertheless, because of slight luminance and colour differences generated during the conversion to RGB colours and rendering process on the monitor, measured colour properties were slightly different. In the next step, we therefore used the physically observed measured luminance (L*) and measured colour (a*b*) to dissect their relative contributions to the identified colour decoding.

To this end, we created three “models” (Fig. 5A) and used RSA to ask how well each of them independently fit the empirical EEG data. We observed that measured differences in luminance (L*; Luminance model in Fig. 5A) between the 12 colour-bins could be decoded from the EEG signal yielding a significant cluster between 115 - 765 ms (Fig. 5B). Critically, however, in addition to the luminance model, we also compared the data to the dissimilarity matrix of the measured colour values (a*b*; Fig. 5A), as well as the (highly similar) CIELAB model used in the previous analyses. This revealed that the colour model and the CIELAB model each fit the observed data significantly better than the luminance model, with significant clusters between 155 - 855 ms and 165 - 735 ms, respectively (indicated by the increased line width in the horizontal cluster-significance lines in Fig. 5B). This is also appreciated by comparing the three models to the empirical dissimilarity matrices, at several time slices, in the bottom of Fig. 5B. These data confirm that the empirical dissimilarity matrix in the EEG data is more similar to the colour and CIELAB models, than to the model constructed from measured luminance values, and that this is so consistently across time. Thus, although unintended luminance differences between our stimuli may have made some contribution, the vast share of our colour decoding appears to be driven by colour(a*b*) differences between our stimuli.

Open in a separate window

Fig 5

Decoding measured luminance and a*b* differences between stimuli. A) Z-scored dissimilarity models representing the measured differences is luminance (L*), Euclidean distances between 12 pairs of measured a*b* values, and a CIELAB model that assumes equal dissimilarity between all neighbouring colours (and that was used in all other analyses). B) Similarity between dissimilarity models and the LDA evidence in the EEG data for each time point between all 12 colour bins. Going from left to right, the matrices below the x-axis show the average LDA evidence between −100 to 150 ms, 150 to 400 ms, 400 to 650 ms, and 650 to 900 ms post stimulus onset. Clusters of significant differences between a random model and measured a*b* (purple), measured luminance model (black) and CIELAB (green) are indicated with the coloured horizontal lines below the similarity time courses. Larger line width of the horizontal green and purple lines indicate cluster-corrected differences with luminance evidence. Error bars show 95% confidence intervals, calculated across participants (n = 30).

We would like to thank Sammi Chekroud for his help with the experimental work as well as Hannah Smithson and Allie Hexley for lending us their colorimeter equipment.

This research was funded by an ESRC Grand Union studentship and the Scatcherd European Scholarship awarded to J.E.H., a Marie Skłodowska-Curie Fellowship from the European Commission (ACCESS2WM) and an ERC Starting Grant from the European Research Council (MEMTICIPATION, 850636) to F.v.E., and was supported by a James S. McDonnell Foundation Scholar Award (220020405), an ESRC grant (ES/S015477/1), the Medical Research Council Career Development Award (MR/J009024/1), and a Biotechnology and Biological Sciences Research Council award (BB/M010732/1) to M.G.S., as well as a James S. McDonnell Foundation Understanding Human Cognition Collaborative Award (number 220020448), and a Wellcome Trust Senior Investigator Award (104571/Z/14/Z) to A.C.N., and was supported by the NIHR Oxford Health Biomedical Research Centre. The Wellcome Centre for Integrative Neuroimaging is supported by core funding from the Wellcome Trust (203139/Z/16/Z). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.