Model components affecting the score of a model.

Given the range of MSE_n scores we observed, we next wished to perform a secondary analysis on how each of the two components defining a model (its encoding stage and plasticity rule) affected its error score on the benchmark above. One general trend we observed was that models built with encoding stages from deeper layers of DCNNs tended to produce more human-like learning behavior (see Fig 5B). On the other hand, the choice of plasticity rule (which defined by the tunable decision stage) appeared to have little effect in a model’s ability to generate human-like learning behavior (see Fig 5A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Evaluating the effect of model design choices on predictive accuracy of human learning.

A. Example model scores across encoding stages and plasticity rules. A typical example of the relative contributions of plasticity rule (y-axis) and encoding stage (x-axis) on model scores (MSE_n, encoded by color). B. Overview of all models tested. In total, we tested encoding stages drawn from a total of n = 19 DCNN architectures, varying widely in depth. A model’s similarity to humans was highly affected by the choice of encoding stage; those based on deeper layers of DCNNs showed the most human-like learning behavior. On the other hand, the choice of plasticity rule had a minuscule effect. C. Predictive accuracy increases as a function of relative network depth. Learning models with encoding stages based on DCNN layers closer to the final layer of the architecture tended to be better.

https://doi.org/10.1371/journal.pcbi.1011713.g005

We quantified these observations by performing a two-way ANOVA over all model scores (see Section 2.1 in S1 Appendix), treating the plasticity rule and encoding stage as the two factors. This analysis showed that the choice of plasticity rules explained less than 0.1% of the variation in model scores; by contrast, 99.8% of the variation was driven by the encoding stage, showing that the predominant factor affecting the alignment of the learning model with humans was the encoding stage.

Put another way, we found that different plasticity rules (out of n = 7 rules we tested), when combined with any given encoding stage, led to learning models with nearly identical levels of human similarity (i.e. highly similar MSE_n benchmark scores). Still, this does not necessarily imply that all plasticity rules generated similar predictions: intuitively, each learning rule could be equally wrong in different ways (see Fig 6A, left).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Further testing for potential differences between plasticity rules.

A. Possible scenarios, given similar benchmark scores. In the “behavioral measurement space” of this experiment, the MSE_n metric is an estimate of the (squared) distance between a human and a model. There is a spectrum of possible scenarios consistent with the observation that a set of learning models have similar human MSE_n scores. B. Rules generate highly similar predictions across all tested encoding stages. Across a range of representational models (random subset shown, drawn from 20 equally-sized bins across the observed MSE_n range), the rule-to-rule distances are near zero, showing different rules consistently lead to highly similar predictions. The y-axis (MSE_u) is a symmetric version of the MSE_n metric that performs noise correction for both arguments (see Section 2.2 in S1 Appendix).

https://doi.org/10.1371/journal.pcbi.1011713.g006

To disambiguate between these possibilities, we directly estimated the average pairwise MSE values between learning models that shared the same encoding stage (differing only in their plasticity rules). We found that across a range of encoding stages (Fig 6B), different plasticity rules not only led to models with similar benchmark scores, but also highly similar predictions of behavior, relative to their average similarity to humans (median relative dissimilarity 0.04%, max: 8.26%; see Section 2.2 in S1 Appendix). In a supplementary analysis, we also found the high level of between-rule similarity extended beyond the image-averaged behavioral measures we consider above, with different rules converging to largely similar weight solutions and image-by-image patterns of behavior (see S3 Fig). In sum, we found that the seven plasticity rules are all equally correct (equivalently, all equally incorrect), and they are virtually identical to each other along key functional measures.

Strong baseline models are largely, but not perfectly, correlated with human performance patterns.

The set of MSE_n scores we observed indicated that all models tested had significant differences from humans, based on their respective learning curves. There are several ways in which these differences could originate—for example, a model might have “fast” learning curves for tasks that humans learn slowly, and “slow” learning curves for tasks that humans learn rapidly. Alternatively, a model might have the same pattern of difficulty across subtasks as humans, but simply be slower at learning, overall.

To gain insight into these possibilities, we performed an additional analysis in which we compared each model to humans along two more granular statistics: 1) its overall accuracy over all subtasks and trials tested, and 2) its consistency with the patterns of difficulty exhibited by humans across subtasks (see Fig 7A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Comparing models to humans along more granular behavioral signatures of learning.

A. Decomposing model behavior into two metrics. We examined model behavior along two specific aspects of learning behavior: overall accuracy (top right), which is the average accuracy of the model over the entire experiment (i.e. averaging across all 100 trials and all 64 subtasks), and consistency (bottom right), which conveys how well a model’s pattern of trial-averaged performance over different subtasks rank-correlates with that of humans (Spearman’s rank correlation coefficient). B. Consistency and overall accuracy for all models. Strong baseline models (top-right) matched (or exceeded) humans in terms of overall accuracy, and had similar (but not identical) patterns of performance with humans (consistency). The gray regions are the bootstrap-estimated 95% range of each statistic between independent repetitions of the behavioral experiment. The color map encodes the overall score (MSE_n) of each model (colorbar in Fig 5A).

https://doi.org/10.1371/journal.pcbi.1011713.g007

Intuitively, overall accuracy is a gross measure of a learning system’s overall ability to learn, ranging from 0.5 (chance, no learning occurs) to 0.995 (learning completed after just one trial). Consistency (ρ) quantifies the extent to which a model finds the same subtasks easy and hard as humans (per the trial-averaged accuracies for each subtask). It ranges from ρ = −1 (perfectly anticorrelated pattern of performance) to ρ = 1 (perfectly correlated pattern of performance). A value of ρ = 0 indicates no correlation between the patterns of difficulty across subtasks in a model and humans.

These metrics are theoretically unrelated to each other; given any overall accuracy, a model may have a high or low consistency with humans, and vice versa (except for the special cases where a model achieves 100% or 0% accuracy after a single trial for all subtasks; then consistency is undefined). Nevertheless, we observed that these two metrics covaried for these models (Pearson r = 0.73); models with high overall accuracy also tended to have high consistency (see Fig 7B). In S5 Fig, we also show that MSE_n and consistency scores for these models are highly coupled.

Humans learn new objects faster than all tested models in low-sample regimes.

Though many models matched or exceeded human-level overall accuracy (i.e. accuracy averaged over trials 1–100 for all subtasks), we noticed that all models’ accuracy early on in learning consistently appeared to be below that of humans (see Fig 8A). We tested for this by comparing the accuracy of models and humans in an initial phase of learning (trials 1–5 for all subtasks), and indeed found that all of the models were significantly worse than humans in the early phase of learning (all p<0.05, bootstrap hypothesis test, see Fig 8B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Humans outperform all strong baseline models in low-sample regimes.

A. Subtask-averaged learning curves for humans and strong baseline models. The y-axis is the percent chance that the subject made the correct object report (chance is 50%). The x-axis is the total number of image examples shown prior to the test trial (log scale). The average learning curves for humans (black) and models (magenta) show that humans outperform all models in low-sample learning regimes. Errorbars on the learning curves are the bootstrapped SEM; model errorbars are not visible. B. No model achieves human-level early accuracy. We tested all models for whether they could match humans early on in learning. Several models (including all strong baseline models) were capable of matching or exceeding late accuracy in humans (average accuracy over trials 95–100), but no model reached human-level accuracy in the early regime (average over trials 1–5). This trend was present in subtasks across different levels of difficulty (bottom row). The gray region shows the 95% bootstrapped CI for each statistic; 95% CIs for models are too small to be shown.

https://doi.org/10.1371/journal.pcbi.1011713.g008

We wondered whether this gap was present across levels of difficulty (e.g., that models tended to perform particularly poorly on “hard” subtasks relative to humans, but were human-level for other subtasks), and repeated this analysis across four different difficulty levels of subtasks (where each level consisted of 16 out of the 64 total subtasks we tested, grouped by human difficulty levels). We found models were consistently slower than humans across the difficulty range, though we could not reject a subset (11/20) of the strong baseline models at the easiest and hardest levels (see Fig 8B).

Lastly, though all models failed to match humans in the early regime, many models readily matched or exceeded human performance late in learning (i.e. the average accuracy on trials 95–100 of the experiment).

Linear learning on fixed image representations are strong baseline models of human object learning.

On our first benchmark, which compares a learning model to humans under high view-variation learning conditions, we found a subset of models produced relatively accurate predictions of human learning behavior. The observed alignment of these models with humans does not originate from the fact they successfully learn new objects—these models also fail to rapidly learn the same objects that humans find difficult (Fig 7B), suggesting they have nontrivial similarities with humans, at least behaviorally.

We were surprised by the extent of similarity we observed between these models and humans, partly because some have suggested that DCNNs are unlikely to support adequate descriptions of human learning (e.g. [15, 39–42]), and partly because of the simplicity of these models. The results reported here suggest that learning models based on contemporary models of high-level visual neural representations and rudimentary, one-layer plasticity rules are a strong starting point to quantitatively account for the ability (and inability) of humans to learn arbitrary, new objects.

We note the present work does not directly engage the ongoing issue of whether the optimization mechanism of backpropagation is somehow involved in human learning [43]. Though the DCNN-based representations used in the learning models in this work were originally created using backpropagation, they were kept completely “frozen” over behavioral learning, with all behavioral learning achieved through a single layer of weight changes (which does not require backpropagation). At a conceptual level, this work regards DCNN-based representations as estimates of the (adult) human subject’s internal neural representation at the beginning of task learning, and is agnostic to how they were created before that point. In any case, this work suggests that invoking end-to-end plasticity computational mechanisms (such as backpropagation) may not be necessary to provide a quantitative account of human learning behaviors, at least on the timescales we considered.

In general, the models we considered are composed only of operations that closely hew to those executed by first-order models of neurons—namely, linear summation of upstream population activity, ramping nonlinearities, and adjustment of local associational strengths using reward signals. This makes them not only plausible descriptions for the computations executed by the brain over object learning, but, with some additional assumptions, they make predictions of neural phenomena.

For example, the schematic shown in Fig 2B assumes that the image representations used for object learning are neural population firing rates in IT cortex, a late stage of the ventral stream (an assumption we find support for in a supplementary analysis, see S4 Fig). If this is the case, these models predict that ventral stream representations used by humans over object learning need not undergo plastic changes to mediate behavioral improvements over the duration of the experiments we conducted (seconds-to-minutes timescale). This prediction is in line with prior studies showing adult ventral stream changes are typically moderate and take place on longer timescales (see [44] for review).

Moreover, the entirety of the computational learning mechanisms used by these models (i.e. the learning of thresholded, linear combinations of upstream neural activity using reward signals) can plausibly be executed at a single visuomotor synaptic interface where reward-based feedback signals are available. Several regions downstream of the ventral visual stream are possible candidates for this locus of plasticity during invariant object learning; we point to striatal regions receiving both high-level visual inputs and midbrain dopaminergic signals and involved in premotor processing, such as the caudate nucleus, as one set of candidates [45, 46].

We considered learning models based on multiple variants of simple, reward-optimizing plasticity rules which guide changes at the locus of plasticity (Fig 2D). In principle, each rule variant offers distinct predictions for the learning signal (e.g. neuromodulatory signal) present on any given trial. However, in this work, we found these rules were largely interchangeable with respect to the primary behavioral metric we considered (Fig 2), and along more granular metrics (Figs 6 and S3). Thus, we do not consider this work to strongly drive the belief that one should expect one kind of learning signal over another amongst the set we tested, beyond what is common to all rules.

Gaps between models and humans in few-shot learning.

Despite the predictive strength of some models we tested, all models tested were unable to fully explain all replicable human behavior on either behavioral benchmark. One consistent prediction failure we observed in all models was an inability to learn new objects as rapidly as humans in low-sample regimes. We found this to be the case in both Experiment 1 (see Fig 8) and in Experiment 2, where we found that all tested models had lower accuracy than humans after one-shot across a variety of generalization tests (see Fig 9C). For example, we found that these models cannot one-shot generalize as well to scale shifts as humans, replicating previous work [47].

Taken together, these observations show all tested learning models currently have quantitative deficiencies from humans in the few-shot regime. We note that even if our experiments have underestimated human learning speed (e.g. from increased inattention rates on Mechanical Turk [48]), this inference would not change; the estimated gaps in few-shot learning abilities between these models and humans would be larger than the ones we report here. However, other aspects of similarity we found between models and humans—such as their shared patterns of relative difficulty—would be robust to such biases in our experiment.

Extensions of task paradigm.

The two benchmarks we developed here certainly do not encompass all aspects of object learning. For example, each benchmark focused on discrimination learning between two novel objects, but humans can potentially learn and report on many more objects simultaneously. Moreover, humans can readily learn object categories at different levels of abstraction, each of which may encompass multiple specific objects [51]. The models tested here scale naturally to task paradigms involving additional objects (via the incorporation of new linear choice preferences to the decision stage), and are capable of learning categories of varying abstraction; comparing them to humans in those richer learning settings (and identifying any of their limits in those settings) could strongly motivate the consideration of more complex models.

Extending stimulus presentation time.

For presenting stimuli, we followed conventions used in previous visual neuroscience studies [18, 52] of object perception: achromatic images containing single objects rendered with high view uncertainty on random backgrounds, presented at <10 degrees of visual field and for <200 milliseconds.

The chosen stimulus presentation time of 200 milliseconds is too short for a subject to initiate a saccadic eye movement based on the content of the image [53]. Such a choice simplifies the input of any model (i.e., to a single image, rather than the series of images induced by saccades); on the other hand, active viewing of an image via target-directed saccades might be a central mechanism deployed by humans to mediate learning of new objects.

We note that if this is the case, our task paradigm (which would disrupt any such saccade-based mechanisms from being used) would be overestimating the number of images needed by humans to achieve learning on new objects, compared to a scenario in which subjects had unlimited viewing time on each trial. Thus, removing such a bias in our experimental design could potentially reveal larger differences between models and humans.

Beyond extending viewing time, designing tasks which more closely hew to typical object learning contexts for humans (e.g. involving colored images and/or movies of potentially multiple objects physically embedded in natural scenes) will be an important direction for future work.

Differences between individual subjects.

We primarily focused on studying human learning at the subject-averaged level, where behavioral measurements are averaged across several individuals (i.e. subject-averaged learning curves; see Fig 1D). However, individual humans may have systematic differences in their learning behavior that are (by design) ignored with this approach.

For example, we found that individual subjects may differ in their overall learning abilities: we identified a subpopulation of humans who were significantly more proficient at learning compared to other humans (see Fig 10B). We did not attempt to model this individual variability in this study; whether these differences can be explained by alterations to this model family, if at all, (e.g. through the introduction of random effects to the parameters of the encoding stage and/or plasticity rules) remains an area for future study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Individual differences in learning ability.

A. Individual-level learning curves. We identified 22 subjects who performed all 64 subtasks in Experiment 1, and computed their subtask-averaged learning curves. Each gray curve corresponds to the learning curve for a different individual subject (smoothed using state-space estimation from [38]). In humans, a range of overall learning performance is seen: some subjects consistently outperformed others (e.g. Subject M, highest accuracy over all trials and subtasks), while others consistently underperformed (e.g. Subject L, lowest average accuracy). In magenta are subtask-averaged learning curves corresponding to individual model simulations from the highest-performing model tested in this study (encoding stage=ResNet152/avgpool, plasticity rule=square). B. Some individual humans outperform all models. Five out of 22 subjects had significantly higher overall performance than the highest performing model we tested (one-tailed Welch’s t-test, Bonferroni corrected, p<0.05).

https://doi.org/10.1371/journal.pcbi.1011713.g010

Moreover, performing subject-averaging is known to lead to the masking of learning dynamics present at the level of single subjects, such as delayed rises or step increases in accuracy [54]. Performing analyses to compare any such learning dynamics between individual humans and learning models is another important extension of our work.

Lastly, we did not attempt to model any systematic increases in a subject’s learning performance as they performed more and more subtasks available to them (in either Experiment 1 or 2). This phenomenon (learning-to-learn, learning sets, or meta-learning) is well-known in psychology [55], but to our knowledge has not been systematically measured or modeled in the domain of human object learning. Expanding these benchmarks (and models) to measure and account for such effects is an important future step of building models of the work done here.

Test image presentation.

Each trial began with a display start screen that was uniformly gray except for a small black dot at the center of the screen, which indicated the future center of each test image; note that the center of the test image is not necessarily the center of the object. We intended for this fixation point to encourage the subject to consistently view each test image at the center of their field of view (see S2 Fig for in-lab eye measurements). The subject then initiated the trial by pressing the space bar on their keyboard. Once pressed, a test image (occupying ≈ 6° of the visual field) belonging to one of the two possible object categories immediately appeared. That test image remained on the screen for ≈ 200 milliseconds before disappearing (and returning the screen to uniform gray). Note that our report of the visual degrees of angle of the stimulus is only an estimate: we assumed our subjects used computer monitors with a 16:9 aspect ratio, and naturally positioned themselves so the horizontal extent of the monitor occupied between 40°-70° degrees of their visual field. Under those assumptions, the visual angle of the stimulus would vary between a minimum and maximum of ≈ 4° − 8°. Given a monitor has a 60 Hz refresh rate, we expect the actual test image duration to vary between ≈ 183 − 217 milliseconds.

For each subject and each trial, the test image was selected by first randomly picking (with equal probability) one of the two objects as the generator of the test image. Then, given that selected object, an image of that object was randomly selected from a pool of pre-rendered possible images. Test images were always selected without replacement (i.e. once selected, that test image was removed from the pool of possible future test images for that behavioral session).

Subject choice reporting.

Fifty milliseconds after the disappearance of the test image, the display cued the subject to report the object that was “in” the image. The display showed two identical white circles—one on the lower left side of the fixation point and the other on the lower right side of the fixation point. The subject was previously instructed to select either the “F” or “J” keys on their keyboard. We randomly selected one of the two possible object-to-key mappings prior to the start of each session, and held it fixed throughout the entire session. This mapping was not told to the subject; thus, on the first trial, subjects were (by design) at chance accuracy.

To achieve perfect performance, a subject would need to associate each test image of an object to its corresponding action choice, and not to the other choice (i.e., achieving a true positive rate of 1 and a false positive rate of 0).

Subjects had up to 10 seconds to make their choice. If they failed to make a selection within that time, the task returned to the trial initiation phase (above) and the outcome of the trial was regarded as being equivalent to the selection of the incorrect choice. In practice, this was quite rare and corresponded to ≈ 0.04% of all trials that are included in the results in this work.

Trial feedback.

As subjects received feedback which informed them whether their choice was correct or incorrect (i.e. corresponding to the object that was present in the preceding image or not), they could in principle learn object-choice associations that enabled them to make correct choices on future trials.

Trial feedback was provided immediately after the subject’s choice was made. If they made the correct choice, the display changed to a feedback screen that displayed a reward cue (a green checkmark). If they made an error, a black “x” was displayed instead. Reward cues remained on the screen for 50 milliseconds, and were accompanied by an increment to their monetary reward (see above). Error cues remained on the screen for 500 milliseconds. Following either feedback screen, a 50 millisecond delay occurred, consisting of a uniform gray background. Finally, the display returned to the start screen, and the subject was free to initiate the next trial.

High-variation stimulus image generation.

We designed 3D object models (n = 128) using the “Mutator” generative design process [57]. We generated a collection of images for each of those 3D objects using the POV-Ray rendering program [58]. To generate each image, we randomly selected the viewing parameters of the object, including its projected size on the image plane (25%-50% of total image size, uniformly sampled), its location (±40% translation from image center for both x and y planes, uniformly sampled), and its pose relative to the camera (uniformly sampled random 3D rotations). We then superimposed this view on top of a random, naturalistic background drawn from a database used in a previously reported study [52]. All images used in this experiment were grayscale, and generated at a resolution of 256x256 pixels.

We show an example of 32 objects (out of 128 total) in Fig 1A, along with example stimulus images for two of those objects on the right. Taken together, the multiple axes of variation for the images in this experiment—i.e. random viewing parameters and backgrounds—were intended to reproduce the kinds of variation which may be present in “real world” object learning contexts.

Design of subtasks.

We randomly allocated the 128 novel objects described above into pairs (without replacement) to create n = 64 subtasks for Experiment 1, where each subtask consists of a distinct pair of novel objects. Each behavioral session for a subtask consisted of 100 trials, regardless of the subject’s performance. On each trial of a session, one of the two objects was randomly selected, and then a test image of that object was drawn randomly without replacement from a pre-rendered set of 100 images of that object (generated using the process above). That test image was then presented to the subject (as described in Test image presentation). We collected 50 sessions per subtask and all sessions for each subtask were obtained from separate human subjects, each of whom we believe had not seen images of either of the subtask’s objects before participation.

Subject recruitment and data collection.

Human subjects were recruited on the Mechanical Turk platform [56] through a two-step screening process. The goal of the first step was to verify that our task software successfully ran on their personal computer, and to ensure our subject population understood the instructions. To do this, subjects were asked to perform a prescreening subtask with two common objects (elephant vs. bear) using 100 trials of the behavioral task paradigm (described in Task paradigm above). If the subject failed to complete this task with an average overall accuracy of at least 85%, we intentionally excluded them from all subsequent experiments in this study.

The goal of the second step was to allow subjects to further familiarize themselves with the task paradigm. To do this, we asked subjects to complete a series of four “warmup” subtasks, each involving two novel objects (generated using the same “Mutator” software, but distinct from the 128 described above). Subjects who completed all four of these warmup subtasks, regardless of accuracy, were enrolled in Experiment 1. Data from these warmup subtasks were not included in any analysis presented in this study. In total, we recruited n = 70 individual Mechanical Turk workers for Experiment 1.

Once a subject was recruited, they were allowed to perform as many of the 64 subtasks as they wanted, though they were not allowed to perform the same subtask more than once (median n = 61 total subtasks completed, min = 1, max = 64). We aimed to measure 50 sessions per subtask (i.e. 50 unique subjects), where each subject’s session consisted of an independently sampled, random sequence of trials. Each of these subtasks followed the same task paradigm (see Task paradigm), and each session lasted 100 trials. Thus, the total amount of data we aimed to collect was 64 subtasks × 100 trials × 50 subjects = 320k measurements.

Behavioral statistics in humans.

We aimed to estimate a typical subject’s accuracy at each trial, conditioned on a specific subtask. We therefore computed 64 × 100 accuracy estimates (subtask × trial) by taking the sample mean across subjects. We refer to this [64, 100] matrix of point statistics as . Each row vector has 100 entries, and corresponds to the mean human “learning curve” for subtask s = {1, 2, …64}.

Because each object was equally likely (over sessions) to be shown on any given test trial, each of these 100 values of may be interpreted as an estimate of the average of the true positive and true negative rates (i.e. the balanced accuracy). The balanced accuracy is related to the concept of sensitivity from signal detection theory—the ability for a subject to discriminate two categories of signals [59]. We note that an independent feature of signal detection behavior is the bias—the prior probability with which the subject would report a category. We did not attempt to quantify or compare the bias in models and humans in this study.

Simulating behavioral sessions in computational models.

To obtain the learning curve predictions of each computational model, we required that each model perform the same set of subtasks that the humans performed, as described above.

We imposed the same requirements on the model as we did on the human subjects: that it begins each session without knowledge of the correct object-choice contingency, that it should generate a action choice based solely on a pixel image input, and that it can update its future choices based on the history of scalar-valued feedback (“correct” or “incorrect”). If the choices later in the session are more accurate than those earlier in the session, then we colloquially say that the model has “learned”, and comparing and contrasting the learning curves of models with those of humans was a key goal of Experiment 1.

We ran n = 32,000 simulated behavioral sessions for each model (500 simulated sessions for each of the 64 subtasks), where on each simulation a random sequence of trials was sampled in an identical fashion as in humans (see above). During each simulation, we recorded the same raw “behavioral” data as in humans (i.e. sequences of correct and incorrect choices), then applied the same procedure we used to compute (see above) to compute an analogous collection of point statistics on the model’s raw behavior, which we refer to as .

In summary: each model followed the same experimental procedure that any given human subject followed. The primary difference was that we permitted each model to perform any given subtask multiple times (here 500 times), to improve our statistical estimates of the model’s expected behavior (unlike an individual human participant, a model may have its internal state reset to “forget” the subtask it just learned).

Comparing model learning with human learning.

The learning behavior generated by the “correct” model of human learning () should minimally replicate the empirical measurements of learning behavior in humans () to the limits of statistical noise. To identify any such models, we developed a scoring procedure to compare the similarity of the learning behavior in humans with any candidate learning model. We describe this procedure below.

Bias-corrected mean squared error.

Given a collection of human measurements (here, a matrix of accuracy estimates for S = 64 subtasks over T = 100 trials) and corresponding model measurements (), we computed a standard goodness-of-fit metric, the mean-squared error (MSE; lower is better). The formula for the MSE is given by: (1)

Because and are random variables (i.e. sample means), the MSE itself is a random variable. It can be seen that the expected value of consists of two conceptual components: 1) the difference between the expected behavior of model and humans, and 2) noise components: (2) Where denotes the expected value, and σ²(⋅) denotes the variance due to finite sampling (a.k.a. “noise”).

Eq (2) shows that the expected MSE for a model depends not only on its expected predictions , but also its sampling variance . In the present case where is the mean over independent (but not necessarily identically distributed) Bernoulli variables, the value of happens to depend on the expected prediction of the model itself, . This can be seen by the expression for the variance of , which is a mean over independent Bernoulli variables: , where is the expected behavior of the model on trial t of subtask s, and n_st is the number of model simulations.

Because the sampling variance of the model depends on its predictions, it is therefore conceptually possible that a model with worse (expected) predictions could achieve a lower expected MSE, simply because its associated sampling variance term is lower (in addition to the dependence of on p_st, it also depends on the number of model simulations n_st—though in this study, all models performed the same number of simulations).

We corrected for this inferential bias by estimating, then subtracting, these variance terms from the “raw” MSE for each model we tested. We refer to this bias-corrected error as MSE_n: (3) Where is an unbiased estimator of the variance of . In practice, because of the high number of simulations that were conducted, the size of the summed noise components tended to be small relative to the size of the summed expected difference components (median relative size, over models: 0.5%, max: 3.8%). We write the equation for below, where k_st is the number of observed correct choices over the n_st model simulations conducted for subtask s and trial t: (4)

Because , the expected value of MSE_n can be shown to be: (5)

Intuitively, MSE_n is an estimate of the mean-squared error that would be achieved by a model if we had a noiseless estimate of its predictions (i.e. had an infinite number of simulations of that model been performed). We note that the value of its square root, , gives a rough (i.e. statistically biased) estimate of the average deviation between a model’s prediction and human measurements, in units of the measurements (in this study, units of accuracy).

There are potential alternatives to using the MSE_n metric to compare the behavior of a model with humans. For example, approaches based on correlational metrics have been previously used (e.g. [18, 52]). We note that correlational metrics are insensitive to arbitrary linear rescaling and offsets. Thus, a model with behavior that is distinct from humans may potentially achieve a high (perfect) correlation with humans. Because the goal in the present work is to gauge the extent to which a model can exactly reproduce the learning behaviors we measured in humans, we were motivated to develop and use the MSE_n metric described above.

Noise floor estimation.

It can be seen in Eq (5) that there are terms , originating from the uncertainty in our experimental estimates of human behavior. These terms are always positive, and create a lower bound on the expected MSE_n for all models. That is, even if a model is expected to perfectly match the subject-averaged behavior of humans (where , for all subtasks s and trials t), it cannot be expected to achieve an error below this lower bound. We call this lower bound the “noise floor”, and use the symbol to refer to it: (6)

It is possible to make an unbiased estimate of the noise floor () if one can make unbiased estimates of each term. We did so by using the unbiased estimator from Eq (4). We write the full expression for our estimate of the noise floor, , below: (7) Where k_st is the number of human subjects (out of n_st total subjects) that made a correct choice on subtask s and trial t. The square root of this value, σ_h, gives a rough estimate of the average deviation one would expect in our subject-averaged measurements of behavior () over repetitions of the experiment (i.e. upon another resampling of human subjects and behavioral sessions).

Null hypothesis testing.

For each model we tested, we attempted to reject the null hypothesis that , for all subtasks s and trials t. To do so, we first approximated the distribution for that would be expected under this null hypothesis, using bootstrapping.

To do so, we first computed bootstrap replicates of and approximated samples of the null model (where ). A bootstrap replicate of was constructed by first resampling individual human sessions without replacement, taking the same number of resamples per subtask as in the original experiment. We then computed the replicate using the same procedure described in Behavioral statistics in humans. Behavior from the null model cannot be sampled directly (because we do not have the “true model” of human learning), but by definition shares the same expected behavior as a randomly sampled, individual human. We therefore created a bootstrap sample of the null model by (also) taking resamples of individual human sessions, setting the number of resamples per subtask to the number of model simulations conducted per subtask (here, n = 500 simulations per subtask). We then computed and saved for that iteration, and repeated this process for B = 1,000 iterations to obtain an approximate null distribution for MSE_n.

If a model’s actual score fell above the α-quantile of the estimated null distribution, we rejected it on the basis of having significantly more error than what would be expected from a “true” model of humans (with estimated significance level α).

Lapse rate correction.

Lastly, we corrected for any lapse rates present in the human data. We defined the lapse rate as the probability with which a subject would randomly guess on a trial, and we assumed this rate was constant across all trials and subtasks. To correct for any such lapse rate in the human data, we fit a simulated lapse rate γ parameter to each model, prior to computing its MSE_n. Given a lapse rate parameter of γ (ranging between 0 and 1), a model would, on each trial, guess randomly with probability γ. For each model, we identified the value of γ that minimized its empirical MSE_n.

We note that fitting γ can only drive the behavior of a model toward randomness; it cannot artificially introduce improvements in its learning performance.

One-shot behavioral testing.

As in Experiment 1, we used two-way object discrimination with trial-and-error feedback as our task paradigm (described in greater detail in Task paradigm). We created 64 object models for this experiment (randomly paired without replacement to give a total of 32 subtasks). These objects were different from the ones used in the previous benchmark (described in Experiment 1: Learning objects under high view variation).

At the beginning of each session, we randomly assigned the subject to perform one of the 32 subtasks. Identical to Experiment 1, each trial required the subject to view an image of an object, make a choice (“F” or “J”), and receive feedback based on their choice. Each session consisted of 20 trials total, which was split into a “training phase” (10 trials) and “testing phase” (10 trials), which we describe below.

Training phase.

During the first ten trials of a session, the subject learned a novel object-choice contingency. Unlike Experiment 1, trials in the first 10 trials were based on only 2 distinct images (a single image for each object). These two images were therefore repeated across ten trials (here, we ensured that each training image was repeated an equal number of times, i.e. n = 5 times each). Trials were presented in a random order.

Testing phase.

Following the one-shot training phase, the subject was tested on randomly sampled, unseen images over the final ten trials of the session (feedback continued to be delivered throughout). We describe these test images in detail in One-shot stimulus image generation; see Fig 1B for examples. On the 15th and 20th trial, we presented “catch trials” consisting of the original training images, to allow us to estimate inattention rates.

One-shot stimulus image generation.

Here, we describe how we generated all of the images used in Experiment 2. First, we generated each 3D object model using the Mutator process (see High-variation stimulus image generation)). Then, for each object (n = 64 objects), we generated a single canonical training image—a 256x256 grayscale image of the object occupying ≈ 50% of the image plane, centered on a gray background. We randomly sampled its three axes of pose from the uniform rotational distribution.

For each training image, we generated a corresponding set of test images by applying different kinds of image transformations we wished to measure human generalization on. In total, we generated test images based on 9 transformation types, and we applied each transformation type at 4 levels of “strength”. We describe those 9 types with respect to a single training image, below.

Translation.

We translated the object in the image plane of the training image. To do so, we randomly sampled a translation vector in the image plane (uniformly sampling an angle from θ ∈ [0°, 360°]), and translated it r pixels in that direction. We repeated this process (independently sampling θ each time) for r = 16, 32, 64, and 96 pixels (where the total image size 256 × 256 pixels), for two iterations (for a total of eight translated images).

Backgrounds.

We gradually replaced the original, uniform gray background with a randomly selected, naturalistic background. Each original background pixel b_ij in the training image was gradually replaced with a naturalistic image c using the formula . We varied α at four logarithmically spaced intervals, α = 0.1, 0.21, 0.46, 1. Note that at α = 1, the original gray background is completely replaced by the new, naturalistic background. We generated two test images per α level, independently sampling the background on each iteration (for a total of eight images per object).

Scale.

We rescaled the object’s size on the image to 12.5%, 25%, 50%, and 150% of the original size (four images of the object at different scales).

Out-of-plane rotations.

We rotated the object along equally spaced 45° increments, rendering a test image at each increment. We did so along two separate rotational axes (horizontal and vertical), leading to n = 13 test images total based on out-of-plane rotations.

In-plane rotation.

We rotated the object inside of the image-plane, along 45° increments. This resulted in n = 7 test images based on in-plane rotations.

Contrast.

We varied the contrast of the image. For each pixel p_ij (where pixels range in value of 0 and 1), we adjusted the contrast using the equation , varying c from −0.8, −0.4, 0.4 and 0.8.

Pixel deletion.

We removed pixels corresponding to the object in the training image, replacing them with the background color (gray). We removed 25%, 50%, 75%, and 95% of the pixels, selecting the pixels randomly for each training image.

Blur.

We blurred the training image using a Gaussian kernel. We applied blurring with kernel radii of 2, 4, 8, and 16 pixels (with an original image resolution of 256 × 256 pixels) to create a total of 4 blurred images.

Gaussian noise.

We applied Gaussian noise to the pixels of the training image. For each pixel p_ij, we added i.i.d. Gaussian noise: (8)

We applied noise with σ = 0.125, 0.25, 0.375 and 0.5 (where pixels range in luminance value between 0 and 1). We then clipped the resultant pixels to lie between 0 and 1.

Human behavioral measurements for Experiment 2.

Subject recruitment.

We used the same two-step subject recruitment procedure described above (see Subject recruitment and data collection), and recruited n = 170 human subjects. Some of these subjects overlapped with those in Experiment 1 (n = 9 subjects participated in both experiments).

All recruited subjects were invited to participate in up to 32 behavioral sessions. We disallowed them from repeating subtasks they had performed previously. Subjects were required to perform a minimum of four such behavioral sessions. In total, we collected n = 2,547 sessions (≈ 51,000 trials) for Experiment 2.

One-shot behavioral statistics in humans.

We aimed to estimate the expected accuracy of a subject on each of the 36 possible transformations, correcting for attentional and memory lapses.

To do so, we first combined observations across the eight test trials in the testing phase to compute accuracy estimate for each of the 36 transformations; that is, we did not attempt to quantify how accuracy varied across the testing phase (unlike the previous benchmark). We also combined observations across the 32 subtasks in this experiment. In doing so, we were attempting to measure the average generalization ability for each type of transformation (at a specific magnitude of transformation change from the training image), ignoring the fact that generalization performance likely depends on both the objects to be discriminated (i.e. the appearance of the objects in each subtask), the specific training images that were used, and the testing views of each object (e.g. the specific way in which an object was rotated likely affects generalization—not just the absolute magnitude of rotation). In total, we computed 36 point statistics (one per transformation).

Estimating performance relative to catch performance.

Here we assumed that each human test performance measurement was based on a combination of the subject’s ability to successfully generalize, a uniform guessing rate (i.e. the probability with which a subject executes a 50–50 random choice), and the extent to which the subject successfully acquired and recalled the training image-choice contingency (i.e. from the first 10 trials). We attempted to estimate the test performance of a human subject that could 1) fully recall the association between each training image and its correct choice during the training phase, and 2) had a guess rate of zero on the test trials.

To do so, we used trials 15 and 20 of each session, where one of the two training images was presented to the subject (“catch trials”). Our main assumption here was that performance on these trials would be 100% assuming the subject had perfect recall, and had a guess rate of zero. Under that assumption, the actual, empirically observed accuracy p_catch would be related to any overall guess and/or recall failure rate γ by the equation γ = 2 − 2p_catch. We then adjusted each of the point statistics (i.e. test performances) to estimate their values had γ been equal to zero, by applying the following formula:

We refer to the collection of 36 point statistics (following lapse rate correction) as .

Comparing model one-shot learning with human one-shot learning.

Model simulation of Experiment 2.

For this benchmark, we required that a model perform a total of 16,000 simulated behavioral sessions (500 simulated sessions for each of the 32 possible subtasks). Each simulated session proceeded using the same task paradigm as in humans (i.e. 10 training trials, followed by a test phase containing 8 test trials and 2 catch trials). Based on the model’s behavior over those simulations, we computed the same set of point statistics described above, though we did not correct for any attentional lapses or recall lapses in the model, which we assumed was absent in models. In this manner, for each model, we obtained a collection of point statistics reflecting their behavior on this experiment, .

Bias-corrected mean-squared error and null hypothesis testing.

We used the same statistical approach for our primary benchmark (introduced in Comparing model learning with human learning) to summarize the alignment of a model with humans. That is, we used the bias-corrected error metric MSE_n as our metric of comparison:

We estimated the null distribution for MSE_n using bootstrap resampling, following the same procedure outlined in the first benchmark (bootstrap-resampling individual sessions).

Encoding stages.

The encoding stages were intermediate layers of deep convolutional neural network architectures (DCNNs). We drew a selection of such layers from a pool of 19 network architectures available through the PyTorch library [60], each of which had pretrained parameters for solving the Imagenet object classification task [37].

For each architecture, we selected a subset of these intermediate layers to test in this study, spanning the range from early on in the architecture to the final output layer (originally designed for Imagenet). We resized pixel images to a standard size of 224x224 pixels using bilinear interpolation. In total, we tested n = 344 intermediate layers as encoding stages.

Dimensionality reduction.

Once an input image is fed into a DCNN architecture, each of its layers produces a representational vector of a dimensionality specified by the architecture of the model. Depending on the layer, this dimensionality may be relatively large (>10⁵), making it hard to efficiently perform numerical calculations on contemporary hardware. We therefore performed dimensionality reduction as a preprocessing step using random Gaussian projections to a standard size of 2048 (if the original dimensionality of the layer was greater than this number). This procedure approximately preserves the original representational structure of the layer (i.e., pairwise distances between points in that space) [61] and is similar to to computing and retaining the first 2048 principal components of the representation.

Feature normalization.

Once dimensionality reduction was performed, we performed another standardization step: We computed centering and scaling parameters for each layer, so that its activations fit inside a sphere of radius 1 centered about the origin (i.e. , for all ).

To do so, we computed the activations of the layer over using the images from the “warmup” tasks human subjects were exposed to prior to performing any task in this study (i.e. 50 randomly selected images of 8 objects, see Subject recruitment and data collection). We computed the sample mean of those activations, and set this as the new origin of the encoding stage (i.e. the centering parameter). Then, we took the 99th quantile of the activation norms (over those same images) to calculate the approximate radius of the representation, and set this as our scaling parameter (i.e. dividing all activations by this number). Any activations with a norm greater than this radius were scaled to have a norm of 1.

Other kinds of feature standardization schemes are possible: for instance, one could center and scale the sensory representations for each subtask separately. However, such a procedure would expose models to the statistics of subtasks that are meant to be independent tests of their ability to learn new objects—statistics which we considered to be predictions of the encoding stage.

Tunable decision stage.

Once the encoding stage re-represents an incoming pixel image as a multidimensional vector , a tunable decision stage takes that vector as an input, and produces a choice as an output.

Generating a decision.

To select a choice, the tunable decision stage first generates choice preferences for each of the C possible actions, using the dot products for i = 1…C. Then, the choice with the highest preference is selected (). If all choices have the same preference value, a choice is randomly selected.

Learning from feedback.

Once a choice is selected, the environment may convey some scalar-valued feedback (e.g. reward or punish signals). The model may use this feedback to change its future behavior (i.e., to learn). For all models considered here, this may be accomplished (only) by changing its weight vectors . Thus, learning on each trial can be summarized by the changes to each weight vector by some :

There are many possible choices on how each may be computed from feedback; here, we focused on a set of seven rules based on the stochastic gradient descent algorithm for training a binary classifier or regression function. In all cases except one, the underlying strategy of each decision stage can be understood as predicting the reward following the selection of each possible choice, and using these predictions to select the choice it believes will lead to the highest reward (the exception is the REINFORCE plasticity rule, which is a “policy gradient” rule that optimizes parameters directly against the rate of reward; it does not aim to predict reward itself).

Specifically, we tested the plasticity rules induced by the gradient descent update on the perceptron, cross-entropy, exponential, square, hinge, and mean absolute error loss functions (shown in Fig 2D), as well as the REINFORCE plasticity rule. They are summarized in S1 Table. These rules happen to induce weight updates of the particularly simple form , where is the current image representation, and β_c is a scalar (which is computed by the plasticity rule). In other words, all plasticity rules drive weight updates which consist of a (positively or negatively scaled) addition of the current image representation.

Each of these plasticity rules has a single free parameter—a learning rate. For each plasticity rule, there is a theoretical range of learning rates that guarantees the non-divergence of the decision stage, based on the smoothness or Lipschitz constant of each of the plasticity rule’s associated loss function [62]. We did not investigate different learning rates in this study; instead, we simply selected the highest learning rate possible (such that divergence would not occur) for each plasticity rule.