Tuning-free coreset Markov chain Monte Carlo
Naitong Chen Jonathan H. Huggins Trevor Campbell
Department of Statistics University of British Columbia [email protected] Department of Mathematics & Statistics Boston University [email protected] Department of Statistics University of British Columbia [email protected]
Abstract
A Bayesian coreset is a small, weighted subset of a data set that replaces the full data during inference to reduce computational cost. The state-of-the-art coreset construction algorithm, Coreset Markov chain Monte Carlo (Coreset MCMC), uses draws from an adaptive Markov chain targeting the coreset posterior to train the coreset weights via stochastic gradient optimization. However, the quality of the constructed coreset, and thus the quality of its posterior approximation, is sensitive to the stochastic optimization learning rate. In this work, we propose a learning-rate-free stochastic gradient optimization procedure, Hot-start Distance over Gradient (Hot DoG), for training coreset weights in Coreset MCMC without user tuning effort. Empirical results demonstrate that Hot DoG provides higher quality posterior approximations than other learning-rate-free stochastic gradient methods, and performs competitively to optimally-tuned ADAM.
1 INTRODUCTION
Bayesian inference provides a flexible framework for parameter estimation and uncertainty quantification in statistical models. Markov chain Monte Carlo (Robert and Casella, 2004; Robert and Casella, 2011; Gelman et al., 2013, Chs. 11 and 12), the standard methodology for performing Bayesian inference, involves simulating carefully constructed Markov chains whose stationary distribution is the target Bayesian posterior. In the large-scale data setting, this procedure can become prohibitively expensive, as it requires iterating over the entire data set to simulate the next state.
Bayesian coresets (Huggins et al., 2016) are a popular approach for speeding up Bayesian inference in the large-scale data setting. A Bayesian coreset is a weighted subset of data that replaces the full data set during inference, leveraging the insight that large datasets often exhibit a significant degree of redundancy. With a carefully constructed coreset, one can significantly reduce the computational cost of inference while still obtaining samples from a high quality approximation of the full Bayesian posterior. In fact, given a data set of points, a coreset of size is sufficient for providing a near-exact posterior approximation in exponential family and other sufficiently simple models (Naik et al., 2022, Thms. 4.1 and 4.2; Chen et al., 2022, Prop. 3.1) and is sufficient for more general cases (Campbell, 2024, Cor. 6.1).
Constructing a coreset involves picking the data points to include in the coreset and assigning each data point its corresponding weight. The state-of-the-art method, Coreset MCMC (Chen and Campbell, 2024), selects coreset points by sampling them uniformly from the full data set, and learns the weights using stochastic gradient optimization techniques, e.g., ADAM (Kingma and Ba, 2014), where the gradients are estimated using MCMC draws targeting the current coreset posterior. However, as we demonstrate in this paper, there are two issues with this approach. First, the quality of the constructed coreset is sensitive to the learning rate of the stochastic optimization algorithm. And second, gradient estimates using MCMC draws are affected strongly in early iterations by initialization bias, leading to poor optimization performance.
To address these challenges, we first propose Hot-start Distance over Gradient (Hot DoG), a tuning-free stochastic gradient optimization procedure that can be used for learning coreset weights in Coreset MCMC. Hot DoG is a stochastic gradient method combining techniques from Do(W)G (Ivgi et al., 2023; Khaled et al., 2023), ADAM (Kingma and Ba, 2014), and RMSProp (Hinton et al., 2012) to set learning rates automatically. Hot DoG also includes an automated warm-up phase prior to weight optimization, which guards against usage of low quality MCMC draws when estimating the objective function gradients. Fig. 1 demonstrates that Hot DoG performs competitively to optimally-tuned ADAM across a wide range of models, datasets, and coreset sizes, and can be multiple orders of magnitude more accurate than ADAM using other learning rates. Beyond the result in Fig. 1, we provide an extensive empirical investigation of the reliability of Hot DoG in comparison to other methods across many different synthetic and real experiments.
2 BACKGROUND
2.1 Bayesian Coresets
We are given a data set of observations, a log-likelihood for observation given , and a prior density . We would like to sample from the Bayesian posterior with density
(2) |
where is the unknown normalizing constant. A Bayesian coreset replaces the sum over log-likelihood terms with a weighted sum over a subset of size , where . Without loss of generality, we assume that these are the first points. The coreset posterior can then be written as
(3) |
where is a vector of coreset weights. Recent coreset construction methods uniformly select points to include in the coreset (Naik et al., 2022; Chen et al., 2022; Chen and Campbell, 2024), and then optimize the weights of those points as a variational inference problem (Campbell and Beronov, 2019),
(4) |
with objective function gradient
(5) | ||||
(6) |
2.2 Coreset MCMC
The key challenge in solving Eq. 4 is that does not admit tractable i.i.d. draws, and so unbiased estimates of the gradient in Eq. 5 are not readily available. Coreset MCMC (Chen and Campbell, 2024) is an adaptive algorithm that addresses this issue. The method first initializes weights and samples . At iteration , given coreset weights and samples , it then updates the weights using the stochastic gradient estimate based on the draws ,
(7) | ||||
(8) |
where is a uniform subsample of indices of size , and . To complete the iteration, the method updates the samples by independently drawing for each , where is a family of Markov kernels that have invariant distribution . The pseudocode for Coreset MCMC is outlined in Algorithm 1.
3 TUNING-FREE CORESET MCMC
A key design choice when using Coreset MCMC is to specify how gradient estimates are used to optimize the weights. One can use ADAM (Kingma and Ba, 2014), which is used as the default optimizer for Coreset MCMC (Chen and Campbell, 2024): at iteration , with being the user-specified learning rate, we set
(9) |
where and are exponential averages of past gradients and their element-wise squares, and is a small constant. There are a wide range of other first-order stochastic methods available that could be used (e.g., vanilla stochastic gradient descent, AdaGrad (Duchi et al., 2011), etc.). However, like ADAM, most of these algorithms require setting a learning rate . And as we show in Fig. 2, the quality of samples obtained from Coreset MCMC can be highly sensitive to the selected learning rate. In particular, Fig. 2 shows that when using ADAM, no single learning rate applies well across all problems and coreset sizes; and for a given problem, the performance can vary by orders of magnitude as one varies the learning rate. Furthermore, the default ADAM learning rate of (Kingma and Ba, 2014) provides poor results in most of the problems tested. As a result, careful tuning of the learning rate is required to obtain high quality posterior approximations. This usually involves a search on a log-scaled grid, which is computationally wasteful as the results for all but one of the parameter values are thrown out. Moreover, in practice determining which learning rate provides the best posterior approximation may not be straightforward.
A number of recent works in the literature propose learning-rate-free stochastic optimization methods to address this issue (Carmon and Hinder, 2022; Ivgi et al., 2023; Khaled et al., 2023; Defazio and Mishchenko, 2023; Mishchenko and Defazio, 2024). Many of these methods are shown empirically to work competitively compared to optimally-tuned SGD on a wide range of large-scale, non-convex deep learning problems. Although different at first glance, all of these methods arise from the same insight. Suppose one would like to solve the stochastic optimization problem
(10) |
where for all , is convex and we only have access to unbiased stochastic gradient . Define the initial optimality gap and the sum of all gradient norms . By setting the SGD learning rate
(11) |
the average iterate satisfies the optimal error bound
(12) |
after iterations (Carmon and Hinder, 2022; Orabona and Cutkosky, 2020). Learning-rate-free methods therefore essentially try to estimate or bound the initial optimality gap , which is unknown in practice. To the best of our knowledge, there are four state-of-the-art methods that do this in a manner that does not require multiple optimization runs, knowledge of unknown constants, or the ability to query the objective function: DoG (Ivgi et al., 2023), DoWG (Khaled et al., 2023), D-Adaptation (Defazio and Mishchenko, 2023) and prodigy (Mishchenko and Defazio, 2024). DoG and DoWG run vanilla stochastic gradient descent (SGD),
(13) |
with learning rate schedules
(14) |
where is set to some small constant and, for ,
(15) |
For D-Adaptation and prodigy, in Eq. 14 is replaced with a lower bound on , which is updated using estimated correlations between the gradient and step direction :
(16) |
D-Adaptation replaces in Eq. 14 (DoG) with , while prodigy replaces in Eq. 14 (DoWG) with . Both D-Adaptation have SGD and ADAM-based variants. All four methods have been shown empirically to match the performance of optimally-tuned SGD.
Fig. 3 shows the results from direct applications of DoG, DoWG, D-Adaptation (SGD), and prodigy (ADAM) to Coreset MCMC. We see that the quality of posterior approximation from all of four methods are orders of magnitude worse than optimally-tuned ADAM. With initialized far away from high density regions of , the initial gradient estimates are large in magnitude, which leads to small learning rates. The accumulation of these large gradient norms in the learning rate denominator eventually causes the learning rate to vanish, halting the progress of coreset weight optimization. We address these problems in the next section.
Before concluding this section, we note that there are other approaches for making SGD free of learning rate tuning: some methods involve using stochastic versions of line search (Vaswani et al., 2019; Paquette and Scheinberg, 2020), and others do the same for the Polyak step size (Loizou et al., 2021). These methods are not applicable in our setting as they require evaluating the objective function. Recall that due to the unknown term in Eq. 3, we do not have access to estimates of the objective function.
4 HOT DOG
In this section, we develop our novel Markovian optimization method, Hot-start DoG (Hot DoG), presented in Algorithm 2. Our method extends the original DoG optimizer in two ways: (1) we add a tuning-free hot-start test that automatically detects when the Markov chains have properly mixed and stochastic gradient estimates are stable, at which point we start coreset weight optimization; and (2) we apply acceleration techniques to DoG.
4.1 Hot-start test
Poorly initialized Markov chain states can be detrimental to the performance of learning-rate-free methods in Coreset MCMC. Fig. 5, and especially Figs. 5(c), 5(d) and 5(e) show that this is likely due to the bias of initial gradient estimates. In particular, the initial gradient estimates often have a norm orders of magnitude larger than they would if they had been computed using i.i.d. draws, resulting in a quickly vanishing learning rate in Eq. 14. Therefore, it is crucial to hot-start the Markov chains to ensure they are properly mixed before training the coreset weights. There are MCMC convergence diagnostics that could be used for this purpose (e.g, (Vehtari et al., 2021)); many work only with real-valued variables, and are overly stringent for our application. We require a test that works for general coreset posteriors of the form Eq. 3 and checks only that gradient estimates have stabilized reasonably.
To address this challenge, we propose keeping the weights fixed at their initialization (i.e., ) until a hot-start test passes. For the test, for each Markov chain , we split the iterates into 3 segments, each of equal length . We compute the average log-potentials for the two latter segments , , and the standard deviations of residual errors from a linear fit:
(17) | ||||
(18) | ||||
(19) |
where is the log-potential for chain at iteration ,
(20) |
Our test monitors the difference between and relative to and . A small difference between and indicates that the Markov chain states have stabilized. The residual standard errors from the linear fits allows us to remove trends from the noise computation. We define, for each ,
(21) |
and use the median of as our test statistic. This test statistic is checked against a threshold ; the test passes when the median test statistic becomes smaller than . The pseudocode for the hot-start test is given in Algorithm 3. From our experiments, we find that setting works well in general.
4.2 Acceleration
To accelerate DoG, we begin by noting that the denominator of the DoG learning rate in Eq. 14 is similar to that of AdaGrad (Duchi et al., 2011) in that it is a cumulative sum of some function of the gradient. Therefore, we can leverage the idea used in RMSProp (Hinton et al., 2012) for accelerating AdaGrad to accelerate DoG. In particular, at iteration , we can replace with , the bias-corrected exponential moving average of the squared gradient. This allows us to exponentially decrease the weights of past gradient norms. As a result, the effect of the early terms on the learning rate diminishes over time, resulting in less conservative learning rates. To account for situations where the gradient estimates differ in scale across dimensions, we apply the above acceleration technique in a coordinate-wise fashion and obtain the following update rule for :
(22) |
where , , and denotes the vector with each entry of squared. We further apply the same idea to , the maximum distance traveled from , and , the gradient estimate itself, thereby arriving at our proposed optimization procedure. Note that in Algorithm 2, the operator denotes coordinate-wise maximum and the operater denotes coordinate-wise multiplication.
5 EXPERIMENTS
In this section, we demonstrate the effectiveness of Hot DoG and compare our method against other learning-rate-free stochastic gradient methods: optimally-tuned ADAM from a log scale grid search, as well as prodigy ADAM (Mishchenko and Defazio, 2024), DoG (Ivgi et al., 2023), and DoWG (Khaled et al., 2023) over different initial parameters. We compare the quality of posterior the approximations over different coreset sizes and weight optimization procedures for each experiment. Following Chen and Campbell (2024), for all optimization methods, we set the number of Markov chains to and subsample size to in Eq. 7. We set the Markov kernel to the hit-and-run slice sampler with doubling (Bélisle et al., 1993; Neal, 2003) for all real data experiments. For the Gaussian location model, we use a kernel that directly samples from (Chen and Campbell, 2024, Sec. 3.4); for the sparse regression example, we use Gibbs sampling (George and McCulloch, 1993).
We compare these algorithms using six different Bayesian models (two synthetic, four real): a synthetic Gaussian location, a synthetic sparse regression, a (non-conjugate) linear regression, a logistic regression, a Poisson regression, and a Bradley-Terry model. See Section A.1 for details. We use Stan (Carpenter et al., 2017) to obtain full data inference results for real data experiments, and Gibbs sampling (George and McCulloch, 1993) for the sparse regression model with discrete variables. For all experiments, we measure the posterior approximation quality using the average squared z-score, defined as , where and are, respectively, the coordinate-wise mean and standard deviation estimated using the full data posterior, and is the coordinate-wise mean estimated using draws from Coreset MCMC. This estimate is computed in a streaming fashion using the second half of all draws simulated at the time; note that this includes draws from before the hot-start test passes.
Each algorithm was run on 8 single-threaded cores of a 2.1GHz Intel Xeon Gold 6130 processor with 32GB memory. Code for these experiments is available at https://github.com/NaitongChen/automated-coreset-mcmc-experiments. More experimental details and additional plots are in Sections A.1 and A.2.
Effect of hot-start test. Fig. 4 compares Hot DoG with and without the hot-start test for across all experiments; the same plots for other coreset sizes can be found in Section A.2. Without the hot-start test, the traces often hit a long plateau, before the effect of exponentially-weighted averaging is able to decay early large gradient norms. On the other hand, with burn-in, we begin by simulating from Markov chains targeting , and start optimizing the coreset weights only after the hot-start test has passed. In terms of the number of log potential evaluations, Hot DoG with burn-in leaves the plateau sooner than without burn-in.
Fig. 5 examines the behaviour of the hot-start test in more detail, showing the traces of the gradient estimate norms and test statistics median across optimization iterations when using Hot DoG. Note that here we only show plots for ; the same plots for other coreset sizes can be found in Section A.2. In some experiments, the Markov chains are initialized reasonably well where the gradient norms are already stabilized, and the test passes almost immediately. In others, the Markov chains are initialized poorly and the gradient norms are large, but nevertheless, the hot-start test passes shortly after they stabilize. Across all of these experiments, a test statistic threshold of 0.5 worked well.
Robustness to fixed parameter . Figure 6 provides an examination of the robustness of the proposed method to the fixed initial learning rate parameter . Across all experiments, different values of spanning multiple orders of magnitude result in similar posterior approximations across optimization iterations. Note that is for all plots in Fig. 6. The same trends can be observed over different coreset sizes (see Section A.2).
Comparison with other related methods. Figure 7 shows a comparison between our method and DoG, DoWG, ADAM, as well as prodigy ADAM. We fix and for Hot DoG. Since the hot-start test itself can be applied to all methods, Hot DoG is compared against others both with and without burn-in. The posterior approximation quality of Hot DoG is orders of magnitude better than all other methods in many settings tested, and remain competitive otherwise. In particular, Hot DoG is capable of matching the performance of optimally-tuned ADAM without tuning.
6 CONCLUSION
This paper introduced Hot DoG, a learning-rate-free stochastic gradient method designed for learning coreset weights within the framework of Coreset MCMC. Our method extends DoG, but includes adjustments tailored to the Markovian setting of Coreset MCMC. In particular, Hot DoG includes a hot-start test for detecting when to start training coreset weights, as well as acceleration techniques. The quality of constructed coresets by Hot DoG and their corresponding posterior approximation is robust to all of its input parameters. Empirically, Hot DoG produces better posterior approximations than other learning-rate-free stochastic gradient methods, and is competitive to those of optimally-tuned ADAM.
Acknowledgements
T.C. was supported by NSERC Discovery Grant RGPIN-2019-03962, and J.H.H. was partially supported by National Science Foundation CAREER award IIS-2340586. We acknowledge the use of the ARC Sockeye computing platform from the University of British Columbia.
References
- Robert and Casella (2004) Christian Robert and George Casella. Monte Carlo Statistical Methods. Springer, edition, 2004.
- Robert and Casella (2011) Christian Robert and George Casella. A short history of Markov chain Monte Carlo: subjective recollections from incomplete data. Statistical Science, 26(1):102–115, 2011.
- Gelman et al. (2013) Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin. Bayesian Data Analysis. CRC Press, edition, 2013.
- Huggins et al. (2016) Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable Bayesian logistic regression. In Advances in Neural Information Processing Systems, 2016.
- Naik et al. (2022) Cian Naik, Judith Rousseau, and Trevor Campbell. Fast Bayesian coresets via subsampling and quasi-Newton refinement. In Advances in Neural Information Processing Systems, 2022.
- Chen et al. (2022) Naitong Chen, Zuheng Xu, and Trevor Campbell. Bayesian inference via sparse Hamiltonian flows. In Advances in Neural Information Processing Systems, 2022.
- Campbell (2024) Trevor Campbell. General bounds on the quality of Bayesian coresets. In Advances in Neural Information Processing Systems, 2024.
- Chen and Campbell (2024) Naitong Chen and Trevor Campbell. Coreset Markov chain Monte Carlo. In International Conference on Artificial Intelligence and Statistics, 2024.
- Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In International Conference on Learning Representations, 2014.
- Ivgi et al. (2023) Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is SGD’s best friend: a parameter-free dynamic step size schedule. In International Conference on Machine Learning, 2023.
- Khaled et al. (2023) Ahmed Khaled, Konstantin Mishchenko, and Chi Jin. DoWG unleashed: an efficient universal parameter-free gradient descent method. In Advances in Neural Information Processing Systems, 2023.
- Hinton et al. (2012) Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a: overview of mini-batch gradient descent, 2012.
- Campbell and Beronov (2019) Trevor Campbell and Boyan Beronov. Sparse variational inference: Bayesian coresets from scratch. In Advances in Neural Information Processing Systems, 2019.
- Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011.
- Carmon and Hinder (2022) Yair Carmon and Oliver Hinder. Making SGD parameter-free. In Conference on Learning Theory, 2022.
- Defazio and Mishchenko (2023) Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by D-adaptation. In International Conference on Machine Learning, 2023.
- Mishchenko and Defazio (2024) Konstantin Mishchenko and Aaron Defazio. Prodigy: an expeditiously adaptive parameter-free learner. In International Conference on Machine Learning, 2024.
- Orabona and Cutkosky (2020) Francesco Orabona and Ashok Cutkosky. International Conference on Machine Learning tutorial on parameter-free stochastic optimization, 2020.
- Vaswani et al. (2019) Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: interpolation, line-search, and convergence rates. In Advances in Neural Information Processing Systems, 2019.
- Paquette and Scheinberg (2020) Courtney Paquette and Katya Scheinberg. A stochastic line search method with expected complexity analysis. Society for Industrial and Applied Mathematics Journal on Optimization, 30(1):349–376, 2020.
- Loizou et al. (2021) Nicolas Loizou, Sharan Vaswani, Issam Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for SGD: an adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, 2021.
- Vehtari et al. (2021) Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank-normalization, folding, and localization: an improved for assessing convergence of MCMC (with discussion). Bayesian Analysis, 16(2):667–718, 2021.
- Bélisle et al. (1993) Claude Bélisle, Edwin Romeijn, and Robert Smith. Hit-and-run algorithms for generating multivariate distributions. Mathematics of Operations Research, 18(2):255–266, 1993.
- Neal (2003) Radford Neal. Slice sampling. The Annals of Statistics, 31(3):705–767, 2003.
- George and McCulloch (1993) Edward George and Robert McCulloch. Variable selection via Gibbs sampling. Journal of the American Statistical Association, 88(423):881–889, 1993.
- Carpenter et al. (2017) Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: a probabilistic programming language. Journal of Statistical Software, 76(1):1––32, 2017.
- Elo (1978) Aprad Elo. The Rating of Chessplayers, Past and Present. Arco, edition, 1978.
Appendix A APPENDIX
A.1 Details of Experiments
A.1.1 Model Specification
In this subsection, we describe the six examples (two synthetic and four real data) that we used for our experiments. Processed versions of all datasets used for the experiments are available at https://github.com/NaitongChen/automated-coreset-mcmc-experiments. For each of the regression models, we are given a set of points , each consisting of features and response .
Bayesian sparse linear regression: This is based on Example 4.1 from George and McCulloch (1993). We use the model
(23) | ||||
(24) | ||||
(25) | ||||
(26) |
where we set , and . Here we model the variance , the vector of regression coefficients and the vector of binary variables indicating the inclusion of the feature in the model. We set , , , and generate a synthetic dataset by
(27) | ||||
(28) | ||||
(29) |
Bayesian linear regression: We use the model
(30) | ||||
(31) |
where is a vector of regression coefficients and is the noise variance. Note that the prior here is not conjugate for the likelihood. The dataset consists of flight delay information from observations and was constructed using flight delay data from https://www.transtats.bts.gov/Homepage.asp and historical weather information from https://www.wunderground.com/. We study the difference, in minutes, between the scheduled and actual departure times against features including flight-specific and meteorological information.
Bayesian logistic regression: We use the model
(32) | ||||
(33) |
where is a vector of regression coefficients. Here we use the same dataset as in linear regression, but instead model the relationship between whether a flight is cancelled using the same set of features. Note that of all flights included, only were cancelled.
Bayesian Poisson regression: We use the model
(34) | ||||
(35) |
where is a vector of regression coefficients. The dataset consists of observations, and we model the hourly count of rental bikes against features (e.g., temperature, humidity at the time, and whether or not the day is a workday). The original bike share dataset is available at https://archive.ics.uci.edu/dataset/275/bike+sharing+dataset.
The remaining two non-regression models are specified as follows.
Gaussian location: We use the model
(36) | ||||
(37) |
where . Here we model the mean . We set and generate a synthetic dataset by
(38) |
Bradley-Terry model: We use the model
(39) | ||||
(40) |
where . The dataset was constructed using games statistics from https://www.nba.com/stats and consists of data of NBA games between the 2004 and 2022 seasons. and are the home team and visitor team IDs for the game in the dataset, and denotes the outcome of the game ( if the home team won and if the visitor team won). represents the Elo ratings or relative skill levels (Elo, 1978, Ch. 1) for each of the teams. We model the Elo ratings using outcomes of pairwise comparisons between teams using game outcomes.
A.1.2 Parameter Settings
For full-data inference results of all examples except for the sparse linear regression model, we ran Stan (Carpenter et al., 2017) with 10 parallel chains, each taking steps with the first discarded, for a combined draws. For full-data inference result of the sparse linear regression example, we use the Gibbs sampler developed by George and McCulloch (1993) to generate draws, with the first half discarded as burn-in.
To account for changes in , for all real data experiments, we use the hit-and-run slice sampler with doubling (Bélisle et al., 1993; Neal, 2003); for the Gaussian location model, we use a kernel that directly samples from (Chen and Campbell, 2024, Sec. 3.4). for the sparse regression, we use the Gibbs sampler developed by George and McCulloch (1993).
We use Stan (Carpenter et al., 2017) to obtain full data inference results for real data experiments, and Gibbs sampling (George and McCulloch, 1993) for the sparse regression model with discrete variables. The true posterior distribution for the Gaussian location model is available in closed form.
For ADAM, we test multiple learning rates over a log scale grid for . For each experiment under each coreset size, the optimally-tuned ADAM is the one that obtained the lowest average squared z-score after iterations of weight optimization. For all learning-rate-free methods, we test different initial parameters (initial lower bound for prodigy ADAM and for Hot DoG, DoG, and DoWG) over a log scaled grid for .
For the logistic regression example, to account for the class imbalance problem, we include all observations from the rare positive class if the coreset size is more than twice as big as the total number of observations with positive labels. Otherwise we sample our coreset to have positive labels and negative labels. Coreset points are uniformly subsampled for all other models.
A.2 Additional Results
Figs. 4 to 6 in the main text show the traces of average squared coordinate-wise z-scores, as well as the gradient estimate norms and hot-start test statistics for Hot DoG when . In this subsection, we show the same sets of plots for and . Similarly to Fig. 4, Figs. 8 and 9 compare Hot DoG with and without hot-start test. Similarly to Fig. 5, Figs. 10 and 11 show the gradient estimate norms and hot-start test statistics during burn-in. Similarly to Fig. 6, Figs. 12 and 13 compare Hot DoG (with hot-start test) and optimally-tuned ADAM. We see that all plots show the same trends as the ones in Section 5, where . As a result, we arrive at similar observations as in Section 5. In particular, Hot DoG with burn-in leaves the plateau sooner than without burn-in; the hot-start test passes and thus burn-in terminates shortly after gradient norms are stabilized; Hot DoG is robust to the fixed parameter .