Gray-box Inference for Structured Gaussian Process Models

Pietro Galliani, Amir Dezfouli, Edwin Bonilla, Novi Quadrianto
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:353-361, 2017.

Abstract

We develop an automated variational inference method for Bayesian structured prediction problems with Gaussian process (GP) priors and linear-chain likelihoods. Our approach does not need to know the details of the structured likelihood model and can scale up to a large number of observations. Furthermore, we show that the required expected likelihood term and its gradients in the variational objective (ELBO) can be estimated efficiently by using expectations over very low-dimensional Gaussian distributions. Optimization of the ELBO is fully parallelizable over sequences and amenable to stochastic optimization, which we use along with control variate techniques to make our framework useful in practice. Results on a set of natural language processing tasks show that our method can be as good as (and sometimes better than, in particular with respect to expected log-likelihood) hard-coded approaches including SVM-struct and CRF, and overcomes the scalability limitations of previous inference algorithms based on sampling. Overall, this is a fundamental step to developing automated inference methods for Bayesian structured prediction.

Cite this Paper

BibTeX
@InProceedings{pmlr-v54-galliani17a, title = {{Gray-box inference for structured Gaussian process models}}, author = {Galliani, Pietro and Dezfouli, Amir and Bonilla, Edwin and Quadrianto, Novi}, booktitle = {Proceedings of the 20th International Conference on Artificial Intelligence and Statistics}, pages = {353--361}, year = {2017}, editor = {Singh, Aarti and Zhu, Jerry}, volume = {54}, series = {Proceedings of Machine Learning Research}, month = {20--22 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v54/galliani17a/galliani17a.pdf}, url = {https://proceedings.mlr.press/v54/galliani17a.html}, abstract = {We develop an automated variational inference method for Bayesian structured prediction problems with Gaussian process (GP) priors and linear-chain likelihoods. Our approach does not need to know the details of the structured likelihood model and can scale up to a large number of observations. Furthermore, we show that the required expected likelihood term and its gradients in the variational objective (ELBO) can be estimated efficiently by using expectations over very low-dimensional Gaussian distributions. Optimization of the ELBO is fully parallelizable over sequences and amenable to stochastic optimization, which we use along with control variate techniques to make our framework useful in practice. Results on a set of natural language processing tasks show that our method can be as good as (and sometimes better than, in particular with respect to expected log-likelihood) hard-coded approaches including SVM-struct and CRF, and overcomes the scalability limitations of previous inference algorithms based on sampling. Overall, this is a fundamental step to developing automated inference methods for Bayesian structured prediction.} }
Endnote
%0 Conference Paper %T Gray-box Inference for Structured Gaussian Process Models %A Pietro Galliani %A Amir Dezfouli %A Edwin Bonilla %A Novi Quadrianto %B Proceedings of the 20th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2017 %E Aarti Singh %E Jerry Zhu %F pmlr-v54-galliani17a %I PMLR %P 353--361 %U https://proceedings.mlr.press/v54/galliani17a.html %V 54 %X We develop an automated variational inference method for Bayesian structured prediction problems with Gaussian process (GP) priors and linear-chain likelihoods. Our approach does not need to know the details of the structured likelihood model and can scale up to a large number of observations. Furthermore, we show that the required expected likelihood term and its gradients in the variational objective (ELBO) can be estimated efficiently by using expectations over very low-dimensional Gaussian distributions. Optimization of the ELBO is fully parallelizable over sequences and amenable to stochastic optimization, which we use along with control variate techniques to make our framework useful in practice. Results on a set of natural language processing tasks show that our method can be as good as (and sometimes better than, in particular with respect to expected log-likelihood) hard-coded approaches including SVM-struct and CRF, and overcomes the scalability limitations of previous inference algorithms based on sampling. Overall, this is a fundamental step to developing automated inference methods for Bayesian structured prediction.
APA
Galliani, P., Dezfouli, A., Bonilla, E. & Quadrianto, N.. (2017). Gray-box Inference for Structured Gaussian Process Models. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 54:353-361 Available from https://proceedings.mlr.press/v54/galliani17a.html.

Related Material