Video Prediction with Appearance and Motion Conditions

Yunseok Jang, Gunhee Kim, Yale Song
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:2225-2234, 2018.

Abstract

Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model behave when there are multiple correct, equally probable future? We propose an Appearance-Motion Conditional GAN to address this challenge. We provide appearance and motion information as conditions that specify how the future may look like, reducing the level of uncertainty. Our model consists of a generator, two discriminators taking charge of appearance and motion pathways, and a perceptual ranking module that encourages videos of similar conditions to look similar. To train our model, we develop a novel conditioning scheme that consists of different combinations of appearance and motion conditions. We evaluate our model using facial expression and human action datasets and report favorable results compared to existing methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-jang18a, title = {Video Prediction with Appearance and Motion Conditions}, author = {Jang, Yunseok and Kim, Gunhee and Song, Yale}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {2225--2234}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/jang18a/jang18a.pdf}, url = {https://proceedings.mlr.press/v80/jang18a.html}, abstract = {Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model behave when there are multiple correct, equally probable future? We propose an Appearance-Motion Conditional GAN to address this challenge. We provide appearance and motion information as conditions that specify how the future may look like, reducing the level of uncertainty. Our model consists of a generator, two discriminators taking charge of appearance and motion pathways, and a perceptual ranking module that encourages videos of similar conditions to look similar. To train our model, we develop a novel conditioning scheme that consists of different combinations of appearance and motion conditions. We evaluate our model using facial expression and human action datasets and report favorable results compared to existing methods.} }
Endnote
%0 Conference Paper %T Video Prediction with Appearance and Motion Conditions %A Yunseok Jang %A Gunhee Kim %A Yale Song %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-jang18a %I PMLR %P 2225--2234 %U https://proceedings.mlr.press/v80/jang18a.html %V 80 %X Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model behave when there are multiple correct, equally probable future? We propose an Appearance-Motion Conditional GAN to address this challenge. We provide appearance and motion information as conditions that specify how the future may look like, reducing the level of uncertainty. Our model consists of a generator, two discriminators taking charge of appearance and motion pathways, and a perceptual ranking module that encourages videos of similar conditions to look similar. To train our model, we develop a novel conditioning scheme that consists of different combinations of appearance and motion conditions. We evaluate our model using facial expression and human action datasets and report favorable results compared to existing methods.
APA
Jang, Y., Kim, G. & Song, Y.. (2018). Video Prediction with Appearance and Motion Conditions. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:2225-2234 Available from https://proceedings.mlr.press/v80/jang18a.html.

Related Material