Neural Discrete Representation Learning

ABSTRACT

Learning useful representations without supervision remains a key challenge in machine learning. We propose the Vector Quantised-Variational AutoEncoder (VQ-VAE), a simple yet powerful generative model that learns discrete representations of data. Unlike standard VAEs that use continuous latent spaces, VQ-VAE utilizes vector quantisation to overcome the 'posterior collapse' problem, where latents are ignored when paired with a powerful autoregressive decoder. We show that VQ-VAE can learn highly compressed, discrete latent representations of images, audio, and video, which can then be used by an autoregressive prior (like PixelCNN or WaveNet) to generate high-quality, coherent samples.