Understanding disentangling in $\beta$-VAE

Abstract

We present new intuitions and theoretical assessments of the emergence of\ndisentangled representation in variational autoencoders. Taking a\nrate-distortion theory perspective, we show the circumstances under which\nrepresentations aligned with the underlying generative factors of variation of\ndata emerge when optimising the modified ELBO bound in $\\beta$-VAE, as training\nprogresses. From these insights, we propose a modification to the training\nregime of $\\beta$-VAE, that progressively increases the information capacity of\nthe latent code during training. This modification facilitates the robust\nlearning of disentangled representations in $\\beta$-VAE, without the previous\ntrade-off in reconstruction accuracy.\n