VI Tutorial at UA

Schedule

Check the branch UA2020 for all modules

Day 1 (21 January 9:30-12:30)

• Deep generative models

Day 2 (22 January 9:30-12:30)

• Variational inference
• Discrete latent variables

Day 3 (23 January 9:30-12:30)

• Continuous latent variables
• Variance reduction
• Relaxations to discrete latent variables

Labs

Day 1 We prepared a lab to get you started with probabilistic models parameterised by NNs. In this first exercise, we do not have latent variables, but we will stick closely to terminology and concepts presented in class. For example, you will noticed that we will be predicting distributions and that our loss will be derived from log likelihood of observations under such distributions (rather than the more familiar DL terminology involving cross entropies).

• CharLM

Day 2 We prepared a lab to introduce you to an application of the score function estimator in discrete latent variable models. This is a latent factor model used to generate wordforms.

• LatentFactorModel

Day 3 We prepared a lab to introduce you to reparameterised gradients. This is a variational auto-encoder for wordforms.

• WordVAE

Extra

We will use this space to post additional material based on questions we get in class.

Beyond

We can build expressive distributions by transforming draws from simpler ones with a continuously differentiable bijection, this leads to a class of models known as normalising flows. NFs can be used for better density estimation but also for better variational inference.

• Normalising flows

We can build transformations of variables into our generative models to enable reparameterisation of distributions that are not differentiably reparameterisable. This leads to ADVI:

• Automatic Differentiation VI