Estimating the rate-distortion function of real-world data, part 2

Your (beta-)VAE might be secretly estimating the rate-distortion function of your data.

Estimating the rate-distortion function of real-world data, part 1

What is the rate-distortion function, and why we may care about it.

The Ill-defined Problem of Maximum Likelihood Estimation

Why we use maximum likelihood for density estimation, when it breaks down (especially on real-world data), and what can be done about it.

All the Ways to Carve Up the ELBO

My list of fancy decompositions of the aggregate ELBO, focusing on the role of the aggregate KL regularizer, its relation to the aggregate posterior, and the mutual information between the data and the latent variable.

Equivalent Definitions of the (Best) Lipschitz Constant

A quick note on the equivalent (dual?) definitions of the Lipschitz constant.

Variational Autoencoder

Training a VAE on MNIST from scratch using Keras.

Projections of Probability Distributions and the Reparameterization Trick

A brief review of M-projection, I-projection, REINFORCE, and the Reparameterization Trick.

Restricted Boltzmann Machine and Contrastive Divergence

RBM training with CD from scratch on MNIST data.

Linear Regression as Conditional Random Field

Classical linear regression simply uses a linear Gaussian model (CRF).

Gibbs Sampling in Pairwise Markov Networks

Discrete pairwise Markov networks are pretty straightforward to work with analytically. Let's use the canonical overcomplete representation, where the sufficient statistics functions are indicator functions …

Kernel => RKHS + Feature Map

Given a kernel function, find the Reproducing Kernel Hilbert Space and the feature map it defines.

Separating Hyerplane Example

What it takes to separate an affine and a convex set.

Relative Interior

A relative notion of set interior.

Geometric Interpretations of First-Order Taylor Approximation

Geometric views from the output space, input space, and graph space.

Polar Decomposition Example

A small numeric example illustrating polar decomposition.

Eigendecomposition as Change of Basis

An operator has a particularly simple matrix description in its eigenbasis.

Change of Basis

How to encode a linear operator in a different matrix.

The Matrix of a Linear Map

Matrices are descriptions of linear maps.

Example of a Polynomial Basis

Example basis of a function space.