Mar 02 2022

Estimating the rate-distortion function of real-world data, part 2
Your (beta-)VAE might be secretly estimating the rate-distortion function of your data.

Mar 02 2022

Estimating the rate-distortion function of real-world data, part 1
What is the rate-distortion function, and why we may care about it.

Oct 28 2021

The Ill-defined Problem of Maximum Likelihood Estimation
Why we use maximum likelihood for density estimation, when it breaks down (especially on real-world data), and what can be done about it.

Mar 10 2020

All the Ways to Carve Up the ELBO
My list of fancy decompositions of the aggregate ELBO, focusing on the role of the aggregate KL regularizer, its relation to the aggregate posterior, and the mutual information between the data and the latent variable.

Dec 14 2018

Equivalent Definitions of the (Best) Lipschitz Constant
A quick note on the equivalent (dual?) definitions of the Lipschitz constant.

May 25 2018

Variational Autoencoder
Training a VAE on MNIST from scratch using Keras.

Aug 21 2017

Projections of Probability Distributions and the Reparameterization Trick
A brief review of M-projection, I-projection, REINFORCE, and the Reparameterization Trick.

Jul 21 2017

Restricted Boltzmann Machine and Contrastive Divergence
RBM training with CD from scratch on MNIST data.

Feb 07 2017

Linear Regression as Conditional Random Field
Classical linear regression simply uses a linear Gaussian model (CRF).

Feb 01 2017

Gibbs Sampling in Pairwise Markov Networks
Discrete pairwise Markov networks are pretty straightforward to work with analytically. Let's use the canonical overcomplete representation, where the sufficient statistics functions are indicator functions …

Dec 15 2016

Kernel => RKHS + Feature Map
Given a kernel function, find the Reproducing Kernel Hilbert Space and the feature map it defines.

Nov 30 2016

Separating Hyerplane Example
What it takes to separate an affine and a convex set.

Oct 14 2016

Relative Interior
A relative notion of set interior.

Sep 05 2016

Geometric Interpretations of First-Order Taylor Approximation
Geometric views from the output space, input space, and graph space.

Apr 01 2016

Polar Decomposition Example
A small numeric example illustrating polar decomposition.

Mar 28 2016

Eigendecomposition as Change of Basis
An operator has a particularly simple matrix description in its eigenbasis.

Mar 27 2016

Change of Basis
How to encode a linear operator in a different matrix.

Jan 19 2016

The Matrix of a Linear Map
Matrices are descriptions of linear maps.

Dec 21 2015

Example of a Polynomial Basis
Example basis of a function space.