3. Variational Lower Bound (VLB)

The variational lower bound (VLB), also called the evidence lower bound (ELBO), is a tractable lower bound on a latent-variable model's log evidence. Use it when exact posterior inference or marginal-likelihood evaluation is intractable.

Reading time

2 min

Length

421 words

Updated

Sep 15, 2026

Total views

--

#math/ELBO#math/variational-inference#math/VLB

1. Variational Lower Bound (VLB) / Evidence Lower Bound (ELBO)

VLB and ELBO are two names for the same objective in variational inference. “Variational lower bound” emphasizes that the bound is constructed with a variational distribution; “evidence lower bound” emphasizes that it lower-bounds the log evidence (log marginal likelihood). They should not be treated as distinct losses.

Note : for the concept of the posterior probability, see^[1]

(1) Setup

Let $x$ be an observed datum, $z$ a latent variable, $p_{\theta }(z)$ a prior, and $p_{\theta }(x\mid z)$ a likelihood. The evidence (marginal likelihood) is :

$$\begin{array}{}\text{(1.1.1)}& p_{\theta }(x)=\int p_{\theta }(x,z)dz=\int p_{\theta }(x\mid z)p_{\theta }(z)dz.\end{array}$$

Bayes’ rule gives the exact posterior $p_{\theta }(z\mid x)$ from (2.2)^[1:1]. However, the evidence integral above is often intractable. Variational inference therefore chooses a tractable distribution $q_{\varphi }(z\mid x)$ to approximate that posterior :

$$\begin{array}{}\text{(1.1.2)}& q_{\varphi }(z\mid x)\approx p_{\theta }(z\mid x).\end{array}$$

Here, $\theta$ parameterizes the generative model and $\varphi$ parameterizes the variational distribution. The KL divergence used below is defined in (1.3)^[2].

The problem is about the posterior $p_{\theta }(z|x)$, and the variance lower bound is a lower bound estimation of posterior :

$$\log p_{\theta }(z|x)$$

(2) Derivation of the common objective

Start from the KL divergence^[2:1] between the approximate and exact posteriors :

$$\begin{array}{}\text{(1.2.1)}& \begin{array}{rl}{D}_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x))& ={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log \frac{q_{\varphi }(z\mid x)}{p_{\theta }(z\mid x)}]\\ & ={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z\mid x)-\log p_{\theta }(x,z)]+\log p_{\theta }(x).\end{array}\end{array}$$

Rearranging gives :

$$\begin{array}{}\text{(1.2.2)}& \log p_{\theta }(x)=\underset{{\displaystyle \mathcal{L}(\theta ,\varphi ;x)}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x,z)-\log q_{\varphi }(z\mid x)]}}+D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x)).\end{array}$$

Since $D_{\mathrm{KL}}\ge 0$,

$$\begin{array}{}\text{(1.2.3)}& \mathcal{L}(\theta ,\varphi ;x)\le \log p_{\theta }(x).\end{array}$$

The shared VLB/ELBO objective is therefore :

$$\begin{array}{}\text{(1.2.4)}& \boxed{{\displaystyle \mathrm{VLB}(\theta ,\varphi ;x)=\mathrm{ELBO}(\theta ,\varphi ;x)=\mathcal{L}(\theta ,\varphi ;x)={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log \frac{p_{\theta }(x,z)}{q_{\varphi }(z\mid x)}].}}\end{array}$$

This form (1.2.4) can still be referenced from wiki^[3]. The bound is tight exactly when the variational posterior equals the exact posterior:

$$\begin{array}{}\text{(1.2.5)}& q_{\varphi }(z\mid x)=p_{\theta }(z\mid x)⟺D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x))=0.\end{array}$$

(3) Equivalent likelihood–regularization form

Using $p_{\theta }(x,z)=p_{\theta }(x\mid z)p_{\theta }(z)$ gives :

$$\begin{array}{}\text{(1.3.1)}& \boxed{{\displaystyle \mathcal{L}(\theta ,\varphi ;x)=\underset{\text{expected log-likelihood}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]}}-\underset{\text{regularization toward the prior}}{\underbrace{D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z))}}.}}\end{array}$$

Maximizing this single objective simultaneously (1) improves how well the model explains the observed data and (2) makes the variational posterior remain close to the chosen prior. Equivalently, optimization may minimize $-\mathcal{L}(\theta ,\varphi ;x)$.

2. When to use it

Use the VLB/ELBO when:

• a model contains latent variables and requires an approximate posterior $q_{\varphi }(z\mid x)$;
• direct evaluation or differentiation of $\log p_{\theta }(x)$ is intractable;
• the variational family and its gradient estimator are tractable enough to optimize.

The term evidence means the marginal likelihood $p_{\theta }(x)$, not a separate model component. Thus the difference between the names is terminological, not mathematical.

---

1. 1. Summary for basics of probability theory-Prior and Posterior Probability. ↩︎ ↩︎

2. 2. Kullback-Leibler divergence. ↩︎ ↩︎

3. https://en.wikipedia.org/wiki/Evidence_lower_bound ↩︎