Evidence Lower Bound

The evidence lower bound (ELBO) is a tractable lower bound on the log marginal likelihood of observed data under a probabilistic model with latent variables. Maximising the ELBO is equivalent to minimising the divergence between an approximate posterior and the true posterior, making intractable inference tractable. It is the training objective of variational autoencoders and a cornerstone of variational inference.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:hasPart ml:ReconstructionTerm))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:hasPart ml:KLDivergenceRegulariser))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:hasPart ml:ApproximatePosterior))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:hasPart ml:PriorDistribution))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:hasPart ml:VariationalParameters))

Dependency Relationships

SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:requires ml:ProbabilisticModel))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:requires ml:KullbackLeiblerDivergence))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:requires ml:LatentVariable))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:dependsOn ml:ReparametrisationTrick))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:dependsOn ml:StochasticGradientDescent))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:dependsOn ml:Backpropagation))

Capability Relationships

SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:VariationalAutoencoder))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:VariationalInference))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:AmortisedInference))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:ProbabilisticProgramming))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:BayesianDeepLearning))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:enables ml:LatentDiffusionModel))

Implementation Relationships

SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:implements ml:ApproximateBayesianInference))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:implements ml:VariationalObjective))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:supports ml:GenerativeModel))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:supports ml:LatentVariableModel))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:uses ml:MonteCarloEstimation))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:uses ml:NormalisingFlows))

Reduction Relationships

SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:reducesTo ml:NegativeVariationalFreeEnergy))
SubClassOf(ml:ImportanceWeightedELBO
  ObjectSomeValuesFrom(ml:reducesTo ml:EvidenceLowerBound))
SubClassOf(ml:EvidenceLowerBound
  ObjectSomeValuesFrom(ml:reducesTo ml:KLMinimisationProblem))
SubClassOf(ml:MeanFieldELBO
  ObjectSomeValuesFrom(ml:reducesTo ml:EvidenceLowerBound))

About

The Evidence Lower Bound emerges from the fundamental challenge of Bayesian inference: computing the posterior distribution p(z|x) = p(x|z)p(z)/p(x) requires evaluating the marginal likelihood p(x) = ∫ p(x|z)p(z) dz, an integral that is analytically intractable for virtually all models of practical interest. Early responses to this problem were either exact but exponentially costly (enumerate all latent configurations) or asymptotically exact but slow (Markov Chain Monte Carlo sampling). Variational Bayes, developed through work by Hinton and van Camp (1993), Waterhouse et al. (1996), and Jaakkola and Jordan (1999), proposed a fundamentally different strategy: recast inference as optimisation by introducing a tractable parametric family of distributions Q and finding the member closest to the true posterior.

The ELBO provides the optimisation objective for this strategy. The key insight, known since at least the 1960s in information theory and formalised in the context of machine learning by Neal and Hinton (1998) through the connection to the Expectation-Maximisation algorithm, is that the log marginal likelihood decomposes as:

log p(x) = ELBO(φ) + KL(q(z;φ) || p(z|x))

Since KL divergence is always non-negative, ELBO(φ) ≤ log p(x) for all φ — hence “lower bound.” The bound is tight (ELBO = log p(x)) if and only if q equals the true posterior p(z|x). Maximising the ELBO therefore simultaneously tightens the approximation to the true posterior and improves the model evidence. Because log p(x) is constant with respect to φ, maximising the ELBO is exactly equivalent to minimising KL(q(z;φ) || p(z|x)).

The critical breakthrough enabling modern deep variational methods was the Reparameterisation Trick introduced by Kingma and Welling (2013) and Rezende, Mohamed and Wierstra (2014). For distributions where z can be written as z = g(ε, φ) for noise variable ε ~ p(ε) independent of φ — notably the Gaussian case z = μ(φ) + σ(φ)·ε where ε ~ N(0, I) — the gradient of the ELBO with respect to φ can be estimated by low-variance Monte Carlo, enabling backpropagation through the sampling operation. This made it possible to train deep encoder-decoder architectures end-to-end: the encoder outputs variational parameters φ(x) = (μ(x), σ(x)) and the decoder models p(x|z). The resulting architecture, the Variational Autoencoder, brought variational inference into the mainstream of deep learning.

The ELBO thus sits at the intersection of classical Bayesian statistics, information theory, optimisation, and deep learning, providing the theoretical glue that allows principled probabilistic reasoning at the scale of modern neural network architectures.

Mathematical Derivation and Decomposition

Beginning from Bayes’ theorem and Jensen’s inequality, the ELBO can be derived as follows. For any distribution q(z):

log p(x) = log ∫ p(x, z) dz = log ∫ q(z) · [p(x, z) / q(z)] dz ≥ ∫ q(z) log[p(x, z) / q(z)] dz (by Jensen’s inequality applied to the concave log function) = E_{q(z)}[log p(x, z) - log q(z)] = E_{q(z)}[log p(x, z)] + H[q(z)]

where H[q] denotes the entropy of q. This is the ELBO.

Expanding p(x, z) = p(x|z)p(z) yields the canonical decomposition:

ELBO(φ) = E_{q(z;φ)}[log p(x|z)] - KL(q(z;φ) || p(z))

The first term is the reconstruction term or expected log-likelihood: it measures how well the generative model reconstructs observations given latent samples drawn from the approximate posterior. In a Variational Autoencoder with Gaussian encoder and Bernoulli or Gaussian decoder, this reduces to binary or mean-squared reconstruction loss respectively.

The second term is the KL regularisation term: it penalises approximate posteriors q that deviate from the prior p(z), regularising the latent space. For Gaussian q with mean μ and diagonal covariance σ² and standard Gaussian prior N(0, I), the KL has the closed form: KL = -½ Σ_j (1 + log σ_j² - μ_j² - σ_j²), which is analytically tractable and does not require Monte Carlo sampling.

The ELBO can also be written in terms of KL divergence directly:

ELBO(φ) = log p(x) - KL(q(z;φ) || p(z|x))

confirming that maximising the ELBO minimises the KL from the approximate to the true posterior, simultaneously tightening the bound and improving the approximation quality.

Key Mechanisms and Algorithmic Variants

Reparameterisation Trick

The Reparameterisation Trick enables low-variance gradient estimation through the ELBO’s reconstruction term. For the Gaussian case q(z|x) = N(μ(x), σ²(x)·I), sampling z ~ q(z|x) is reparameterised as z = μ(x) + σ(x) ⊙ ε where ε ~ N(0, I). The ELBO gradient with respect to encoder parameters φ becomes:

∇_φ ELBO ≈ ∇_φ log p(x | g(ε, φ)) - ∇_φ KL(q(z;φ) || p(z))

estimated by sampling ε from its fixed noise distribution and differentiating through the deterministic function g. This achieves dramatically lower variance than the score-function (REINFORCE) estimator and is standard in all major deep learning frameworks.

Mean-Field Variational Inference

The mean-field approximation restricts the variational family to fully factored (independent) distributions: q(z; φ) = ∏_i q_i(z_i; φ_i). Under this assumption and exponential family models, coordinate ascent variational inference (CAVI) provides closed-form update equations that iteratively optimise each factor q_i while holding others fixed. Mean-field CAVI is efficient and converges to a local ELBO maximum but cannot represent posterior correlations between latent dimensions, leading to systematic underestimation of posterior variance.

Stochastic Variational Inference (SVI)

Stochastic Variational Inference (Hoffman et al., 2013) scales mean-field VI to large datasets by subsampling the ELBO’s data term. The local ELBO at a mini-batch B of size M drawn from N total data points is:

ELBO_B(φ) ≈ (N/M) Σ_{x ∈ B} E_q[log p(x|z)] - KL(q(z;φ) || p(z))

Natural gradient descent on the ELBO with respect to global variational parameters (those shared across data points) converges faster than vanilla gradient descent for exponential family models, exploiting the Riemannian geometry of the space of probability distributions.

Importance-Weighted ELBO (IWAE)

Burda, Grosse and Salakhutdinov (2016) introduced the Importance-Weighted Autoencoder (IWAE), which tightens the ELBO using importance-weighted samples:

IWAE_K(φ) = E_{z_1,…,z_K ~ q}[log (1/K Σ_k p(x,z_k)/q(z_k;φ))]

IWAE_K ≥ IWAE_K’ when K ≥ K’, and IWAE_K → log p(x) as K → ∞. Tighter bounds improve likelihood estimation but, as Rainforth et al. (2018) showed, they can actually produce worse posterior approximations: the variance of the gradient estimator with respect to the inference network parameters increases with K, a phenomenon called “tighter bounds are not necessarily better.” This trade-off between tightening the bound and the quality of the learned encoder is an active research topic.

Black-Box Variational Inference (BBVI)

For latent variable models where the reparameterisation trick does not apply — discrete latent variables, models with non-differentiable likelihoods — Black-Box Variational Inference (Ranganath et al., 2014) uses the score-function estimator (REINFORCE): ∇_φ ELBO = E_q[log p(x,z) · ∇_φ log q(z;φ)]. This is unbiased but high-variance; variance reduction techniques such as control variates, Rao-Blackwellisation, and NVIL are employed in practice. Extensions include Discrete VAEs using Gumbel-Softmax relaxation (Jang et al., 2017; Maddison et al., 2017) for differentiable approximations to discrete sampling.

Normalising Flows and Enriched Posteriors

Normalising Flows enrich the variational family beyond mean-field Gaussians by composing a sequence of invertible differentiable transformations, mapping a simple base distribution (e.g., isotropic Gaussian) to a complex approximate posterior. Variants include planar flows, radial flows (Rezende & Mohamed, 2015), Real-NVP (Dinh et al., 2017), Glow (Kingma & Dhariwal, 2018), and Neural Spline Flows (Durkan et al., 2019). Each transformation preserves tractable density computation via the change-of-variables formula, enabling tighter ELBO estimation with more expressive posteriors.

Locally-Enhanced Bounds

For hierarchical latent variable models with multiple levels of stochastic variables, locally-enhanced bounds (Llinares-López et al., 2022) apply importance-weighting selectively at local (per-datapoint) latent levels while retaining efficient global parameter updates, achieving tighter bounds for hierarchical models without the full cost of global importance weighting.

Applications and Use Cases

Variational Autoencoders (VAE)

The Variational Autoencoder is the canonical application of the ELBO. An encoder neural network maps input x to variational parameters (μ(x), σ(x)); the ELBO provides the training objective combining reconstruction quality and latent space regularisation. The regularised, continuous latent space supports:

• Image generation: smooth interpolation and random sampling in latent space; Kingma & Welling (2013) demonstrated digit generation on MNIST; subsequent hierarchical VAEs (NVAE, VDVAE, Ladder VAE) achieve near-GAN quality on face generation benchmarks.

• Representation learning: disentangled representations (β-VAE, FactorVAE) enforce stronger regularisation to learn interpretable, semantically meaningful latent factors.

• Anomaly detection: reconstruction error and ELBO estimates serve as anomaly scores; datapoints lying far from the prior in latent space or reconstructed poorly are flagged as anomalous.

• Semi-supervised learning: VAE latent representations transfer to downstream classification with few labelled examples.

  Latent Diffusion Models

  Stable Diffusion and related Latent Diffusion Model architectures train a VAE encoder-decoder as the first stage, compressing high-dimensional images into a regularised latent space using the ELBO objective. The diffusion denoising model then operates in this compressed latent space rather than pixel space, achieving approximately 4-8× computational reduction while preserving perceptual quality. The VAE’s ELBO-trained regularised latent space is essential for the diffusion model’s Gaussian noise process to be well-defined: unregularised autoencoders produce latent spaces with holes and discontinuities that are incompatible with diffusion.

  Scalable Bayesian Inference for Topic Models

  Latent Dirichlet Allocation (LDA, Blei et al. 2003) uses variational inference with the ELBO objective for approximate posterior inference over document-topic and topic-word distributions. Stochastic VI (Hoffman et al., 2013) scaled LDA to corpora of millions of documents by mini-batch ELBO optimisation, replacing slow MCMC sampling. Subsequent neural topic models (ProdLDA, NTM) replace the CAVI updates with amortised inference networks trained via the ELBO, inheriting the scalability of deep learning.

  Bayesian Deep Learning and Uncertainty Quantification

  Bayes by Backprop (Blundell et al., 2015) applies the ELBO to train neural networks with distributions over weights rather than point estimates. The ELBO here decomposes as E_q[log p(D|w)] − KL(q(w|θ) || p(w)), where the reconstruction term measures data fit and the KL term regularises weight distributions toward the prior. This yields calibrated predictive uncertainty for safety-critical applications including medical diagnosis, autonomous driving, and scientific simulation.

  Probabilistic Programming

  Probabilistic Programming frameworks (Pyro, NumPyro, Stan via ADVI, TensorFlow Probability) expose the ELBO as a first-class training objective. Users specify a generative model (prior p(z) and likelihood p(x|z)) in a probabilistic programming language; the framework automatically derives an ELBO objective, chooses a variational family (mean-field Gaussian by default, or user-specified), and runs gradient-based optimisation. This democratises approximate Bayesian inference, enabling practitioners without expertise in variational calculus to apply VI to custom models.

  Drug Discovery and Molecular Design

  Graph VAEs and molecule VAEs (JT-VAE, GCPN) trained with the ELBO learn continuous latent spaces over molecular graphs. Bayesian optimisation in this latent space — proposing new molecules predicted to have high activity by a surrogate model — is an emerging paradigm for lead compound identification in pharmaceutical research. The University of Edinburgh’s Drug Discovery group and UCL’s Alzheimer’s Research UK Drug Discovery Institute both employ deep generative models in active learning pipelines, with the ELBO serving as the training objective for molecular property prediction and generation.

  Model Comparison and Selection

  The ELBO provides an approximation to the log marginal likelihood log p(x|M) for model M, which is the Bayesian criterion for model comparison (Bayes factor). Comparing ELBO values across models of different complexity gives a principled basis for model selection that automatically balances fit and complexity — the Bayesian Occam’s Razor. In practice, ELBO-based model selection is used in Gaussian mixture model order selection, state-space model dimension selection, and neural architecture search over generative model families.

  Connection to Free Energy and Information Theory

  The ELBO has deep connections to classical information theory and statistical physics. Rewriting the ELBO:

  ELBO(φ) = E_q[log p(x|z)] − KL(q(z;φ) || p(z)) = E_q[log p(x, z)] + H[q] = −F[q]

  where F[q] is the variational free energy — the difference between the expected energy (−E_q[log p(x,z)]) and the entropy H[q]. Minimising variational free energy (equivalently maximising the ELBO) is the statistical physics principle of free energy minimisation, applied to probability distributions.

  Karl Friston’s Free Energy Principle in computational neuroscience (Friston, 2010) argues that biological brains implement exactly this variational free energy minimisation as the universal objective driving perception and action. Under this framework, perception corresponds to updating the approximate posterior q to minimise free energy given sensory observations (ELBO maximisation), while action (active inference) selects motor commands that make future observations conform to the prior’s predictions, also via free energy minimisation. The ELBO thus serves as a unifying mathematical principle connecting probabilistic machine learning, information theory, and theories of biological cognition.

  The Kullback-Leibler Divergence KL(q || p) that appears in the ELBO is asymmetric: it penalises places where q assigns mass but p does not, leading mean-field approximations to under-estimate posterior variance (zero-avoiding behaviour). The reverse KL(p || q) is used in Expectation Propagation and leads to moment-matching behaviour. This asymmetry has practical consequences: ELBO-optimised variational posteriors tend to be over-concentrated relative to the true posterior, an important limitation for uncertainty quantification.

  Academic Context

  The ELBO’s theoretical history spans information theory, statistics, and machine learning:

• Neal and Hinton (1998): unified the EM algorithm and variational Bayes, showing that the E-step of EM performs ELBO maximisation over per-datapoint local variational parameters, while the M-step maximises over global model parameters. This connection clarified the ELBO’s role as the central objective for latent variable models.

• Jaakkola and Jordan (1999): developed variational lower bounds for sigmoid belief networks and logistic regression, demonstrating practical variational Bayes before the deep learning era.

• Blei, Ng, and Jordan (2003): applied variational inference with the ELBO to Latent Dirichlet Allocation, establishing VI as the standard scalable inference algorithm for topic models.

• Kingma and Welling (2013) / Rezende, Mohamed, and Wierstra (2014): independently introduced the reparameterisation trick and the Variational Autoencoder, launching the era of deep variational methods and bringing the ELBO to the centre of deep generative modelling.

• Hoffman, Blei, Wang, and Paisley (2013): introduced Stochastic Variational Inference, scaling the ELBO to large datasets via natural gradient mini-batch updates.

• Ranganath, Gerrish, and Blei (2014): introduced Black-Box Variational Inference via score-function gradient estimation, making VI applicable to non-conjugate and non-reparameterisable models.

• Burda, Grosse, and Salakhutdinov (2016): introduced the Importance-Weighted ELBO (IWAE), providing tighter bounds and illuminating the trade-off between bound tightness and encoder quality.

• Rainforth et al. (2018): proved that tighter ELBO bounds are not necessarily better for learning the inference network, establishing a fundamental tension in bound optimisation.

• Blei, Kucukelbir, and McAuliffe (2017): wrote the canonical review of variational inference, systematising the field’s theoretical foundations, algorithmic variants, and connections to EM.

  The ELBO and variational inference are active research areas with open problems in: discrete variational methods, tighter bound construction without gradient variance explosion, combining VI with MCMC for refinement, VI for causal models, and information-geometric analysis of variational families.

  Current Landscape (2026)

  Several active trends shape ELBO research and application in 2025-2026:

  Latent diffusion as dominant generative paradigm: The success of Stable Diffusion (Rombach et al., 2022) and successors such as FLUX, Stable Diffusion 3, and Sora has established ELBO-trained VAE encoders as the standard compression backbone for image and video generation. The ELBO’s role has expanded from a training objective for generative models to a foundational component of the latent space infrastructure supporting trillion-parameter multimodal systems.

  Tighter bounds for diffusion models: Recent work (2025) shows that diffusion model reweighted loss objectives correspond to weighted sums of cascaded variational lower bounds on the log-likelihood, yielding provably tighter bounds than the standard ELBO for diffusion processes while preserving tractable generation. This has renewed theoretical interest in the ELBO’s relationship to score-based generative modelling.

  Analytical ELBO bounds for binary models: Analytic lower bounds on the ELBO for binary VAEs using Taylor expansion of the Bernoulli likelihood have been derived, providing closed-form surrogate objectives that facilitate model selection and initialisation without Monte Carlo noise.

  Upper and lower bound bracketing: New VAE variants introduce extra encoders and fixed posteriors to yield both evidence lower bounds (ELBO) and evidence upper bounds (EUBO), improving convergence diagnostics and enabling tighter posterior certification — particularly valuable for safety-critical uncertainty quantification.

  Probabilistic programming adoption: NumPyro and Pyro continue to grow as production VI platforms. PyData Berlin 2025 featured stochastic VI with NumPyro as a tutorial, reflecting mainstream adoption of ELBO-based inference in data science practice beyond academia.

  Quantum variational autoencoders: Hybrid quantum-classical VAE architectures trained with ELBO objectives are being explored for molecular property prediction, with potential computational advantages on quantum hardware (Berta et al., 2022; Cherrat et al., 2024), though practical quantum advantage remains undemonstrated.

  Free Energy Principle applications: Active inference frameworks implementing the ELBO as a neural circuit objective continue to attract neuroscience and AI safety research, with groups at UCL (Karl Friston’s group), Oxford, and Edinburgh developing continuous-time active inference agents.

  UK Context

  UK academic contributions to ELBO and variational inference span foundational theory, applications, and computational tools:

  UCL: Karl Friston’s group at UCL remains internationally dominant in the Free Energy Principle and active inference, which grounds biological perception and action in ELBO/variational free energy minimisation. UCL also hosts the UKRI Centre for Doctoral Training in Foundational AI and the Gatsby Computational Neuroscience Unit, where Maneesh Sahani and collaborators have contributed to variational methods for neuroscience data analysis. The UCL Alzheimer’s Research UK Drug Discovery Institute uses deep generative models including VAEs in molecular screening pipelines.

  University of Edinburgh: The Edinburgh School of Informatics is home to foundational Bayesian deep learning research. Amos Storkey’s group works on generative models and principled uncertainty quantification; the Bayesian and Graphical Models group (formerly including Zoubin Ghahramani, now at Cambridge/Google Brain) has historical roots in variational Bayes. The Edinburgh Drug Discovery unit employs latent variable models in drug-target interaction prediction.

  University of Cambridge: Zoubin Ghahramani (now Chief Scientist at Google Brain, affiliated with Cambridge) co-developed many foundational variational methods. The Cambridge machine learning group (Richard Turner and collaborators) works on neural processes, meta-learning, and structured variational families.

  DeepMind (London): DeepMind has applied variational inference and the ELBO in world models (DREAMER series), molecular docking (AlphaFold relies on Bayesian uncertainty quantification), and protein design. The PlaNet and DREAMER model-based RL systems use ELBO-trained recurrent state-space models as world models for planning.

  Alan Turing Institute: The Turing’s probabilistic AI research programme supports variational inference research applied to climate modelling, epidemiology, and materials science across its partner universities.

  Northern England: The University of Manchester’s machine learning group (Gavin Brown and collaborators) works on ensemble methods and calibration that connect to ELBO-based uncertainty quantification. Manchester’s relationship with the NHS Informatics Collaborative drives application of Bayesian generative models in electronic health records analysis.

  Future Directions (2026-2030)

  Several research and application trajectories will shape ELBO and variational inference over the next four years:

• Continuous-time and stochastic differential equation (SDE) ELBO: Variational objectives for continuous-time latent variable models defined by SDEs, bridging diffusion models and classical variational inference into a unified framework. Active research at Cambridge, Edinburgh, and leading US groups.

• Causal variational inference: Integrating causal graphical models into the ELBO framework to enable counterfactual reasoning and causal discovery from observational data; addresses a fundamental limitation of current variational methods that learn correlational rather than causal structure.

• Combinatorial and discrete ELBO: Efficient gradient estimation for discrete latent variables beyond Gumbel-Softmax, including reinforcement learning applications with categorical action spaces and language model structure.

• Certified uncertainty quantification: ELBO-based bounds combined with conformal prediction to provide coverage-guaranteed prediction intervals, meeting regulatory requirements for safety-critical AI applications in healthcare and autonomous systems.

• Neurosymbolic variational models: Incorporating symbolic prior knowledge into the prior p(z) and variational family Q, enabling hybrid systems that combine neural flexibility with structured reasoning.

• Quantum-enhanced variational inference: As fault-tolerant quantum computing matures post-2028, quantum variational algorithms may offer quadratic speedups for specific variational inference sub-problems, with near-term hybrid classical-quantum ELBO optimisation as an intermediate step.

  Research and Literature

  1. Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. ICLR 2014. https://arxiv.org/abs/1312.6114 — Introduced the VAE and reparameterisation trick.
  2. Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ICML 2014. https://arxiv.org/abs/1401.4082
  3. Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. JASA 2017. https://doi.org/10.1080/01621459.2017.1285773 — Canonical review of VI theory and practice.
  4. Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic Variational Inference. JMLR 14. https://jmlr.org/papers/v14/hoffman13a.html
  5. Ranganath, R., Gerrish, S., & Blei, D. M. (2014). Black Box Variational Inference. AISTATS 2014. https://arxiv.org/abs/1401.0118
  6. Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan (Ed.), Learning in Graphical Models. Kluwer. https://doi.org/10.1007/978-94-011-5014-9_12
  7. Hinton, G. E., & van Camp, D. (1993). Keeping the neural networks simple by minimizing the description length of the weights. COLT 1993. https://doi.org/10.1145/168304.168306
  8. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. JMLR 3. https://jmlr.org/papers/v3/blei03a.html
  9. Burda, Y., Grosse, R., & Salakhutdinov, R. (2016). Importance Weighted Autoencoders. ICLR 2016. https://arxiv.org/abs/1509.00519
  10. Rainforth, T., Kosiorek, A. R., Le, T. A., Maddison, C. J., Igl, M., Wood, F., & Teh, Y. W. (2018). Tighter Variational Bounds are Not Necessarily Better. ICML 2018. https://arxiv.org/abs/1802.04537
  11. Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138. https://doi.org/10.1038/nrn2787
  12. Jaakkola, T. S., & Jordan, M. I. (1999). Variational probabilistic inference and the QMR network. Journal of Artificial Intelligence Research, 10, 291-322.
  13. Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. ICML 2015. https://arxiv.org/abs/1505.05770
  14. Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2017). Density estimation using Real-NVP. ICLR 2017. https://arxiv.org/abs/1605.08803
  15. Kingma, D. P., & Dhariwal, P. (2018). Glow: Generative Flow with Invertible 1x1 Convolutions. NeurIPS 2018. https://arxiv.org/abs/1807.03039
  16. Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight Uncertainty in Neural Networks. ICML 2015. https://arxiv.org/abs/1505.05424
  17. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., … & Lerchner, A. (2017). β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. ICLR 2017. https://openreview.net/forum?id=Sy2fchgb
  18. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. https://arxiv.org/abs/2112.10752
  19. Jang, E., Gu, S., & Poole, B. (2017). Categorical Reparameterization with Gumbel-Softmax. ICLR 2017. https://arxiv.org/abs/1611.01144
  20. Maddison, C. J., Mnih, A., & Teh, Y. W. (2017). The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. ICLR 2017. https://arxiv.org/abs/1611.00712
  21. Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2020). Dream to Control: Learning Behaviors by Latent Imagination (DREAMER). ICLR 2020. https://arxiv.org/abs/1912.01603
  22. Llinares-López, F., Quost, B., Blundell, C., & Kavukcuoglu, K. (2022). Variational Inference with Locally Enhanced Bounds for Hierarchical Models. ICML 2022. https://arxiv.org/abs/2203.04432
  23. Berta, M., Wilde, M. M., & Winter, A. (2022). Quantum Rényi Divergences and the Strong Converse Exponent. IEEE Transactions on Information Theory. https://doi.org/10.1109/TIT.2022.3179905
  24. Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., … & Aspuru-Guzik, A. (2018). Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. ACS Central Science, 4(2), 268-276. https://doi.org/10.1021/acscentsci.7b00572
  25. Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2017). Automatic Differentiation Variational Inference. JMLR 18(14). https://jmlr.org/papers/v18/16-107.html
  26. Parr, T., Pezzulo, G., & Friston, K. J. (2022). Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press.
  27. Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., … & Goodman, N. D. (2019). Pyro: Deep Universal Probabilistic Programming. JMLR 20(28). https://jmlr.org/papers/v20/18-403.html
  28. Phan, D., Pradhan, N., & Jankowiak, M. (2019). Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro. arXiv:1912.11554. https://arxiv.org/abs/1912.11554

Provenance