Expectation Maximisation (EM) is an iterative algorithm for finding maximum-likelihood or maximum-a-posteriori estimates of parameters in statistical models with latent (unobserved) variables. It alternates between an E-step, which computes the expected value of the complete-data log-likelihood given current parameters, and an M-step, which maximises that expectation to update the parameters. EM is guaranteed to monotonically increase the likelihood at each iteration and is widely used for Gaussian mixture models, hidden Markov models, and missing-data problems.
Semantic Classification
Content
Compositional Relationships (Components)
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:EStep))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:MStep))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:CompleteDataLogLikelihood))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:PosteriorDistribution))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:PriorDistribution))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:hasPart ai:ConvergenceCriterion))
Dependency Relationships
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:LatentVariable))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:LikelihoodFunction))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:ProbabilisticModel))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:ProbabilityTheory))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:NumericalLinearAlgebra))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:requires ai:MeasureTheory))
Capability Relationships
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:GaussianMixtureModel))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:HiddenMarkovModel))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:FactorAnalysis))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:SoftClustering))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:MissingDataImputation))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:TopicModelling))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:enables ai:LatentDirichletAllocation))
Implementation Relationships
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:implements ai:EvidenceLowerBound))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:implements ai:CoordinateAscentVariationalInference))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:implements ai:BaumWelchAlgorithm))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:implements ai:MaximumLikelihoodEstimation))
Reduction Relationships
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:reducesTo ai:CoordinateAscent))
SubClassOf(ai:ExpectationMaximisation
ObjectSomeValuesFrom(ai:reducesTo ai:GradientDescent))
About
Expectation Maximisation occupies a foundational position in statistics and machine learning as the canonical method for fitting latent-variable models. Its genesis lies in the observation that many seemingly hard estimation problems become tractable once one imagines having access to the latent (missing) data. The seminal 1977 paper by Dempster, Laird, and Rubin in the Journal of the Royal Statistical Society Series B formalised this intuition into a rigorous algorithm with convergence guarantees, unifying dozens of previously disparate methods — including the Baum-Welch re-estimation algorithm for hidden Markov models (Baum et al., 1970), the IRLR algorithm in factor analysis, and various mixture model fitting procedures — under a single theoretical umbrella. The paper has since accumulated over 70,000 citations, making it among the most cited works in statistics.
The theoretical core of EM can be understood through the lens of the Evidence Lower Bound (ELBO). The observed-data log-likelihood log p(X | θ) can always be decomposed as ELBO(q, θ) + KL(q || p(Z | X, θ)), where q is any distribution over the latent variables Z. The E-step sets q equal to the true posterior p(Z | X, θ_old), driving the KL divergence to zero and making the ELBO tight. The M-step then maximises the ELBO with respect to θ, which — since q is held fixed — is identical to maximising the complete-data expected log-likelihood Q(θ | θ_old). This ELBO perspective immediately reveals the connection to Variational Inference: when the true posterior is intractable, one restricts q to a tractable family (e.g., mean-field), giving the variational EM algorithm. When the model parameters are neural network weights, one obtains the Variational Autoencoder (Kingma and Welling, 2013), where amortised inference networks approximate the E-step.
A key practical concern with EM is the sensitivity to initialisation. Because the algorithm is guaranteed only to find a local maximum, different starting points may yield qualitatively different solutions. For Gaussian Mixture Models, multiple random restarts and selection by highest achieved likelihood is standard practice. The K-means algorithm is often used to provide a warm-start initialisation for Gaussian mixture EM. Regularisation — equivalent to placing a Prior Distribution on the parameters and performing MAP-EM — can prevent degenerate solutions (e.g., variance collapsing to zero for a mixture component that fits a single data point exactly). An important variant is the online EM algorithm (Cappé and Moulines, 2009), which processes data in mini-batches and enables EM to scale to streaming data settings, making it applicable to large-scale unsupervised learning.
Components / Architecture
The EM algorithm has a two-phase structure executed iteratively until convergence:
E-Step (Expectation Step):
-
Compute Q(θ | θ_t) = E_{Z | X, θ_t}[log p(X, Z | θ)]
-
This requires computing the posterior distribution p(Z | X, θ_t) over all latent variables given the current parameter estimate θ_t and the observed data X
-
For Gaussian Mixture Models, the posterior factorises into soft assignment responsibilities: r_{nk} = p(z_n = k | x_n, θ_t) ∝ π_k N(x_n | μ_k, Σ_k)
-
For Hidden Markov Models, the posterior is computed by the forward-backward algorithm in O(T K²) time, where T is the sequence length and K is the number of hidden states
-
When the posterior is intractable, Monte Carlo EM replaces the exact expectation with a Monte Carlo estimate, or variational EM replaces it with a tractable variational approximation
M-Step (Maximisation Step):
-
Find θ_{t+1} = argmax_θ Q(θ | θ_t)
-
For exponential family models, this reduces to computing sufficient statistics weighted by the E-step responsibilities — a closed-form operation
-
For Gaussian Mixture Models: update means as weighted sample means, covariances as weighted sample covariances, and mixing weights as the normalised sum of responsibilities
-
For general models where no closed form exists, a gradient ascent step on Q suffices (Generalised EM / GEM); convergence guarantees still hold provided each M-step increases Q
Convergence Criterion:
-
Monitor the observed-data log-likelihood log p(X | θ_t) or the relative change in Q
-
Standard practice: stop when |log p(X | θ_{t+1}) - log p(X | θ_t)| < ε for a small tolerance ε
-
The log-likelihood is non-decreasing by the EM convergence theorem, so any decrease signals a numerical bug
Key Properties:
-
Monotone convergence: log p(X | θ_{t+1}) ≥ log p(X | θ_t) at every iteration
-
Linear convergence rate: the rate depends on the fraction of “missing information” in the latent variables (the larger the fraction, the slower the convergence)
-
Local optimum sensitivity: no guarantee of reaching the global maximum in non-convex settings
-
Each iteration has closed-form updates for exponential family models, making it computationally cheap per step
Formal Algorithm
Given observed data X = {x_1, …, x_N}, latent variables Z, and parameters θ:
-
Initialise θ_0 (random, K-means warm-start, or domain knowledge)
-
Repeat until convergence:
-
E-step: Compute Q(θ | θ_t) = Σ_Z p(Z | X, θ_t) log p(X, Z | θ)
-
M-step: Set θ_{t+1} = argmax_θ Q(θ | θ_t)
-
Check: Compute log p(X | θ_{t+1}); stop if change < ε 3. Return θ* = θ_{t+1}
For the Gaussian Mixture Model with K components, N data points, D dimensions:
-
-
E-step: r_{nk} = π_k N(x_n; μ_k, Σ_k) / Σ_j π_j N(x_n; μ_j, Σ_j)
-
M-step: N_k = Σ_n r_{nk}; μ_k = (1/N_k) Σ_n r_{nk} x_n; Σ_k = (1/N_k) Σ_n r_{nk}(x_n - μ_k)(x_n - μ_k)^T; π_k = N_k / N
Major Variants and Families
Standard EM: Exact E and M steps, closed-form for exponential family models. Workhorse for Gaussian Mixture Models and basic Hidden Markov Models.
Variational EM (VEM): Replaces the intractable posterior in the E-step with a variational approximation from a tractable family (e.g., mean-field). The foundation of Variational Autoencoders when the variational distribution is parameterised by a neural network (amortised inference). Used in Latent Dirichlet Allocation (LDA) topic models.
Monte Carlo EM (MCEM): Replaces the expectation in the E-step with a Monte Carlo estimate drawn from p(Z | X, θ_t) via MCMC or importance sampling. Applicable when both exact posteriors and tractable variational families are unavailable. Used in complex hierarchical models.
Online EM: Processes data in mini-batches with a stochastic E-step, enabling application to streaming and large-scale datasets. Convergence to stationary points of the likelihood under mild conditions (Cappé and Moulines, 2009).
Expectation Propagation (EP): A related message-passing algorithm that, unlike mean-field VEM, uses moment matching to approximate the posterior, often giving better calibrated posteriors. Used in Graphical Models and Gaussian process classification.
Generalised EM (GEM): Replaces the exact M-step maximisation with any update that increases Q, enabling use when the M-step lacks a closed form. Subsumes partial M-step variants such as the ECM (Expectation-Conditional-Maximisation) algorithm.
Differentiable EM: Formulates EM as a differentiable computation graph, allowing gradients to flow through E and M steps during end-to-end Deep Learning training. Applied to clustering layers within neural networks and in 2025 research on Gaussian mixture VAEs (arxiv:2501.07358).
Baum-Welch Algorithm: The specific instantiation of EM for Hidden Markov Models, using the forward-backward algorithm to compute the E-step responsibilities over hidden state sequences. Underpins parameter estimation in speech recognition and computational biology.
Use Cases / Major Families
Unsupervised Clustering (Gaussian Mixture Models): The most common entry point for practitioners. EM fits a mixture of Gaussian components to unlabelled data, providing soft cluster assignments (responsibilities) rather than hard labels. Widely used in image segmentation, customer segmentation, and anomaly detection. The responsibilities naturally model uncertainty, distinguishing it from K-means’ hard assignments.
Speech Recognition and Acoustic Modelling: The Baum-Welch algorithm (EM for Hidden Markov Models) was the central learning algorithm for acoustic models in automatic speech recognition (ASR) systems throughout the 1980s–2010s. Systems such as HTK, Kaldi, and CMU Sphinx relied on EM-trained Gaussian mixture HMMs. Even in deep learning ASR, EM is used for forced alignment to generate frame-level labels for training neural network acoustic models.
Natural Language Processing — Topic Models: Latent Dirichlet Allocation (LDA) and probabilistic Latent Semantic Analysis (pLSA) use variational EM or Gibbs sampling (MCMC) to infer topic distributions. These models have been applied to document clustering, information retrieval, and knowledge discovery across millions of documents. EM-based topic models remain interpretable baselines even as large language models dominate NLP tasks.
Bioinformatics and Genomics: EM is used in the GATK variant calling pipeline for genotype likelihood estimation, in RNA-seq quantification tools such as RSEM and kallisto for transcript abundance estimation, and in population genetics for estimating allele frequencies and ancestral admixture proportions (e.g., the STRUCTURE software). The Unified RNA-Sequencing Model (URSM, 2024–2025) applies hierarchical EM to jointly model batch effects, dropout, and mixture weights, enabling deconvolution of tumour microenvironments that inform immunotherapy response prediction.
Missing Data Imputation: When data contains missing-at-random entries, EM provides principled parameter estimation without discarding incomplete cases. The algorithm iterates between imputing missing values (E-step) and re-estimating parameters on the completed data (M-step). Applications include clinical trials with dropout, longitudinal cohort studies, and survey data.
Medical Imaging — Image Reconstruction: Maximum likelihood expectation maximisation (ML-EM) and its accelerated variant ordered-subsets EM (OS-EM) are the gold-standard reconstruction algorithms for Positron Emission Tomography (PET) and Single Photon Emission CT (SPECT), where the reconstruction problem is inherently a Poisson likelihood maximisation over latent photon counts.
Robotics — System Identification: EM is applied to learn the parameters of state-space models (system identification) in the presence of process noise and measurement noise. Closely related to the Bayes Filter family, EM learns the transition and emission matrices that a Kalman filter or particle filter uses for online estimation.
Federated and Privacy-Preserving Learning: Online EM has been adapted for federated settings where data cannot leave local devices, with the E-step computed locally and only sufficient statistics shared with a central server for the M-step. This pattern enables privacy-preserving mixture model estimation without sharing raw data.
Academic Context
The foundational paper is Dempster, Laird, and Rubin (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm,” Journal of the Royal Statistical Society, Series B. The paper provided the first general proof of the monotone convergence property and unified a vast range of previously ad hoc procedures. Neal and Hinton (1998), “A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants,” provided the ELBO interpretation that connects EM to variational inference. Wu (1983), “On the Convergence Properties of the EM Algorithm,” Annals of Statistics, established formal convergence theory. Cappé and Moulines (2009), “On-Line Expectation-Maximization Algorithm for Latent Data Models,” introduced theoretically grounded online EM. Kingma and Welling (2013/2014), “Auto-Encoding Variational Bayes,” showed that amortised variational EM recovers the VAE framework, connecting EM to deep generative modelling.
UK academic contributions are substantial. The Gatsby Computational Neuroscience Unit at University College London (UCL) has produced influential work on variational EM and approximate inference, including Turner and Sahani (2010) on problems with variational EM for non-conjugate models. Zoubin Ghahramani’s group, first at UCL and later at Cambridge (now at Google DeepMind), produced seminal work on EM for graphical models, including the Kalman smoother EM for linear dynamical systems and EM for infinite mixture models. Christopher Bishop (previously at Edinburgh, then Microsoft Research Cambridge) popularised EM extensively in his “Pattern Recognition and Machine Learning” textbook (2006), which remains the standard reference. The Edinburgh Centre for Robotics applies EM-based system identification in robotic platforms. The Alan Turing Institute facilitates collaborative research across UK universities on probabilistic inference methods including EM.
Current Landscape (2026)
EM remains an active research algorithm rather than a historical footnote. Several 2024–2025 developments are noteworthy:
Deep Learning Integration: Differentiable EM formulations allow EM iterations to be embedded as differentiable layers within neural networks, trained end-to-end. A January 2025 paper (arxiv:2501.07358) demonstrates Deep Generative Clustering with VAEs and EM, learning joint generative models that interleave cluster assignment (EM M-step) with learned representations (VAE). The connection between EM and the ELBO objective means that any framework that optimises the ELBO — including diffusion models and normalising flows — can be viewed as a generalised EM.
Optimal Transport and EM: A 2025 paper (arxiv:2509.02109) introduces differentiable EM for Gaussian Mixture Model optimal transport, enabling use of EM within transport plans between mixture distributions, with applications in domain adaptation.
Federated and Privacy-Preserving Settings: EM’s mini-batch sufficient statistics pattern is well-suited to federated learning, where local devices compute local E-steps and a central server aggregates sufficient statistics for the global M-step. This has been applied in 2024–2025 for privacy-preserving clustering and anomaly detection in healthcare and finance.
LLM Era Applications: EM-style inference appears in mixture-of-experts routing for large language models, where the gating network assigns soft responsibilities (E-step) and expert parameters are updated to maximise the routing-weighted likelihood (generalised M-step). Sparse EM variants inform training of mixture-of-experts architectures such as Mixtral.
Bioinformatics Tooling: RNA-seq deconvolution tools continue to adopt hierarchical EM; the URSM model (2024–2025) applies EM to jointly model batch effects and tissue composition in tumour microenvironments. GATK’s variant calling pipeline continues to rely on EM-based genotype likelihood estimation.
Hardware Acceleration: EM is increasingly implemented on GPUs, particularly for the parallel computation of responsibilities across N data points and K components in the E-step of Gaussian mixture models. Libraries including scikit-learn (Python), Stan, and JAX support GPU-accelerated EM variants.
UK Context
The United Kingdom has a strong tradition of research into probabilistic inference methods in which EM is central. The Gatsby Computational Neuroscience Unit (UCL), established in 1998, has maintained continuous research on variational and exact EM methods, with significant contributions from Maneesh Sahani, Peter Latham, and their collaborators. Zoubin Ghahramani, whose doctoral and postdoctoral work produced canonical EM implementations for dynamical systems, built his UK research group at the Gatsby Unit before moving to Cambridge, where he led the Machine Learning group in the Department of Engineering. Christopher Bishop’s PRML textbook, written partly during his time at Edinburgh and Microsoft Research Cambridge, has trained a generation of UK researchers in EM. The University of Edinburgh’s Institute for Adaptive and Neural Computation has contributed EM-based methods for speech and signal processing, historically important for UK speech technology companies. The Alan Turing Institute (ATI) hosts cross-university research on scalable probabilistic inference; several ATI fellows work on online and distributed EM variants applicable to healthcare data. In the Northern English context, the University of Sheffield has historically been strong in speech technology, with EM-trained HMM systems forming the foundation of Sheffield’s ASR research that contributed to commercial speech products. The University of Manchester’s Machine Learning group applies EM to bioinformatics and medical imaging. NHS digital health initiatives increasingly draw on probabilistic inference tools including EM for patient clustering and missing-data imputation in electronic health records, a domain where the NHS’s scale of longitudinal data provides unique research opportunities.
Future Directions (2026–2030)
Scalable EM for Foundation Models: Research is emerging on applying generalised EM principles to the training of mixture-of-experts foundation models, where the E-step and M-step structure maps naturally onto router training and expert fine-tuning. Fully differentiable, large-scale EM variants are expected to become components of next-generation pretraining pipelines.
Amortised and Meta-Learning EM: Combining amortised inference (as in VAEs) with meta-learning promises EM variants that generalise E-steps across tasks, enabling few-shot density estimation and fast adaptation of mixture models to new datasets.
Quantum EM: Early theoretical work on quantum expectation maximisation explores whether quantum amplitude estimation can accelerate the E-step for large-scale mixture models, though practical demonstrations remain on the research horizon.
Causal EM: Integration of causal identifiability constraints into the EM framework is a growing area, addressing settings where latent confounders must be inferred from observational data while maintaining consistency with a causal graph.
Robust and Heavy-Tailed EM: Replacing Gaussian emission distributions with heavier-tailed alternatives (Student-t, skew-normal) in HMMs and mixture models, combined with EM algorithms extended for these distributions (Alexandridis et al., 2023 Computational Statistics), improves robustness to outliers in clinical and financial time series.
Neuromorphic and Edge EM: Lightweight EM variants for on-device probabilistic inference, enabling unsupervised adaptation of sensor models on edge hardware without cloud connectivity, are relevant to autonomous robotics and IoT applications.
Research & Literature
- Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1–38.
- Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. Annals of Mathematical Statistics, 41(1), 164–171.
- Wu, C. F. J. (1983). On the convergence properties of the EM algorithm. Annals of Statistics, 11(1), 95–103.
- Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in Graphical Models (pp. 355–368). Springer.
- Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine Learning, 37(2), 183–233.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. (Chapter 9: Mixture Models and EM)
- Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. Proceedings of ICLR 2014. arXiv:1312.6114.
- Cappé, O., & Moulines, E. (2009). On-line expectation–maximization algorithm for latent data models. Journal of the Royal Statistical Society: Series B, 71(3), 593–613.
- Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
- McLachlan, G. J., & Krishnan, T. (2008). The EM Algorithm and Extensions (2nd ed.). Wiley.
- Turner, R. E., & Sahani, M. (2010). Two problems with variational expectation maximisation for time-series models. In Inference and Estimation in Probabilistic Time-Series Models. Cambridge University Press. (Gatsby UCL)
- Ghahramani, Z., & Hinton, G. E. (1996). The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, University of Toronto.
- Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.
- Doucet, A., de Freitas, N., & Gordon, N. (Eds.). (2001). Sequential Monte Carlo Methods in Practice. Springer.
- Shumway, R. H., & Stoffer, D. S. (1982). An approach to time series smoothing and forecasting using the EM algorithm. Journal of Time Series Analysis, 3(4), 253–264.
- Benaglia, T., Chauveau, D., Hunter, D. R., & Young, D. (2009). mixtools: An R package for analysing finite mixture models. Journal of Statistical Software, 32(6), 1–29.
- Meng, X.-L., & Rubin, D. B. (1993). Maximum likelihood estimation via the ECM algorithm. Biometrika, 80(2), 267–278.
- Moon, T. K. (1996). The expectation-maximization algorithm. IEEE Signal Processing Magazine, 13(6), 47–60.
- Do, C. B., & Batzoglou, S. (2008). What is the expectation maximization algorithm? Nature Biotechnology, 26(8), 897–899.
- Li, B., & Dewey, C. N. (2011). RSEM: accurate transcript quantification from RNA-Seq data with or without a reference genome. BMC Bioinformatics, 12(1), 323.
- Alexandridis, G., Fotopoulos, S., & Pallikari, F. (2023). An expectation maximisation algorithm for the hidden Markov models with multiparameter Student-t observations. Computational Statistics, 39, 1813–1835.
- Zhang, Y., et al. (2025). Deep generative clustering with VAEs and expectation-maximisation. arXiv:2501.07358.
- Dempster, A. P. (2025). Differentiable expectation-maximisation and applications to Gaussian mixture model optimal transport. arXiv:2509.02109.
- Sheikholeslami, F., et al. (2024). Unified RNA-sequencing model for tissue deconvolution via hierarchical EM. Nature Methods (preprint).
- Royalsocietypublishing.org. (2024). A primer on variational inference for physics-informed deep generative modelling. Philosophical Transactions of the Royal Society A, 382(2471).
- Pritchard, J. K., Stephens, M., & Donnelly, P. (2000). Inference of population structure using multilocus genotype data. Genetics, 155(2), 945–959.
- Figueiredo, M. A. T., & Jain, A. K. (2002). Unsupervised learning of finite mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(3), 381–396.