Model complexity is a measure of the capacity of a machine-learning model to fit varied patterns in data, governed by factors such as the number of parameters, the richness of the hypothesis space, and the flexibility of the functional form. Higher complexity lets a model capture intricate structure but raises the risk of overfitting, while lower complexity risks underfitting. Managing complexity is central to achieving good generalisation on unseen data.
Overview
- Complexity determines where a model sits on the bias-variance spectrum. A model that is too simple has high bias and underfits, failing to capture the signal; a model that is too complex has high variance and overfits, memorising noise. Practitioners control complexity through architecture choices, the number of parameters, and hyperparameters, then constrain effective complexity with regularisation, early stopping, and cross-validation. The guiding intuition, echoing Occam’s razor, is to prefer the simplest model that adequately explains the data.
Key aspects
- Capacity measures: parameter count, VC dimension, and Rademacher complexity quantifying expressiveness.
- Bias-variance trade-off: balancing underfitting against overfitting as complexity rises.
- Effective complexity: regularisation and early stopping reduce a model’s usable capacity below its nominal size.
- Model selection: cross-validation and information criteria choose complexity that generalises best.
- Interpretability cost: greater complexity typically reduces transparency and explainability.
Applications
- Choosing network depth and width in deep learning.
- Tuning regularisation strength to control overfitting.
- Model selection across candidate algorithms via validation curves.
- Trading predictive power against interpretability in regulated decisioning.