Benign Overfitting

Benign overfitting is the phenomenon, observed in heavily overparameterised models such as deep neural networks, where a model fits its training data exactly, including noise, yet still generalises well to unseen data. This contradicts the classical bias-variance trade-off, which predicts that interpolating noise should harm generalisation. It is closely associated with the double-descent risk curve and is a central puzzle in modern statistical learning theory.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:DoubleDescent))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:ImplicitBias))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:MinimumNormInterpolator))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:InterpolationThreshold))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:SpectralDecomposition))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:hasPart ai:OverparameterisedRegime))

Dependency Relationships

SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:requires ai:Overparameterisation))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:requires ai:ImplicitBias))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:requires ai:GradientDescent))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:dependsOn ai:DataCovarianceSpectrum))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:dependsOn ai:StatisticalLearningTheory))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:dependsOn ai:TrainingData))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:dependsOn ai:LossFunction))

Capability Relationships

SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:enables ai:Generalisation))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:enables ai:ZeroTrainingError))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:enables ai:NearOptimalTestError))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:supports ai:DeepLearning))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:supports ai:LargeLanguageModels))

Implementation Relationships

SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:contrastsWith ai:ClassicalBiasVarianceTradeoff))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:contrastsWith ai:Regularisation))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:bridges ai:ModelGovernance))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:Grokking))

Reduction Relationships

SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:reducesTo ai:DoubleDescent))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:reducesTo ai:ImplicitBiasPhenomenon))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:reducesTo ai:OverparameterisationEffect))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:reducesTo ai:MinimumNormInterpolation))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:Grokking))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:ScalingLaws))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:bridges ai:AIGovernance))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:TransferLearning))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:contrastsWith ai:ClassicalOverfitting))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:supports ai:FoundationModels))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:Benchmarks))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:relatedTo ai:Memorisation))
SubClassOf(ai:BenignOverfitting
  ObjectSomeValuesFrom(ai:dependsOn ai:TrainingDataSpectrum))

About

Benign overfitting is one of the most theoretically surprising and practically significant phenomena in modern machine learning, because it reveals that the classical wisdom about overfitting — that fitting noise is harmful — breaks down precisely in the regime where modern large-scale models operate. Classical statistical learning theory, rooted in the bias-variance tradeoff formalised by Geman, Bienenstock, and Doursat (1992) and supported by decades of practical experience with shallow models, predicts a U-shaped test error curve: simple models underfit (high bias, low variance), complex models overfit (low bias, high variance), and the optimal model complexity trades these two errors against each other at an intermediate point. This framework dominated machine learning practice for decades, motivating Regularisation techniques (L1/L2 penalties, Dropout, weight decay), model selection via Cross-Validation, and Early Stopping as the standard toolbox for preventing overfitting.

The empirical cracks in this framework became apparent in the 2010s as practitioners discovered that dramatically over-parameterising neural networks — far beyond the point needed to fit training data — often improved rather than harmed test performance, contradicting classical predictions. Belkin et al. (2019) formalised this observation as the “double descent” phenomenon, documenting that the U-shaped test error curve is actually the left portion of a larger pattern: after the classical rise in error at the interpolation threshold, error descends again in the overparameterised regime, sometimes reaching levels below any classically regularised model. Bartlett, Montanari, and Sly (2020) provided the first rigorous theory of benign overfitting in linear regression, establishing precise conditions on the data covariance spectrum under which the minimum-norm interpolator achieves asymptotically optimal test error despite perfectly fitting noise-corrupted training labels.

The phenomenon is not universal: benign overfitting requires specific conditions, and the same overparameterisation that is benign for one problem structure can be catastrophically harmful for another. The key conditions established by theory are: (1) sufficient overparameterisation, so that the model has enough capacity to separate signal from noise in parameter space; (2) appropriate Implicit Bias from the optimiser — gradient descent on square loss converges to the minimum L2-norm interpolator in linear settings, and analogous inductive biases operate in neural networks; (3) a favourable data covariance structure, specifically that the spectrum of the covariance matrix must have a large effective rank so that noise can be absorbed into many weakly-contributing directions; and (4) the signal-to-noise ratio must be high enough that the principal components capturing signal are not overwhelmed by the distributed noise fitting. When these conditions fail — for example, with low-dimensional data where there are few extra directions to absorb noise — benign overfitting does not occur, and the classical overfitting prediction is correct.

Formal Theory

The canonical formal setting for benign overfitting analysis (following Bartlett et al., 2020) is linear regression with overparameterisation. Let X ∈ ℝ^(n × d) be a design matrix with n samples and d >> n features, and let y = Xβ* + ε be noisy observations where β* is the true signal and ε is i.i.d. noise. The minimum-norm interpolator is β̂ = Xᵀ(X****Xᵀ)⁻¹y, the pseudoinverse solution. The test error of β̂ decomposes into a bias term (how well β̂ recovers β* on the signal subspace) and a variance term (how much noise is amplified). For benign overfitting to hold, the bias must converge to zero (signal recovery succeeds) while the variance also converges to zero (noise amplification vanishes) as n, d → ∞ with d/n → ∞. The key spectral condition (Bartlett et al., 2020, Theorem 2) is that the effective rank k*(Σ) = (∑ᵢ σᵢ)²/∑ᵢ σᵢ² of the covariance matrix Σ must satisfy k*(Σ) >> n, ensuring that noise energy is distributed across many directions each with small individual influence.

The minimum-norm interpolator in overparameterised linear regression corresponds to the maximum-margin classifier (the L2-norm minimiser subject to interpolation constraints), establishing a connection to the SVM and the theory of implicit regularisation. In neural network settings, the analogous role is played by the output of gradient descent on the logistic or cross-entropy loss from random initialisation, which has been shown in a growing body of work (Cao et al., 2022; Frei et al., 2023; Kou et al., 2023) to exhibit benign overfitting for two-layer networks under appropriate data models.

Major Families and Variants

• Linear benign overfitting: the foundational setting (Bartlett et al., 2020; Belkin et al., 2020). Minimum-norm interpolation in overparameterised linear regression achieves optimal rates under spectral conditions. Universality results (Xu, 2025) show these conditions hold broadly across random feature models.

• Kernel benign overfitting: extending the linear analysis to kernel methods via the Neural Tangent Kernel (NTK) regime of infinitely wide networks. Results are more mixed: kernel interpolation can exhibit benign overfitting in specific high-dimensional settings but fails in fixed-dimension settings.

• Two-layer ReLU network benign overfitting: Cao et al. (2022) and Kou et al. (2023) proved benign overfitting for two-layer convolutional networks on noisy data when network width grows appropriately with sample size, extending the linear theory to a genuinely nonlinear setting.

• Benign overfitting in attention mechanisms: a 2024 paper (OpenReview, H4UMsoQrdI) demonstrated that the token selection mechanism of attention achieves benign overfitting — maintaining high generalisation despite fitting label noise — providing theoretical grounding for why Transformer models are robust to training noise.

• Epoch-wise double descent: an ICML 2024 paper explored epoch-wise double descent in noisy data, examining signal separation, large activation, and benign overfitting across training epochs — showing that the benign overfitting regime can be entered and exited during a single training run.

• Benign overfitting in deep linear networks: Boix-Adsera et al. (2022) showed that deep linear networks can benignly overfit when shallow ones do, with depth providing additional directions for noise absorption.

• Grokking as delayed benign overfitting: the Grokking phenomenon — delayed generalisation after training accuracy saturates — can be understood as a dynamic instance of benign overfitting where the model transitions from a memorising regime to a generalising regime through continued optimisation. NeuralGrok (2025) accelerated this transition by identifying and manipulating specific network components.

  Use Cases

• Understanding why large models do not need heavy regularisation: benign overfitting theory explains why Large Language Models trained with relatively modest explicit regularisation on enormous datasets generalise across diverse tasks without overfitting in the classical sense. The overparameterisation provides the spectral structure needed for noise absorption.

• Explaining benchmark evaluation anomalies: models that appear to overfit training data (zero training loss, low validation loss) can still score well on downstream Benchmarks, which was puzzling before benign overfitting theory. The theory predicts when this is expected.

• Model capacity decisions: understanding the interpolation threshold informs decisions about when increasing model capacity helps versus hurts — directly relevant to Neural Architecture Search and Hyperparameter Tuning.

• Calibration and uncertainty: benign overfitting models interpolate training labels including noisy ones, which has implications for Calibration of predicted probabilities — interpolating models may be overconfident on training-similar inputs.

• Transfer Learning and fine-tuning: the implicit bias of gradient descent toward low-norm solutions in overparameterised settings provides theoretical grounding for why fine-tuning Foundation Models with many free parameters does not catastrophically overfit on small downstream datasets.

• Regularisation-free training: practitioners at large-scale training runs (Large Language Models, diffusion models) routinely observe that reducing explicit regularisation in the highly overparameterised regime improves or does not harm generalisation — benign overfitting is the theoretical justification.

  Historical Development

  The study of benign overfitting is best understood as a theoretical reckoning with an empirical fact: that overparameterised deep learning models generalise far better than classical theory predicts. Tracing the intellectual lineage from classical bias-variance theory through the empirical anomalies of the 2010s to the formal proofs of the 2020s reveals a paradigm shift in how the field understands the relationship between model complexity, data fitting, and generalisation.

  Classical foundation: the bias-variance tradeoff (1960s-1990s). The statistical theory of model complexity and generalisation was formalised by Geman, Bienenstock, and Doursat (1992), building on the work of Vapnik and Chervonenkis (1971) on uniform convergence and VC dimension. The core insight of the classical framework is that model complexity determines the bias-variance decomposition of generalisation error: simple models have high bias (they cannot capture complex patterns) but low variance (they are stable across different training sets); complex models have low bias (they can fit complex patterns) but high variance (they are sensitive to noise in the training set). The optimal model complexity trades these two errors against each other at a point that depends on the noise level and sample size. This framework motivated the model selection principle: choose the simplest model that adequately fits the data. Occam’s Razor operationalised as a statistical procedure.

  From this framework followed a rich set of regularisation methods designed to prevent models from becoming too complex: L1 and L2 weight penalties (Ridge and Lasso regression), Early Stopping in gradient descent, Cross-Validation for model selection, and capacity control through architecture design (limited depth, small hidden layers). These methods constituted the standard toolbox of machine learning practitioners from the 1990s through the early 2010s and were firmly grounded in the classical theoretical framework.

  The empirical anomaly: deep learning defeats the prediction (2012-2016). The deep learning revolution, launched by AlexNet in 2012, immediately revealed that massively overparameterised models — models with millions of parameters fitted to tens of thousands of training examples — could generalise far better than classical theory predicted. A ResNet-50, trained on ImageNet with 25 million parameters to classify 1.28 million training images, achieves sub-4% top-5 error on the held-out test set while fitting its training data essentially perfectly. According to classical bias-variance theory, this should be impossible: a model complex enough to perfectly memorise 1.28 million images should have enormous variance and fail catastrophically on new examples. Yet it did not.

  The empirical anomaly was sharpened by Zhang et al. (2017) in their landmark ICLR paper “Understanding deep learning requires rethinking generalisation.” Zhang et al. trained identical neural network architectures on CIFAR-10 with progressively corrupted labels — from 0% label noise (genuine data) to 100% label noise (random labels) — and found that the same overparameterised architecture could memorise 50,000 examples with completely random labels (100% training accuracy, random test accuracy), yet also learn genuine patterns from uncorrupted data (95%+ training accuracy, 90%+ test accuracy). This demonstrated conclusively that overparameterised neural networks have the capacity to memorise any dataset, but do not do so when the data is structured — they exhibit some implicit preference for generalising solutions over memorising solutions. The source of this preference was not explicit Regularisation but something intrinsic to the architecture and optimiser.

  The Zhang et al. finding also demolished classical VC dimension-based bounds on generalisation: a network with billions of parameters and VC dimension in the billions has classical generalisation bounds that are vacuously loose (they predict that generalisation is impossible), yet the network generalises well in practice. The mismatch between classical bounds and empirical reality made clear that new theoretical tools were needed.

  The double descent discovery (2018-2019). Mikhail Belkin and collaborators provided the critical empirical and theoretical bridge between classical and modern understanding with the discovery of the double descent phenomenon. Belkin et al. (2019) documented that the test error of random Fourier feature models, decision trees, and ensemble methods does not monotonically increase in the overparameterised regime — instead, it follows a double descent curve: a U-shape in the classical regime (classical bias-variance tradeoff) followed by a second descent in the overparameterised regime, with error eventually falling below the minimum achievable in the classical regime. The first descent is driven by the classical bias-variance tradeoff. The second descent — the surprising one — is what now demands theoretical explanation.

  The double descent phenomenon revealed that the classical U-shaped test error curve is not the complete picture: it is the left portion of a larger pattern that the classical framework simply could not describe, because it was formulated before overparameterised models were practical. The interpolation threshold — the point at which model complexity crosses zero training error — is the peak between the two descents: at exactly the interpolation threshold, generalisation is at its worst because the model has just enough capacity to fit the data but not enough to do so stably (any small perturbation of the training data produces a dramatically different model). As capacity grows beyond the interpolation threshold, the model can fit the data in many ways, and the implicit bias of the optimiser selects among these solutions in a way that promotes generalisation.

  The theoretical programme (2020-2024). Peter Bartlett, Andrea Montanari, and Allan Sly published the first rigorous proof of benign overfitting in 2020 (PNAS), coining the term and establishing the conditions under which the minimum-norm linear interpolator achieves asymptotically optimal test error despite fitting noise-corrupted training labels. Their result was a landmark not only for its content but for its methodological approach: it used tools from random matrix theory — the study of the spectral properties of high-dimensional random matrices — to characterise precisely the spectral conditions on the data covariance matrix under which benign overfitting holds. This connection to random matrix theory opened a rich vein of subsequent work.

  Bartlett et al.’s result, strictly for linear regression, was rapidly extended by a growing community. Belkin, Hsu, and Xu (2020) extended the analysis to random feature models (the kernel approximation setting). Hastie, Montanari, Rosset, and Tibshirani (2022) analysed high-dimensional ridge regression and characterised the double descent curve precisely in the random design setting. The first extension to genuinely nonlinear models came from Cao et al. (2022), who proved benign overfitting for two-layer convolutional neural networks on noisy linear data. This was followed by proofs for two-layer ReLU networks (Frei et al., 2023; Kou et al., 2023), single-head attention mechanisms (2024), and deep linear networks (Boix-Adsera et al., 2022). By 2025, the theory had moved substantially from linear-only proofs toward nonlinear architectures, though deep networks with realistic complexity remain beyond current analytical reach.

  Grokking as a dynamic analogue (2022-2025). The Grokking phenomenon, documented by Power et al. (2022) at OpenAI, introduced a temporal dimension to the benign overfitting discussion. In grokking, a model trained on arithmetic tasks first achieves perfect training accuracy (memorisation / interpolation phase), then after continued training well past convergence, transitions to near-perfect generalisation (generalisation phase). This delayed generalisation — sometimes requiring 100x the training steps to achieve generalisation as memorisation — revealed that benign overfitting is not just a property of the final model but can be a transient that emerges through continued optimisation. The grokking phenomenon has been linked theoretically to phase transitions in the implicit bias of gradient descent, suggesting that the interpolating memorisation solution and the generalising minimum-norm solution are distinct attractors that gradient descent navigates between under continued training. NeuralGrok (2025) dramatically accelerated this transition by identifying the specific network components responsible for generalisation and manipulating their gradient flow.

  Academic Context

  The study of benign overfitting connects the statistical learning theory tradition (Vapnik and Chervonenkis, 1971; Rademacher complexity; VC dimension) with modern deep learning practice through a series of landmark papers:

• Belkin, Hsu, Ma, and Mandal (2019) — “Reconciling modern machine-learning practice and the classical bias–variance trade-off” (PNAS 116(32):15849–15854). The seminal empirical and theoretical demonstration of double descent across a range of models. Introduced the term “interpolation threshold” and documented the two-regime structure: classical U-curve followed by descent in the overparameterised regime.

• Bartlett, Montanari, and Sly (2020) — “Benign overfitting in linear regression” (PNAS 117(48):30063–30070). Coined the term “benign overfitting” and provided the first rigorous proof: matching upper and lower bounds on the minimum-norm interpolator’s test error under spectral conditions on the data covariance.

• Belkin, Hsu, and Xu (2020) — “Two models of double descent for weak features” (SIAM Journal on Mathematics of Data Science 2(4):1167–1180). Extended the analysis to random feature models.

• Mei and Montanari (2019) — “The generalisation error of random features regression: Precise asymptotics and the double descent curve” (Communications on Pure and Applied Mathematics). Derived the double descent curve in the random feature regression setting with precise asymptotics.

• Cao, Chen, Belkin, and Gu (2022) — “Benign overfitting in two-layer convolutional neural networks” (NeurIPS 2022, arXiv:2202.06526). Extended benign overfitting proofs from linear to two-layer convolutional networks.

• Zhang, Bengio, Hardt, Recht, and Vinyals (2017) — “Understanding deep learning requires rethinking generalisation” (ICLR 2017). The empirical observation that neural networks can memorise random labels yet still generalise on structured data that motivated the benign overfitting line of inquiry.

• Frei, Chatterji, and Bartlett (2023) — “Benign overfitting in linear classifiers and leaky ReLU networks from KKT conditions for margin maximisation” (arXiv:2303.01462, NeurIPS 2023 area). Proved benign overfitting for leaky ReLU networks through margin maximisation analysis.

• Kou, Chen, Chen, and Gu (2023) — “Benign overfitting for two-layer ReLU convolutional neural networks” (ICML 2023). Proved benign overfitting on noisy linear data for gradient descent trained two-layer ReLU CNNs.

• Benign Overfitting in Token Selection of Attention Mechanism (2024) (OpenReview H4UMsoQrdI). Extended the theory to the attention mechanism, showing that token selection achieves benign overfitting.

• Deep Exploration of Epoch-wise Double Descent (arXiv:2601.08316, 2025). Analysed signal separation and large activation effects causing epoch-wise benign overfitting in noisy data.

• NeurIPS 2023 Tutorial — Reconsidering Overfitting in the Age of Overparameterised Models (ETH Zurich, Statistical Machine Learning group). Comprehensive tutorial covering the state of the art in benign overfitting and double descent as of 2023.

  Current Landscape (2026)

  By 2026, benign overfitting is a well-established empirical phenomenon whose theoretical explanation remains an active research frontier. The linear regression case is theoretically settled (Bartlett et al., 2020); two-layer neural networks with specific architectures and data models have rigorous proofs (Cao et al., 2022; Frei et al., 2023; Kou et al., 2023); the attention mechanism has been analysed (2024). The outstanding open problems concern: (1) whether benign overfitting holds for deep networks beyond the lazy training regime; (2) the role of architecture, skip connections, and normalisation layers; (3) characterising the precise spectral and distributional conditions on real-world data under which benign overfitting is guaranteed.

  The connection to Grokking — first documented by Power et al. (2022) at OpenAI — has been a major research direction in 2024-2025. NeuralGrok (2025) reduced grokking time by orders of magnitude by identifying specific components responsible for generalisation. The unification of grokking, double descent, and benign overfitting into a coherent theoretical framework (two-phase dynamics: kernel/memorisation regime followed by feature learning regime) is an active area across Berkeley, MIT, ETH Zurich, and Princeton.

  Large language model practice increasingly draws on benign overfitting theory to justify training without heavy regularisation and to understand why scaling model size and data simultaneously (the Chinchilla scaling law regime) produces robust generalisation. The implicit bias toward minimum-norm solutions provides a theoretical account for why overparameterised models trained with Stochastic Gradient Descent on web-scale data generalise without explicit Regularisation.

  Practically, benign overfitting informs MLOps decisions about when to add regularisation versus when to scale model capacity — a question now routinely asked in production training runs for foundation models at organisations including DeepMind, Anthropic, OpenAI, Meta, and Mistral.

  UK Context

  The United Kingdom has contributed to the theoretical and applied study of benign overfitting primarily through:

• University of Cambridge (Statistical Laboratory and Department of Engineering): Cambridge researchers have contributed to statistical learning theory and overparameterisation analysis; the Statistical Laboratory has a strong tradition in random matrix theory (relevant to the spectral analysis underlying benign overfitting proofs).

• University of Edinburgh (School of Informatics): Edinburgh’s machine learning group works on generalisation theory and optimisation dynamics; the EPCC supercomputer facility (£750M national investment, planned 2027) will support large-scale empirical studies of generalisation in overparameterised models.

• University College London (Gatsby Computational Neuroscience Unit and Dept of Computer Science): UCL has strong research programmes in statistical learning theory and deep learning theory relevant to overparameterisation and implicit bias.

• Alan Turing Institute: the ATI’s statistical machine learning programme supports theoretical research into generalisation and overparameterisation across UK universities.

• DeepMind (London): as a frontier AI lab, DeepMind trains highly overparameterised models daily; benign overfitting theory directly informs its training and evaluation practices for Gemini and AlphaCode families.

• Northern England: Manchester (National AI Research Institute), Leeds (medical AI), Sheffield (NLP for manufacturing) conduct applied deep learning where understanding generalisation without heavy regularisation is practically important for data-scarce domains.

• UK AISI: the AI Security Institute’s evaluations of frontier models implicitly depend on understanding benign overfitting — since frontier models are trained in the interpolating regime, their generalisation properties (and thus benchmark performance) can only be interpreted through this theoretical lens.

  Challenges and Open Problems

  Despite significant theoretical progress, benign overfitting raises a set of deep and practically consequential open problems that define the current research frontier:

  The gap between linear and deep network theory: the rigorous theory of benign overfitting has been established for linear regression (Bartlett et al., 2020), kernel methods, and two-layer neural networks (Cao et al., 2022; Kou et al., 2023). However, practical Deep Learning systems are not two-layer networks — they are deep architectures with residual connections, layer normalisation, attention heads, and complex parameter sharing. Whether the spectral conditions and implicit bias arguments that underpin two-layer proofs extend to deep networks with realistic architectures remains an open problem. The NTK (Neural Tangent Kernel) regime, where network width goes to infinity and training dynamics linearise, provides one pathway but applies only in a regime distant from practical training.

  Implicit bias characterisation beyond minimum L2-norm: the minimum-norm interpolator analysis relies on gradient descent converging to the pseudoinverse solution in linear models. In nonlinear models, the implicit bias of gradient descent is far less well understood: it depends on architecture, activation function, initialisation, batch size, and learning rate schedule in ways that have not been fully characterised. Understanding which interpolating solutions gradient descent finds in practice, and whether these solutions have the spectral structure needed for benign overfitting, is a central theoretical gap.

  Signal-to-noise ratio conditions: the benign overfitting result is not unconditional — it requires the data’s signal-to-noise ratio to be sufficiently high and the effective rank of the covariance to be sufficiently large relative to sample size. In practice, assessing whether these conditions hold for a given dataset and model size is non-trivial. Guidelines for when to expect benign overfitting versus harmful overfitting — and how to test for it empirically — would be of significant practical value.

  Calibration and uncertainty: benign overfitting models interpolate their training data exactly, including noisy or mislabelled examples. This raises concerns about calibration — whether the model’s confidence in its predictions accurately reflects the probability of being correct. An interpolating model that assigns high confidence to training examples, including noisy ones, may be systematically miscalibrated in ways that are not visible from test accuracy alone. The interaction between benign overfitting and Calibration is an important unexplored area with direct safety implications.

  Continual learning and catastrophic forgetting: the minimum-norm interpolator concentrates model capacity on the training distribution; when the distribution shifts (as in continual learning or deployment under distributional shift), whether this specialisation persists or disrupts generalisation is unclear. Understanding whether benign overfitting provides any protection against catastrophic forgetting — or conversely, whether it exacerbates it — is relevant to Foundation Models fine-tuning and continual learning research.

  The grokking connection: grokking — the delayed generalisation phenomenon — shares structural similarities with benign overfitting but operates at a different temporal scale. In grokking, the model first memorises (interpolates noisily) then later transitions to genuine generalisation after continued training. Whether this transition can be understood as a phase transition in the implicit bias of the optimiser — shifting from a memorising solution to a minimum-norm generalising solution — and whether benign overfitting theory can be used to predict when grokking will or will not occur, are active research questions in 2026.

  Relationship to Adjacent Phenomena

  Benign overfitting sits within a cluster of related phenomena concerning the generalisation behaviour of overparameterised models. Understanding how these phenomena connect and differ clarifies the significance of benign overfitting for theory and practice:

  Classical overfitting vs. benign overfitting: classical overfitting — the failure mode that Regularisation is designed to prevent — occurs when a model fits noise in its training data and, as a result, generalises poorly to new data. Benign overfitting involves the same fitting of noise (interpolation of training labels including noisy ones) but does not result in poor generalisation. The mechanistic difference is the implicit bias of the optimiser and the spectral structure of the data: in the classical overfitting regime, the model has just enough capacity to interpolate the data in a way that concentrates variance in directions that affect test performance; in the benign overfitting regime, the model has so much extra capacity that noise is absorbed into directions that contribute negligibly to test predictions.

  Double descent vs. benign overfitting: double descent is the descriptive name for the U-shaped-then-descending test error curve observed as model capacity grows through the interpolation threshold. Benign overfitting is the theoretical explanation for why the second descent occurs — why the overparameterised interpolating model generalises well. Double descent is the empirical phenomenon; benign overfitting is the mechanistic account.

  Grokking vs. benign overfitting: grokking is a temporal phenomenon — a model transitions from memorisation to generalisation through continued training — while benign overfitting describes the property of a final model that simultaneously interpolates training data and generalises. Both involve the tension between fitting training labels (including noise) and generalising to new data, but they differ in dynamics. Grokking’s generalisation typically involves learning a more structured, lower-complexity representation than the memorising solution — consistent with the implicit bias interpretation. NeuralGrok (2025) suggests that identifying and strengthening the network components responsible for generalisation can force the transition faster, implying a continuous deformation from memorising to generalising solutions rather than a sharp phase transition.

  Memorisation vs. generalisation in LLMs: large language models memorise a substantial fraction of their training data verbatim (Carlini et al., 2021), yet generalise across diverse tasks they have never seen in training. This coexistence of memorisation and generalisation in the same model is a macroscopic version of benign overfitting: the model interpolates its training corpus (memorising some examples exactly) while simultaneously learning representations that generalise. The interpolation threshold for a 100-billion-parameter model trained on 10 trillion tokens is far below the training data size, placing LLMs firmly in the overparameterised benign overfitting regime.

  Implicit regularisation vs. explicit regularisation: explicit Regularisation methods (L1/L2 penalties, dropout, weight decay, early stopping, data augmentation) are designed to prevent models from fitting noise by constraining the solution space. Implicit regularisation is the inductive bias introduced by the optimiser and architecture independently of explicit penalties — in particular, the bias of gradient descent toward minimum-norm solutions in overparameterised linear models. Benign overfitting theory reveals that implicit regularisation can be sufficient for generalisation even without explicit regularisation, explaining the empirical observation that aggressively regularised large models often perform worse than lightly regularised ones in the deeply overparameterised regime.

  The bias-variance tradeoff revisited: benign overfitting does not eliminate the bias-variance tradeoff — it modifies it. In the overparameterised benign regime, both bias and variance can decrease simultaneously as model capacity increases, because the excess capacity provides directions for absorbing noise without affecting signal. The classical tradeoff (reducing bias increases variance) holds in the underparameterised regime but is transcended in the deeply overparameterised regime where the spectral conditions for benign overfitting are satisfied. This suggests that the classical tradeoff should be understood as a special case of a more general relationship between model capacity and generalisation, valid only when capacity is constrained to be near-optimal.

  Transfer learning and fine-tuning in the benign overfitting framework: the success of fine-tuning pretrained Foundation Models on small downstream datasets — a practice that would seem to guarantee Overfitting by classical standards — can be understood through the benign overfitting lens. A 7-billion-parameter model fine-tuned on 1,000 downstream examples is in the deeply overparameterised regime by any classical standard. However, if the fine-tuning uses gradient descent and the downstream data has appropriate spectral structure (which it may due to the shared structure of language and knowledge across tasks), the implicit bias of the optimiser may select interpolating solutions that are also generalising solutions. LoRA (low-rank adaptation) fine-tuning can be understood as explicitly constraining the optimisation to a low-dimensional subspace, trading some implicit bias for explicit parameterisation — a middle ground between classical regularisation and full benign overfitting.

  Practical Implications for Deep Learning Practitioners

  The benign overfitting literature has direct implications for practitioner decisions about model training, regularisation, and capacity planning, even where formal theoretical proofs for practical architectures are still pending:

  When to regularise less: the central practical implication of benign overfitting is that aggressively regularising overparameterised models can be counterproductive. If a model has many more parameters than data points and is trained with gradient descent, the implicit bias toward low-norm interpolating solutions may provide sufficient effective regularisation. Adding explicit L2 regularisation (weight decay) in this regime can push the model away from the minimum-norm solution and toward solutions with worse generalisation than the unregularised interpolator. This insight is consistent with empirical observations from large-scale pretraining: frontier language models use minimal weight decay and rely primarily on the implicit bias of AdamW optimisation.

  Model scaling as a generalisation tool: in the overparameterised regime, increasing model capacity typically helps rather than harms generalisation — counter to classical intuition. This is because more parameters provide more directions in parameter space for the model to distribute noise fitting, improving the spectral separation between signal and noise components. The empirical observation that scaling Large Language Models consistently improves generalisation across diverse tasks — the foundation of scaling laws research — is consistent with benign overfitting theory operating in the deeply overparameterised regime.

  Data quality versus data quantity trade-offs: benign overfitting theory emphasises that the condition for generalisation depends not just on noise level but on the spectral structure of the data covariance — the effective rank relative to sample size. Adding more data increases n while (ideally) preserving or improving the effective rank, strengthening the conditions for benign overfitting. Data Curation that removes mislabelled or corrupted examples reduces noise but also reduces dataset size; the theory provides insight into when curation hurts (by reducing effective rank or n below thresholds) versus helps (by reducing σ² below thresholds needed for benign conditions). In practice, high-quality data at scale is the dominant strategy for frontier models, consistent with theory.

  Evaluation on Benchmarks: understanding benign overfitting is essential for correctly interpreting benchmark results of overparameterised models. A model that achieves zero training loss does not necessarily exhibit harmful overfitting; its generalisation should be assessed by test performance, not by the presence of interpolation per se. Conversely, a model that overfits a benchmark (achieves near-ceiling performance) may be exhibiting a form of Goodhart’s Law — optimising the metric without improving underlying capability — which is conceptually distinct from benign overfitting in the theoretical sense. Practitioners need to distinguish benchmark contamination (training data leakage inflating test scores) from genuine benign overfitting (interpolating noise while correctly generalising on fresh test data).

  Connections to Scaling Laws and Frontier Model Training

  The theoretical framework of benign overfitting provides important conceptual grounding for understanding scaling laws — the empirical relationships between model size, data size, training compute, and model performance that govern frontier AI development:

  Chinchilla scaling laws and benign overfitting: the Chinchilla scaling laws (Hoffmann et al., 2022, “Training Compute-Optimal Large Language Models”) established that for a given compute budget, models should be trained on approximately 20 tokens per parameter rather than the previously favoured regime of training very large models on relatively few tokens. The Chinchilla regime sits in the moderately overparameterised domain relative to training data, where the model has more parameters than strictly needed to fit the data. Benign overfitting theory provides a partial explanation for why this regime generalises well: the implicit bias of gradient descent selects minimum-norm interpolating solutions that distribute parameter space efficiently, provided overparameterisation is sufficient. The post-Chinchilla trend toward even longer training runs (GPT-4, Llama 3, Claude 3) is pushing models further into the overparameterised regime as token counts grow while parameter counts remain fixed, consistent with increasing the overparameterisation ratio d/n.

  Scaling without explicit regularisation: frontier model training uses minimal explicit regularisation — typically small weight decay (e.g. 0.1) and no L1 penalties, dropout, or explicit capacity control. This regime is only coherent if implicit regularisation from the optimiser and architecture provides sufficient generalisation control, which is precisely what benign overfitting theory predicts. The AdamW optimiser’s decoupled weight decay is often described as L2 regularisation, but in the deeply overparameterised regime its effect is more accurately understood as a slight perturbation of the implicit minimum-norm bias rather than as a classical capacity-controlling regulariser.

  Data quality in the benign overfitting framework: benign overfitting theory implies that the spectral structure of training data matters as much as its volume. Data with high effective rank — diverse, covering many dimensions of variation — provides more directions in parameter space for the model to absorb noise, improving the conditions for benign overfitting. This provides theoretical grounding for the empirical observation that data quality (diversity, coverage, accuracy) is often more important than data quantity for frontier model performance. Data curation practices that remove duplicates (which reduce effective rank by concentrating probability mass on repeated patterns), filter low-quality content, and balance coverage across domains can be understood as improving the spectral conditions for benign generalisation.

  Emergent capabilities and phase transitions: the documented emergence of qualitatively new capabilities at specific scale thresholds — in-context learning emerging in GPT-3 (175B parameters), chain-of-thought reasoning emerging in PaLM (540B), instruction following emerging in models fine-tuned with RLHF — may have connections to benign overfitting phase transitions. As model size and data scale push the overparameterisation ratio higher and change the spectral structure of the learned representations, new capability regimes may become accessible. The theoretical understanding of why emergent capabilities appear at specific scale thresholds remains incomplete, but benign overfitting theory suggests that transitions in the implicit bias regime — from memorising to generalising solutions — are one candidate mechanism.

  LoRA and efficient fine-tuning: Low-Rank Adaptation (LoRA) restricts fine-tuning updates to a low-dimensional subspace by constraining the parameter update matrix to be the product of two low-rank matrices. This can be understood as explicitly constraining the optimisation to a low-rank manifold, trading some implicit bias (full gradient descent in parameter space) for parameterisation constraints. In the benign overfitting framework, LoRA effectively reduces the overparameterisation ratio for the fine-tuning stage while preserving the pretrained model’s minimum-norm structure. Whether this reduction in overparameterisation helps or hurts depends on the downstream data’s spectral structure: for small downstream datasets with low effective rank, LoRA’s constraint may improve generalisation by preventing noise absorption in excessive directions; for larger datasets, full fine-tuning may be better.

  Key Terminology

• Interpolation threshold: the parameter count at which a model first achieves exactly zero training loss on the training set. At the interpolation threshold, the model just barely fits the data; adding more parameters pushes into the overparameterised regime.

• Minimum-norm interpolator: among all models that perfectly fit (interpolate) the training data, the minimum-norm interpolator is the one with the smallest parameter norm. Gradient descent on overparameterised linear models converges to the minimum L2-norm interpolator (the pseudoinverse solution), which is the central object of benign overfitting theory.

• Effective rank: a measure of how many dimensions of a covariance matrix contribute substantially to its variance — formally, the ratio (∑ᵢ σᵢ)²/(∑ᵢ σᵢ²) where σᵢ are eigenvalues. High effective rank (many dimensions with similar variance) is the key condition for benign overfitting: it ensures that noise is distributed across many weakly-contributing directions.

• Implicit bias: the inductive bias introduced by the optimiser (e.g. gradient descent) and architecture, which causes training to converge to a particular type of solution among all solutions that achieve zero training loss. Implicit bias toward minimum-norm or maximum-margin solutions is the mechanism through which benign overfitting occurs.

• Double descent: the phenomenon where test error follows a U-shape in the classical regime (few parameters), peaks at the interpolation threshold, then decreases again in the overparameterised regime — forming a double-descent curve. Benign overfitting is the explanation for why the second descent occurs.

• Signal-to-noise ratio (SNR): the ratio of the magnitude of the true signal β* to the noise variance σ². High SNR is necessary for benign overfitting: if noise dominates, even the minimum-norm interpolator cannot recover the signal.

• Grokking: the delayed generalisation phenomenon in which a neural network first memorises training data (100% training accuracy, random test accuracy) then, after continued training, transitions to genuine generalisation (100% training accuracy, high test accuracy). Conceptually related to benign overfitting but operating as a temporal phase transition rather than a static property.

• Lazy training / NTK regime: the training regime, relevant for infinitely wide networks, where weights change so little during training that the network behaves like a linear kernel model. Benign overfitting results in the NTK regime are better understood than in the feature learning regime where weights change substantially.

• Feature learning regime: the training regime, relevant for practical finite-width networks, where weights change substantially during training, causing the network’s internal representations to adapt to the data. Benign overfitting theory in the feature learning regime is less mature and is an active research frontier.

• Overparameterisation ratio: the ratio of parameters to data points (d/n in linear models). As this ratio grows beyond 1 (past the interpolation threshold), the model enters the overparameterised regime where benign overfitting can occur.

  Safety and Governance Implications

  Benign overfitting, as a theoretical framework that explains why overparameterised models generalise without explicit regularisation, has increasingly important implications for the safety and governance of deployed AI systems. Regulators, safety researchers, and AI Governance frameworks are beginning to grapple with what benign overfitting means for the predictability, interpretability, and control of frontier AI systems:

  Predictability of overparameterised models: benign overfitting theory shows that overparameterised models converge to minimum-norm interpolating solutions whose generalisation depends on the spectral structure of the training data. This dependence on spectral structure means that the model’s behaviour on out-of-distribution inputs — inputs that differ from training data in ways that may not be visible from test accuracy — is theoretically hard to predict. A model that benignly overfits a training distribution may exhibit unexpected behaviour when deployed on distributions that violate the spectral conditions under which benign overfitting was established. Safety cases for frontier AI systems need to address this uncertainty.

  Contamination and benign overfitting on benchmark data: a subtlety in interpreting benchmark performance of overparameterised models is that benchmark contamination and benign overfitting produce different signatures. A model that has seen benchmark test items in training (contamination) will interpolate those items — achieving perfect performance on them — but this is not benign overfitting in the theoretical sense because the interpolated examples are not noise-corrupted training labels but exact copies of the test items. Distinguishing contamination-driven score inflation from genuine benign generalisation requires comparing performance on contaminated vs. uncontaminated benchmark items, a detection task that is inherently difficult without knowledge of the model’s training data.

  Memorisation and privacy: the minimum-norm interpolator for a model with sufficient overparameterisation passes through every training example, including sensitive or private data. Carlini et al. (2021) demonstrated that frontier language models can be prompted to reproduce training data verbatim, including personally identifiable information. Benign overfitting theory explains why this happens: interpolation is a necessary property of the minimum-norm solution in the overparameterised regime. Preventing memorisation in the benign overfitting regime requires either: reducing overparameterisation (classically expensive); adding explicit noise or differential privacy constraints during training; or using data deduplication to reduce the effective weight of sensitive examples. Differential privacy in neural network training (DPSGD) can be understood as adding a noise floor that prevents the model from interpolating individual training examples at the cost of reduced generalisation, trading off benign overfitting for privacy.

  The EU AI Act and evaluation of interpolating models: the EU AI Act’s conformity assessment requirements for high-risk AI systems include evaluation of accuracy, robustness, and non-discrimination on training, validation, and test data. For systems trained in the benign overfitting regime, the traditional split between training performance (interpolating, perfect accuracy) and test performance (the quantity of interest) must be clearly communicated to conformity assessment bodies. Claiming that a model “overfits” its training data — in the sense of achieving zero training loss — may be misinterpreted as a failure mode when in the deeply overparameterised regime it is consistent with optimal test generalisation. AI Governance frameworks need to develop vocabulary and assessment criteria that account for the benign overfitting regime.

  AISI evaluation of frontier models: the UK AI Security Institute evaluates frontier AI systems including models trained in the deeply overparameterised regime. Understanding benign overfitting is relevant to AISI’s evaluation work because: (1) models that achieve perfect or near-perfect performance on safety evaluation training sets may still exhibit the targeted safety properties if the conditions for benign overfitting hold; (2) conversely, models that appear well-calibrated on safety benchmarks due to benign overfitting may fail unexpectedly on out-of-distribution safety scenarios where the spectral conditions break down; (3) evaluation of memorisation and data reproduction (a privacy and safety concern) requires understanding the interpolation structure of the model’s solution.

  Future Directions (2026-2030)

• Deep network theory: extending rigorous benign overfitting proofs from two-layer and linear networks to deep networks with realistic architectures (residual connections, layer normalisation, mixture-of-experts), which remain beyond current analytical techniques. Kernel-based and random feature approximations provide partial progress but do not capture the feature learning dynamics of practical deep networks.

• Stochastic gradient descent beyond lazy training: the minimum-norm interpolator analysis covers the NTK/lazy training regime; understanding benign overfitting in the feature learning regime — where representations change substantially during training — requires new theoretical tools beyond NTK analysis. Random matrix theory and mean-field theory are promising approaches being explored by groups at MIT, Princeton, and ETH Zurich.

• Benign overfitting in reasoning models: as Chain-of-Thought and Reinforcement Learning from Human Feedback training regimes produce models that over-optimise on training distributions, understanding when this is benign versus harmful to generalisation is a critical open question. Process reward models (PRMs) and verifiable reasoning tasks create new versions of the interpolation problem at the reasoning level.

• Data distribution characterisation: practical guidelines for when real-world text and multimodal data satisfies the spectral conditions for benign overfitting would directly inform model capacity decisions. Developing tools to estimate effective rank and signal-to-noise ratios from pre-training data statistics — and relating these to optimal model size — could refine the scaling law literature significantly.

• Calibration and uncertainty in interpolating models: benign overfitting models perfectly interpolate noisy training labels, raising questions about whether their confidence is well-calibrated. Bridging benign overfitting theory with Calibration research and uncertainty quantification (conformal prediction, Bayesian deep learning) is an emerging direction with direct safety relevance.

• Catastrophic forgetting and continual learning: whether benign overfitting provides any protection against catastrophic forgetting when models are continually updated on new data is an open question relevant to Foundation Models deployment. The minimum-norm interpolator’s structure may make it particularly susceptible to forgetting since its solution is optimally compressed to the training distribution.

• Regulatory implications: as regulators demand interpretable and predictable AI behaviour, the conditions under which overfitting is benign versus harmful need to be characterised precisely — informing AI Governance frameworks, Model Governance standards, and safety cases for frontier AI deployment under the EU AI Act and UK AI legislation.

  Research and Literature

  1. Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS, 116(32), 15849–15854. https://doi.org/10.1073/pnas.1903070116
  2. Bartlett, P. L., Montanari, A., & Sly, A. (2020). Benign overfitting in linear regression. PNAS, 117(48), 30063–30070. https://www.pnas.org/doi/10.1073/pnas.1907378117
  3. Belkin, M., Hsu, D., & Xu, J. (2020). Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4), 1167–1180.
  4. Mei, S., & Montanari, A. (2022). The generalisation error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4), 667–766.
  5. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding deep learning requires rethinking generalisation. ICLR 2017. https://arxiv.org/abs/1611.03530
  6. Hastie, T., Montanari, A., Rosset, S., & Tibshirani, R. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. Annals of Statistics, 50(2), 949–986.
  7. Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. NeurIPS 2022. https://arxiv.org/abs/2202.06526
  8. Frei, S., Chatterji, N. S., & Bartlett, P. L. (2023). Benign overfitting in linear classifiers and leaky ReLU networks from KKT conditions for margin maximisation. arXiv:2303.01462. https://arxiv.org/abs/2303.01462
  9. Kou, Y., Chen, Z., Chen, Y., & Gu, Q. (2023). Benign overfitting for two-layer ReLU convolutional neural networks. ICML 2023. https://arxiv.org/abs/2303.04145
  10. Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. ICML 2023. https://arxiv.org/abs/2202.05928
  11. Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalisation beyond overfitting on small algorithmic datasets. ICLR 2022 Workshop. https://arxiv.org/abs/2201.02177
  12. Boix-Adsera, E., Littwin, E., Abbe, E., & Barak, B. (2022). Deep linear networks can benignly overfit when shallow ones do. arXiv:2209.09315. https://arxiv.org/abs/2209.09315
  13. Benign Overfitting in Token Selection of Attention Mechanism. (2024). OpenReview. https://openreview.net/forum?id=H4UMsoQrdI
  14. Trained Transformer Classifiers Generalise and Exhibit Benign Overfitting In-Context. (2024). arXiv:2410.01774. https://arxiv.org/abs/2410.01774
  15. Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfitting. (2025). arXiv:2601.08316. https://arxiv.org/abs/2601.08316
  16. A Classical View on Benign Overfitting: The Role of … (2025). arXiv:2505.11621. https://arxiv.org/abs/2505.11621
  17. NeurIPS 2023 Tutorial: Reconsidering Overfitting in the Age of Overparameterised Models. ETH Zurich Statistical Machine Learning. https://sml.inf.ethz.ch/gml23/neuripstut-blank.html
  18. Xu, J. (2025). Universality of benign overfitting in binary linear classification. arXiv:2501.10538. https://arxiv.org/abs/2501.10538
  19. Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1–58.
  20. Vapnik, V. N., & Chervonenkis, A. Y. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications, 16(2), 264–280.
  21. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. (Chapter 7: Regularisation for Deep Learning)
  22. NeuralGrok: Accelerate Grokking by Neural Gradient Transformation. (2025). arXiv:2504.17243. https://arxiv.org/abs/2504.17243
  23. The Implicit Bias of Benign Overfitting. (2022). arXiv:2201.11489. https://arxiv.org/abs/2201.11489
  24. Benign Overfitting for Regression with Trained Two-Layer ReLU Networks. (2024). arXiv:2410.06191. https://arxiv.org/abs/2410.06191
  25. Farewell to the Bias-Variance Trade-off? An Overview of the Theory of Overparameterised Machine Learning. (2021). arXiv:2109.02355. https://arxiv.org/abs/2109.02355
  26. Bartlett, P. L., & Mendelson, S. (2002). Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3, 463–482.
  27. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929–1958.

Provenance