Benchmarks

Benchmarks are standardised tasks, datasets, or workloads used to measure and compare the capabilities or performance of systems, models, or components under controlled, reproducible conditions. In artificial intelligence they encompass curated evaluation suites that probe language understanding, mathematical reasoning, coding, and multimodal perception to produce comparable scores across model generations and research groups. In computing, robotics, and hardware engineering, benchmarks quantify throughput, latency, accuracy, and energy efficiency against fixed reference workloads. Benchmark results are published via leaderboards and model cards to support reproducible science, informed procurement, and regulatory accountability.

In Plain Terms

• Standard tests that AI models are all put through so their results can be compared fairly. Like a common exam, they give each model a score on the same tasks, letting you see which is stronger at reasoning, coding and so on.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:EvaluationMetric))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:GroundTruthLabels))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:HeldOutTestSet))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:Leaderboard))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:AnnotationPipeline))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:ScoringProtocol))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:hasPart ai:TaskFormulation))

Dependency Relationships

SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:requires ai:DataCuration))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:requires ai:AnnotationPipelines))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:requires ai:StatisticalTesting))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:dependsOn ai:HeldOutTestSets))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:dependsOn ai:EvaluationProtocol))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:uses ai:EvaluationMetric))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:uses ai:Datasets))

Capability Relationships

SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:enables ai:ModelEvaluationResults))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:enables ai:Reproducibility))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:enables ai:ModelSelection))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:enables ai:ModelComparison))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:supports ai:AIGovernance))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:supports ai:MLOps))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:supports ai:AISafety))

Implementation Relationships

SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:implements ai:EvaluationProtocol))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:implements ai:ScoringFunction))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:bridges ai:RegulatoryCompliance))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:standardizedBy ai:MLCommons))

Reduction Relationships

SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:reducesTo ai:PerformanceScore))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:reducesTo ai:EvaluationMetric))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:reducesTo ai:CapabilityRanking))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:reducesTo ai:LeaderboardScore))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:reducesTo ai:AccuracyScore))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:contrastsWith ai:HumanEvaluation))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:relatedTo ai:BenignOverfitting))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:relatedTo ai:GoodhartLaw))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:supports ai:ResponsibleAI))
SubClassOf(ai:Benchmarks
  ObjectSomeValuesFrom(ai:bridges ai:AIRegulation))

About

Benchmarks are the empirical foundation on which the scientific credibility of Machine Learning and Artificial Intelligence research rests. Without standardised evaluation protocols, claims of model improvement are unfalsifiable: two research groups using different datasets, metrics, or experimental conditions cannot compare their results in any meaningful way. The benchmark disciplines this chaos by fixing the task formulation, the test split, the scoring function, and the inference protocol, transforming subjective claims of superiority into reproducible, comparable measurements.

The history of benchmarks in computing predates machine learning. The Standard Performance Evaluation Corporation (SPEC) introduced CPU benchmark suites in 1988 to provide hardware vendors and buyers with comparable performance data. The Transaction Processing Performance Council (TPC) standardised database workloads in the same era. This tradition migrated into AI via pattern recognition competitions — the pioneering MNIST digit classification benchmark (1998) and the Pascal VOC challenge (2005) — before achieving landmark status with ImageNet (Deng et al., 2009; Russakovsky et al., 2015), whose annual Large Scale Visual Recognition Challenge (ILSVRC) drove convolutional neural network development from 2010 through to 2017. The NLP community developed analogous shared tasks through SemEval, CoNLL shared tasks, and GLUE (Wang et al., 2018), creating comparable evaluation conditions across machine translation, named-entity recognition, and reading comprehension.

The Large Language Models era, inaugurated by GPT-3 (Brown et al., 2020) and BERT (Devlin et al., 2019), produced a proliferation of capability benchmarks probing emergent properties of scale. MMLU (Hendrycks et al., 2020) covered 57 academic subjects; BIG-Bench (Srivastava et al., 2022) enumerated 204 tasks beyond few-shot prompting; HELM (Liang et al., 2022) provided holistic multi-metric evaluation across 42 scenarios. The speed of model capability growth repeatedly saturated these suites within 12-18 months, producing the cycle of benchmark saturation that now characterises the field. By 2026, MMLU, GSM8K, HumanEval, and HellaSwag are effectively saturated — frontier models cluster above 90% on all of them — driving adoption of harder replacements: MMLU-Pro, GPQA Diamond (graduate-level science reasoning), HLE (Humanity’s Last Exam), FrontierMath, ARC-AGI-2, and LiveCodeBench.

Components / Architecture

A fully specified benchmark comprises seven interacting components:

• Task Formulation: the input-output mapping defining what a system must do — multiple-choice answers, code generation, span extraction, sequence-to-sequence generation, or structured prediction. Task formulation determines what capability is being measured and must be precise enough to remove ambiguity without constraining the solution space unnecessarily.

• Evaluation Split: train / validation / test partitions with the test set held out from all model development. Leakage between splits — through data contamination or prompt engineering using leaked test items — invalidates score comparability. The integrity of the split is the most critical safeguard against inflated results.

• Evaluation Metric: the scoring function that reduces model outputs to a comparable number — accuracy, BLEU/ROUGE for generation, pass@k for code, F1, Exact Match, or specialised metrics such as BERTScore and FID for multimodal tasks. The metric encodes what “good” means; metric choice has first-order consequences for what behaviours models are rewarded for optimising.

• Ground Truth Labels: human-annotated or programmatically generated reference outputs. Annotation quality is a direct function of Annotation Pipelines — inter-annotator agreement, quality control, expertise of annotators, and cultural representativeness of the labelling workforce.

• Scoring Protocol: deterministic inference settings — temperature, prompt templates, number of samples, context window size, and token budget — that must be fixed to prevent variance in experimental conditions from inflating or deflating scores.

• Leaderboards: public ranking tables that aggregate and display scores across systems. Leaderboards incentivise progress but also concentrate optimisation pressure on narrow benchmark slices, producing Goodhart’s Law dynamics where the benchmark ceases to measure the underlying capability it was designed to probe.

• Model Cards: structured documentation that contextualises benchmark scores alongside training details, intended use, known limitations, and disaggregated evaluation results — a practice institutionalised by Mitchell et al. (2019).

  Taxonomy of Benchmark Types

• Capability benchmarks: probe specific cognitive or task skills. Reasoning: GSM8K (mathematical word problems), MATH (competition mathematics), AIME 2025 (olympiad problems). Coding: HumanEval, SWE-bench Verified, LiveCodeBench (contamination-resistant programming). Knowledge: MMLU, MMLU-Pro, GPQA Diamond (graduate expert reasoning). Multilingual: FLORES-200, M-MMLU.

• Safety and alignment benchmarks: evaluate refusal behaviour, bias, toxicity, and instruction-following fidelity — TruthfulQA (Truthfulness), HellaSwag (commonsense reasoning), BBQ (bias), WMDP (hazardous knowledge). AISI (UK AI Security Institute) published AgentHarm (ICLR 2025), a benchmark for LLM agent misuse.

• Robustness benchmarks: test generalisation under distribution shift, adversarial inputs, and paraphrase variation — AdvGLUE, CheckList, ANLI, related to Adversarial Testing.

• Hardware benchmarks: MLPerf Training v6.0 and Inference v6.0 (2026) measure tokens-per-second, time-to-train, and energy-per-inference across GPU, TPU, and specialised accelerator platforms for 24 participating organisations including AMD, Google, Intel, and NVIDIA.

• Agentic and long-context benchmarks: GAIA, AgentBench, HELMET, RULER, τ-bench, and BFCL evaluate multi-step AI Agents on tool use, web navigation, long-document synthesis, and function-calling.

• Multimodal benchmarks: MMBench, MMMU, MMMU-Pro, SeedBench, VideoMME combine vision and language for Multimodal AI evaluation.

• Domain-specific benchmarks: MedQA (clinical medicine), MedMCQA (Indian medical licensing), LegalBench (legal reasoning), FinanceBench (financial QA), CODEMEDICAL (medical coding).

• Dynamic benchmarks: LiveCodeBench (continuously refreshes programming problems from Leetcode, AtCoder, Codeforces), LiveBench, and HELM-Lite use time-segmented or programmatically regenerated test cases to defeat training contamination.

  Benchmark Contamination

  Training corpora drawn from the open web may include benchmark test sets verbatim or in paraphrase, inflating apparent performance without genuine capability gain. This is the central measurement integrity challenge of the LLM era. Methodological responses include:

• Held-out evaluation sets with pre-publication embargo and cryptographic watermarking.

• Dynamic benchmarks that regenerate problems programmatically (LiveCodeBench, HELM-Lite, ARC-AGI-2 private eval set).

• Third-party blind evaluation services and closed test sets inaccessible to model developers.

• Time-segmented evaluations: problems whose publication dates postdate the model’s training data cutoff.

• Data deduplication pipelines that flag benchmark membership in training corpora. Contamination detection is a growing research area (Golchin and Surdeanu, 2023; Dong et al., 2024) and a central concern of AI Governance frameworks.

  Use Cases

• Model selection and procurement: practitioners use benchmark scores to choose between models for deployment, particularly where compute constraints, latency requirements, or domain specificity are paramount — supports Model Selection workflows in MLOps.

• Research progress tracking: the ML community tracks annual progress on canonical benchmarks to identify capability jumps and emergent behaviours in Large Language Models. Stanford HAI’s AI Index 2025 documents SWE-bench performance rising from 4.4% (2023) to 71.7% (2024) as a landmark capability progression.

• Regulatory compliance evidence: the EU AI Act (2024) and NIST AI Risk Management Framework increasingly reference benchmark performance as evidence for conformity assessments — bridging into Regulatory Compliance. The COMPL-AI framework maps benchmark coverage to EU AI Act requirements; Bench-2-CoP uses LLM-as-judge analysis to validate coverage against regulatory taxonomies.

• Hardware procurement: data-centre operators use MLPerf scores to compare accelerators; cloud providers publish inference throughput benchmarks as part of SLA documentation.

• MLOps regression testing: CI/CD pipelines run lightweight benchmark slices after each model update to detect capability regressions — extends Software Testing into ML workflows.

• Red-teaming and safety evaluation: safety teams use curated adversarial benchmark suites to probe for harmful outputs before public release, feeding into AI Governance processes. The UK AISI uses evaluation suites across cybersecurity, chemistry, biology assistance, and autonomous task-completion in its frontier model evaluations.

• Competition and reproducibility: benchmark-centric competitions (BIG-bench Collaboration, HELM, LMSYS Chatbot Arena / LMArena) accelerate community participation and Reproducibility of results.

  Goodhart’s Law and Benchmark Saturation

  Goodhart’s Law — “when a measure becomes a target, it ceases to be a good measure” — is the epistemological nemesis of benchmark-based evaluation. The history of AI benchmarks is a history of this dynamic playing out: models trained to maximise MMLU scores do not necessarily exhibit the general knowledge the benchmark was intended to measure; models that pass HumanEval’s 164 programming problems do not necessarily generalise to real-world software engineering tasks. The 2025 “Leaderboard Illusion” paper documented systematic divergence between leaderboard rankings and head-to-head deployment performance, underscoring the danger of treating benchmark scores as proxies for real-world capability. Defences include: multi-metric evaluation suites (HELM); adversarial and dynamic benchmarks that resist optimisation; disaggregated reporting that surfaces performance variation across subpopulations; and complementary Human Evaluation to validate automated metric choices.

  Historical Progression of AI Benchmarks

  Understanding the current benchmark landscape requires tracing the historical trajectory from narrow task-specific evaluations to comprehensive multi-capability suites, and from the pre-statistical-learning era through deep learning to the LLM paradigm.

  Era 1: Narrow pattern recognition benchmarks (1980s-2010). The earliest standardised AI benchmarks focused on well-defined pattern recognition problems with small, controlled datasets. MNIST (LeCun et al., 1998) — 60,000 handwritten digits for classification — served as the canonical benchmark for digit recognition and later for validating new neural architectures. It was accompanied by the UCI Machine Learning Repository, which hosted tabular datasets for classification and regression tasks that enabled comparisons across classical methods (decision trees, SVMs, k-NN). These benchmarks were characterised by small data volume, narrow task scope, and high reproducibility, but their narrow scope limited what could be inferred about general-purpose AI capability. The Pascal Visual Object Classes challenge (2005-2012) introduced multi-class object detection benchmarks for natural images, stimulating development of sliding window detectors, deformable part models, and eventually convolutional neural networks.

  Era 2: ImageNet and deep learning catalysis (2009-2018). ImageNet (Deng et al., 2009), with 1.28 million training images across 1,000 categories, transformed the benchmark landscape by providing a dataset large enough to train and differentiate deep neural networks. The annual ILSVRC challenge, running from 2010 to 2017, produced the most consequential benchmark competition in AI history: top-5 classification error fell from 28.2% (2010 winner) to 2.3% (2017 winner, SENet), with AlexNet’s breakthrough in 2012 (15.3% error, halving the previous best) effectively launching the deep learning revolution. ImageNet demonstrated that large-scale benchmarks with sufficient data complexity could drive years of algorithmic innovation and serve as a common measuring stick for the entire research community. This model — large dataset, annual competition, public leaderboard — became the template for subsequent benchmark design. The CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) provided smaller-scale complements for rapid architectural experimentation.

  Era 3: NLP benchmarks and multi-task evaluation (2015-2020). The deep learning revolution in NLP, catalysed by word embeddings (Word2Vec, GloVe), attention mechanisms, and eventually the Transformer (Vaswani et al., 2017), produced a new generation of language understanding benchmarks. SQuAD (Rajpurkar et al., 2016) established reading comprehension as a benchmark task with human performance as the target; models exceeded human performance on SQuAD 1.1 by 2018, prompting the harder SQuAD 2.0 with unanswerable questions. GLUE (Wang et al., 2018) aggregated nine NLP tasks into a single score, enabling comparison of models across diverse linguistic phenomena — textual entailment, sentiment analysis, semantic similarity, and syntactic parsing — with a single leaderboard. SuperGLUE (Wang et al., 2019) raised the difficulty after GLUE saturation, introducing tasks requiring long-document reasoning and common-sense inference. CoNLL shared tasks (2000-2003) standardised sequence labelling benchmarks (named-entity recognition, chunking, dependency parsing) that defined NLP evaluation for a decade. WMT translation competitions provided annual benchmarks for machine translation across dozens of language pairs, driving development from phrase-based to neural MT systems.

  Era 4: Foundation model benchmarks and emergent capability probing (2020-2023). GPT-3 (Brown et al., 2020) and its successors demonstrated that scale produces qualitatively new capabilities — few-shot learning, in-context generalisation, chain-of-thought reasoning — that existing benchmarks were not designed to measure. A rapid proliferation of benchmarks targeting these capabilities followed. MMLU (Hendrycks et al., 2020) covered 57 subjects across STEM, humanities, and social sciences using academic exam questions, becoming the most widely reported benchmark for general knowledge assessment. BIG-Bench (Srivastava et al., 2022) compiled 204 tasks contributed by the community targeting emergent capabilities beyond few-shot prompting — from analogical reasoning to sarcasm detection to moral dilemmas. HELM (Liang et al., 2022) introduced the first attempt at holistic evaluation across 42 scenarios with multiple metrics per scenario, measuring accuracy, calibration, robustness, fairness, and efficiency simultaneously. HumanEval (Chen et al., 2021) and MBPP provided code generation benchmarks that enabled systematic comparison of code-capable models. TruthfulQA (Lin et al., 2022) targeted factuality, measuring whether models produce false but plausible answers on questions where human knowledge often reflects common misconceptions.

  Era 5: Agentic, long-context, and reasoning benchmarks (2023-2026). The deployment of LLMs as autonomous agents capable of tool use, web browsing, and multi-step task execution exposed the inadequacy of single-turn question-answering benchmarks for evaluating agentic capabilities. SWE-bench (Jimenez et al., 2023) — tasks requiring models to resolve GitHub issues in real software repositories — provided a landmark agentic coding benchmark with clear success criteria and verifiable solutions. AI systems improved from 4.4% success (2023) to 71.7% (2024) on SWE-bench, according to Stanford HAI AI Index 2025, representing one of the most dramatic documented capability progressions in benchmark history. GAIA (Mialon et al., 2023) tested general AI assistants on real-world tasks requiring multi-step reasoning, tool use, and web navigation. GPQA Diamond (Rein et al., 2023) introduced graduate-level science questions designed to be too difficult for Google search to answer — a benchmark specifically designed to resist contamination and measure genuine expert-level reasoning. The focus on longer-context tasks produced RULER and HELMET benchmarks for evaluating models across 4k to 128k context windows. By 2025-2026, frontier benchmarks (HLE, FrontierMath, ARC-AGI-2) require solving olympiad-level mathematics, formal proofs, and novel visual puzzles, preserving meaningful discrimination even as model capabilities continue to advance.

  Era 6: Dynamic, preference-based, and regulatory benchmarks (2024-2026). The convergence of three pressures — contamination of static datasets, saturation of capability benchmarks, and regulatory demand for evaluation evidence — produced a shift toward dynamic, preference-based, and compliance-oriented evaluation frameworks. LiveCodeBench (Jain et al., 2024) demonstrated that continuous harvesting of fresh competitive programming problems from Leetcode, AtCoder, and Codeforces can maintain contamination resistance indefinitely. Chatbot Arena / LMArena operationalised large-scale human preference evaluation, accumulating millions of pairwise preference votes to produce Elo rankings that correlate well with deployment quality. The UK AISI’s frontier model evaluation programme introduced bespoke safety evaluation suites covering cybersecurity, chemistry/biology assistance, and autonomous task-completion — the first government-mandated benchmark regime for frontier AI. The EU AI Act’s conformity assessment requirements created demand for benchmark-based compliance evidence, driving development of regulatory benchmark frameworks including COMPL-AI (2024) and Bench-2-CoP (2025).

  Academic Context

• Foundational benchmarks: MNIST (LeCun et al., 1998), Pascal VOC (Everingham et al., 2010), ImageNet (Deng et al., 2009; Russakovsky et al., 2015), CIFAR-10/100 (Krizhevsky, 2009) established the pattern of shared visual recognition challenges.

• NLP era: GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) standardised multi-task NLP evaluation; SQuAD (Rajpurkar et al., 2016) established machine reading comprehension; CoNLL shared tasks (Tjong Kim Sang, 2003) defined sequence labelling.

• LLM era: MMLU (Hendrycks et al., 2020) probed multidisciplinary knowledge; GSM8K (Cobbe et al., 2021) tested mathematical reasoning; HumanEval (Chen et al., 2021) evaluated code generation; BIG-Bench (Srivastava et al., 2022) compiled 204 diverse tasks; HELM (Liang et al., 2022) introduced holistic multi-scenario evaluation; TruthfulQA (Lin et al., 2022) tested factual accuracy.

• Agentic era: GAIA (Mialon et al., 2023), SWE-bench (Jimenez et al., 2023), AgentBench (Liu et al., 2023), GPQA (Rein et al., 2023), HLE (Collins et al., 2025), ARC-AGI-2 (Chollet, 2024) push evaluation toward harder, more realistic, and more agentic challenges.

• Dynamic evaluation: LiveCodeBench (Jain et al., 2024) demonstrated contamination-resistant evaluation through continuous problem harvesting; HEIM and VideoMME pushed multimodal evaluation frontiers.

• Meta-evaluation: The Leaderboard Illusion (2025), Bench-2-CoP (2025), and the AIReg-Bench regulatory compliance benchmark (2025) study benchmark validity at a meta level.

  Current Landscape (2026)

  By June 2026, the benchmark landscape is characterised by saturation of 2022-2023 canonical suites and rapid adoption of harder replacements. MMLU, GSM8K, HumanEval, and HellaSwag are saturated, with GPT-5.x, Claude Opus 4.6, Gemini 3.x, and Llama 4 family models clustered above 90% on all of them. GPQA Diamond remains partially discriminative with Gemini 3.1 Pro at 94.3%, Claude Opus 4.6 at 91.3%, and GPT-5.3 Codex at 81%. The frontier lab model cards in 2026 report primarily on HLE, FrontierMath, ARC-AGI-2, GPQA Diamond, SWE-bench Verified, Aider Polyglot, AIME 2025, τ-bench, BFCL, MMMU-Pro, RULER, and LiveBench.

  MLPerf Inference v6.0 (April 2026, MLCommons) introduced new tests for text-to-video, GPT-OSS 120B, DLRMv3, vision-language models, and YOLOv11 across 24 participating organisations. MLPerf Training v6.0 (June 2026) updated training suite benchmarks for LLMs, text-to-image, and recommender systems. MLPerf Inference v5.1 (September 2025) introduced DeepSeek-R1, Llama 3.1 8B, and Whisper Large V3 as new benchmarks with 27 participating organisations.

  The LMSYS Chatbot Arena has evolved into LMArena, an independent project from UC Berkeley SkyLab, and remains a leading human-preference-based evaluation platform using Elo rankings via the Bradley-Terry model. Dynamic, preference-based, and private-eval benchmarks are increasingly preferred over static public suites for frontier model comparison.

  Regulatory use of benchmarks is accelerating: the EU AI Act’s COMPL-AI framework maps benchmark suites to compliance requirements; AISI (UK) uses bespoke evaluation suites for frontier model safety assessments published in the inaugural Frontier AI Trends Report (December 2025) covering 30+ frontier models evaluated over two years.

  UK Context

  The United Kingdom has distinctive benchmark infrastructure and governance engagement:

• AISI (AI Security Institute): the DSIT-funded AISI conducts mandatory evaluations of frontier AI systems ahead of deployment, using bespoke benchmark suites across cybersecurity, chemistry/biology assistance, and autonomous task-completion. AISI published AgentHarm at ICLR 2025 — a benchmark for LLM agent misuse that has been cited by OpenAI and Anthropic and downloaded thousands of times. The inaugural Frontier AI Trends Report (December 2025) documented capability trajectories across 30+ frontier models evaluated since November 2023.

• Imperial College London: ranked second globally in QS 2025, houses the UK’s largest concentration of computing and AI researchers; contributes to benchmark methodology research across NLP and multimodal AI.

• University of Edinburgh (EPCC): the Spending Review committed up to £750 million to a national supercomputer at Edinburgh Parallel Computing Centre, planned for 2027 — infrastructure that will support large-scale benchmark evaluation for UK researchers.

• Alan Turing Institute: hosts the AI for Science programme and contributes evaluation methodology research supporting Reproducibility in ML research across UK universities.

• Northern England: Manchester (National AI Research Institute Turing-Manchester node), Leeds (LIDA data innovation centre), Sheffield (NLP research groups), and Newcastle (digital health AI) all conduct domain-specific benchmark development and evaluation for healthcare, manufacturing, and transport applications.

• EU AI Act engagement: UK-based AI firms operating in the EU market must comply with conformity assessment requirements under the Act, driving UK adoption of benchmark-based compliance evaluation workflows.

  Standards and Governance Context

  The benchmark ecosystem intersects with a growing set of international standards and regulatory frameworks that are reshaping how evaluation evidence must be documented and used:

  ISO/IEC 22989:2022 (Artificial Intelligence — Concepts and Terminology) provides the foundational vocabulary for AI performance measures and evaluation criteria, including definitions of accuracy, precision, and recall that underpin metric standardisation. ISO/IEC TR 24030 addresses AI use cases and their evaluation requirements across application domains, creating a taxonomy of tasks that maps to benchmark categories. These ISO standards are increasingly cited in procurement contracts and conformity assessment procedures.

  NIST AI Risk Management Framework (AI RMF 1.0, 2023) organises AI risk management into four functions — Govern, Map, Measure, Manage — with MEASURE explicitly requiring organisations to analyse and assess the risk of AI systems using quantitative and qualitative evaluation methods. Benchmark-based evaluation is the primary mechanism through which MEASURE function activities are implemented in practice. The AI RMF’s playbook associates specific benchmark types with risk domains: capability benchmarks for accuracy assessment, safety benchmarks for harm potential, fairness metrics for bias audits.

  EU AI Act (2024): The Act mandates that high-risk AI systems undergo conformity assessment before market placement, which for self-assessed systems requires documented evidence of performance against stated accuracy and robustness specifications. Benchmark-based evaluation provides this evidence. Annex IV specifies the technical documentation requirements, which include information about training, validation, and testing processes and datasets, metric thresholds, and performance evaluation results. The COMPL-AI benchmarking suite (2024) maps existing AI benchmarks to EU AI Act compliance requirements across capability dimensions; Bench-2-CoP (2025) validates this mapping using LLM-as-judge analysis. General-Purpose AI (GPAI) models with systemic risk must also undergo evaluation of adversarial robustness, cybersecurity vulnerabilities, and large-scale risks — tasks addressed by safety benchmarks including those developed by the UK AISI.

  MLCommons / MLPerf: the premier industry benchmark consortium, MLCommons manages both the MLPerf Training and Inference suites that are the de facto hardware benchmarks for AI. The consortium’s open-submission model — with results audited by independent technical committees — provides a governance structure for hardware benchmark integrity that the software/model benchmark community has not yet matched. MLPerf v6.0 (2026) added vision-language, text-to-video, and mixture-of-expert model benchmarks, reflecting the expanding frontier of deployment scenarios.

  Hugging Face Open LLM Leaderboard and EleutherAI’s LM Evaluation Harness are the primary open-source infrastructure for software model benchmarking, enabling any researcher to run standardised evaluations and submit results to a public leaderboard. The evaluation harness supports hundreds of benchmarks with standardised prompting, few-shot configurations, and metric computation — reducing inconsistency across reported results and enabling apples-to-apples comparison across independent research groups.

  LMSYS / LMArena: the human-preference leaderboard (Elo-based via Bradley-Terry model) provides the most deployment-relevant signal by measuring which model users prefer in open-ended conversation. Its evolution from LMSYS Chatbot Arena to the independent LMArena reflects the growing importance of human preference signals as a complement to automated metrics.

  Benchmark Design Principles and Best Practices

  Designing a benchmark that serves the community well over time — providing discriminative power, resisting gaming, and measuring genuinely important capabilities — requires attention to a set of principles that the benchmark community has developed, often through observing failures of earlier designs:

  Difficulty calibration: a well-designed benchmark should discriminate across a meaningful range of model capability, not cluster all models near floor or ceiling. This requires careful item selection across a difficulty gradient. For multiple-choice benchmarks, item difficulty can be estimated from human rater accuracy or from existing model performance. For generative tasks, difficulty must be assessed by human experts or through automated complexity analysis. Benchmarks that are too easy saturate quickly; benchmarks that are too hard have floor effects where no model shows meaningful performance and the leaderboard is dominated by noise.

  Item independence and diversity: test items should be as independent as possible — having one item in the test set should not make adjacent items easier to answer. Diversity across topics, difficulty levels, reasoning types, and linguistic styles ensures that the benchmark measures a broad capability rather than a narrow specialisation. MMLU achieves diversity by spanning 57 academic subjects; BIG-bench achieves it by crowdsourcing 204 tasks from distinct contributors.

  Resistance to shortcut solutions: many benchmarks have been defeated by shortcut-exploiting models that achieve high accuracy without acquiring the intended capability. Reading comprehension benchmarks were defeated by models that matched answer spans using superficial lexical overlap without understanding the passage. Commonsense reasoning benchmarks were defeated by models that learned dataset-specific annotation artefacts — distributional cues in the answer choices that human annotators unconsciously introduced. Designing benchmarks that resist shortcut solutions requires adversarial validation: explicitly testing whether the task can be solved by models that provably lack the target capability.

  Ground truth quality and inter-annotator agreement: the validity of a benchmark as a measure of capability depends entirely on the quality of its ground truth labels. Annotation pipelines must specify: annotator expertise requirements (domain expert vs. crowdworker); annotation instructions and examples; inter-annotator agreement thresholds; and quality control procedures (random spot-checking, calibration rounds, independent expert review of disputed items). Inter-annotator agreement (Cohen’s Kappa, Fleiss’ Kappa, or Krippendorff’s Alpha depending on the task type) should be reported alongside the benchmark to characterise the noise floor of the ground truth.

  Test set security and access control: for benchmarks intended as long-term evaluation standards, the test set must be protected against leakage into training corpora. Strategies include: keeping the test set private with API-only access (ARC-AGI-2); time-locking test items so that they are only released after model training cutoffs have passed (LiveCodeBench); using watermarking to detect reproduction in training data; and releasing test items only to trusted evaluation partners under NDA. These measures trade off accessibility for integrity.

  Metric selection and alignment: the chosen Evaluation Metric must align with the intended capability. This requires thinking carefully about: what constitutes a correct answer (for ambiguous tasks); how to handle near-misses (exact match vs. F1 vs. human judge); whether false positives and false negatives have equal costs; and whether the metric captures the full range of outputs the model might produce. Using multiple metrics (HELM’s approach) mitigates the risk of any single metric being gamed.

  Documentation and reproducibility standards: every benchmark should publish: the complete data collection and annotation methodology; the distribution of item types and difficulty levels; inter-annotator agreement statistics; any pre-processing applied to inputs or outputs; the prompt template and few-shot examples used in evaluation; and the exact code for metric computation. Papers With Code’s benchmark documentation standards and Hugging Face’s datasets card format provide templates. Without complete documentation, results reported by different research groups on the “same” benchmark may not be comparable due to differences in evaluation implementation.

  Community governance and long-term maintenance: benchmarks that become widely adopted require ongoing governance: bug fixes, version management, policy on score submission validation, and handling of requests for additional tasks or harder variants. The BIG-bench Collaboration’s community contribution model and MLCommons’ consortium governance represent two approaches to sustainable benchmark stewardship. Without active maintenance, benchmarks accumulate errors, ambiguities, and outdated items that reduce their validity.

  Reproducibility and Open Science

  Benchmarks are instruments for scientific reproducibility in machine learning — but only if the results they produce can actually be reproduced. Reproducibility in benchmark evaluation depends on several conditions that are frequently not met in practice:

  Determinism: model inference may be non-deterministic due to hardware-level floating-point rounding differences, sampling with random seeds, and parallelism-induced non-determinism. Benchmark evaluation protocols must specify whether results are averaged over multiple seeds, and if so, how many, and how variance is reported. Results from a single seed on a stochastic model cannot be assumed to generalise.

  Software versioning: the exact versions of evaluation frameworks (EleutherAI LM Eval Harness, Hugging Face Evaluate), model libraries (PyTorch, JAX, TensorFlow), tokenisers, and prompt templates all affect benchmark results. Papers reporting benchmark results should specify the exact evaluation framework version and configuration. Even minor version differences in tokenisation can change tokenised input lengths enough to affect results on context-length-sensitive benchmarks.

  Prompt template sensitivity: many benchmarks show significant sensitivity to the exact wording of the instruction prompt — the specific system prompt, the number and selection of few-shot examples, and the exact format of the question and answer choices. Prompt sensitivity is a source of systematic variance that can be as large as the performance differences between competing models. Evaluations should report the exact prompt template used, and ideally test sensitivity to template variation.

  Compute and inference configuration: results on benchmarks can depend on inference-time compute parameters — sampling temperature, number of candidate solutions, context window size, and chain-of-thought prompting. These parameters must be fixed and reported. Results from a model with access to longer chain-of-thought reasoning or more samples are not comparable to results without.

  Model version and training data cutoff: frontier models are updated continuously, often with improvements that are not reflected in version numbers. Benchmark results should specify the exact model checkpoint evaluated (including version, date, and training data cutoff if known) to enable temporal tracking of performance and detection of score improvements attributable to training data updates that may include benchmark contamination.

  Challenges, Limitations, and Criticisms

  Despite their centrality to AI research and deployment, benchmarks face substantial methodological and structural criticisms that the field is only beginning to address systematically:

  Goodhart’s Law in practice: the historical trajectory of AI benchmarks is an extended illustration of Goodhart’s Law. ImageNet drove the development of models optimised for its specific 1000-class taxonomy, producing systems that could be fooled by simple adversarial perturbations invisible to humans. GLUE was saturated by models that relied on dataset-specific annotation artefacts rather than genuine language understanding. MMLU is now gamed by models trained on question patterns and eliminated answer choices rather than genuine multidisciplinary knowledge. Each cycle requires harder benchmarks, which then attract targeted optimisation, completing the loop.

  Benchmark contamination at scale: the scale of modern training data makes contamination near-certain for any publicly released benchmark. A benchmark with 10,000 test questions and a training corpus of 10 trillion tokens will almost certainly contain the questions verbatim — the only question is what percentage. This makes reported scores on static benchmarks increasingly unreliable as training data scales. The field’s response — dynamic benchmarks, time-locked evaluations, private eval sets — introduces its own challenges: dynamic benchmarks are harder to standardise and reproduce; private eval sets cannot be externally validated.

  Metric-capability gap: automated metrics diverge substantially from human judgement on tasks requiring creativity, nuanced reasoning, cultural understanding, and long-form generation. BLEU and ROUGE correlate poorly with human translation quality preferences; automated coding metrics miss architectural elegance and correctness edge cases; safety metrics assessed by automated classifiers diverge from expert human evaluation on borderline cases. Multi-metric frameworks (HELM, BIG-bench) partially address this by aggregating across many metrics, but the fundamental problem of metric-capability misalignment remains unsolved.

  Coverage and representativeness: most benchmarks are developed in English by US/EU academic institutions, systematically under-representing non-English languages, non-Western cultural contexts, and minority dialects. Models evaluated only on standard benchmark suites may perform poorly on dialects, code-switching, and culturally specific knowledge that benchmarks do not cover. Bias and fairness benchmarks (BBQ, WinoBias, CrowS-Pairs) specifically target these gaps but are not yet as widely adopted as capability benchmarks.

  Absence of uncertainty quantification: most benchmark scores are reported as point estimates without confidence intervals or Statistical Testing for significance. On small test sets (HumanEval has 164 problems; GPQA Diamond has 198), performance differences of 2-5 percentage points may not be statistically distinguishable from noise. The community’s adoption of significance testing for leaderboard comparisons remains limited despite clear statistical necessity.

  Benchmark proliferation: the rate at which new benchmarks are introduced — hundreds per year in the LLM era — creates a curation and meta-evaluation challenge. Researchers face difficulty identifying which benchmarks are most predictive of real-world capability, most resistant to contamination, and most relevant for their specific use case. Benchmark meta-evaluation frameworks (HELM’s benchmark-of-benchmarks analysis; Papers With Code leaderboard aggregation) are emerging tools but are still immature.

  Future Directions (2026-2030)

• Adaptive and adversarial benchmarks: automated red-teaming that continuously generates novel evaluation instances the model has not been exposed to, maintaining discriminative power indefinitely. Techniques from adversarial machine learning — worst-case input generation, distribution shift simulation, and compositional generalisation tests — are being incorporated into evaluation pipelines.

• Agent and long-horizon evaluation: as AI systems operate autonomously across longer horizons and multi-step tasks, benchmark design must capture not just accuracy but reliability, safety, and alignment across extended trajectories. τ-bench, OSWorld, WebArena, and GAIA point in this direction, requiring multi-turn interaction logs, tool-use traces, and success-rate-at-completion rather than single-turn accuracy.

• Regulatory benchmark suites: convergence between benchmark methodology and regulatory conformity assessment is expected, with EU and UK regulatory bodies developing official benchmark suites analogous to NIST cybersecurity frameworks. The COMPL-AI mapping and Bench-2-CoP analysis represent first steps; full regulatory benchmark suites with legal standing are anticipated by 2028.

• Efficiency-aware benchmarks: as sustainability and compute efficiency become regulatory priorities, benchmarks that report performance-per-watt and performance-per-dollar alongside raw capability scores (extending MLPerf’s approach) are expected to become standard. The EU AI Act’s energy consumption reporting requirements for GPAI models will drive adoption.

• Multimodal and embodied evaluation: benchmarks for video, audio, sensor fusion, and robotics perception (building on VideoMME, EmbodiedBench, and robotics manipulation suites) will grow in importance as Multimodal AI systems proliferate in physical deployment contexts.

• Living benchmarks with versioning: the community is moving toward benchmark suites with explicit versioning, provenance tracking, and community governance — analogous to software dependency management — to ensure longevity and cross-version comparability. Papers With Code’s benchmark versioning infrastructure and Hugging Face datasets versioning are early implementations.

• AI-generated benchmark instances: LLM-assisted benchmark creation is accelerating (BenchBuilder, task-synthesis pipelines), enabling rapid development of domain-specific suites while raising new validity questions about AI-generated ground truth. The quality, diversity, and difficulty calibration of AI-generated items require human expert validation that may constrain scale.

• Sociotechnical evaluation: as AI systems deploy at societal scale, benchmarks measuring individual model capability are insufficient. Evaluation frameworks that assess systemic effects — market concentration, labour displacement, environmental footprint, and epistemic homogenisation — are nascent but will become more prominent in regulatory and governance contexts.

  Key Terminology

  A glossary of core benchmark concepts essential for correctly reading and interpreting benchmark reports:

  Task formulation: the precise specification of what a model must do, including the input format, output format, permissible resources (retrieval, tool use, chain-of-thought), and any instruction templates. Two benchmarks measuring “reasoning” may use completely different task formulations, making direct score comparison invalid even when they share a name.

  Evaluation split (train / validation / test): the partition of labelled data into subsets used for training, tuning, and final evaluation. The test split must be held out from all model development activities — including prompt engineering, hyperparameter selection, and architectural decisions — to remain a valid measure of generalisation to unseen data. Contamination occurs when test data appears in the training corpus.

  Benchmark contamination: the presence of benchmark test items, or text closely derived from them, in a model’s training corpus. Contamination inflates apparent performance without genuine capability gain, because the model is recalling rather than reasoning. Detection methods include training cutoff analysis, n-gram overlap detection, membership inference attacks, and time-segmented evaluation. In 2026, contamination is presumed for all static public benchmarks used to evaluate frontier models trained on web-scale corpora.

  Static vs. dynamic benchmarks: static benchmarks have a fixed set of test items that do not change after publication; they are simple to reproduce and compare but are vulnerable to contamination and saturation. Dynamic benchmarks regenerate or refresh test items on a continuous or periodic basis, making contamination harder but reducing strict reproducibility. LiveCodeBench (continuous harvesting from online programming competitions) and ARC-AGI-2 (private eval set) exemplify the dynamic approach.

  Evaluation metric: the mathematical function that maps model outputs and ground truth labels to a scalar performance score. The choice of metric is consequential: different metrics reward different model behaviours. Accuracy rewards correctness on every example equally; F1 balances precision and recall; pass@k (used in HumanEval) rewards any correct solution among k samples; human Elo (used in LMArena) rewards user preference in pairwise comparison. The metric defines what the leaderboard is actually measuring.

  Few-shot vs. zero-shot evaluation: few-shot evaluation provides the model with k example input-output pairs (typically k=1, 5, or 10) in the prompt before presenting the test question, testing in-context learning from the examples. Zero-shot evaluation provides no examples, testing the model’s prior knowledge and instruction-following. Few-shot results are typically higher and may reflect in-context generalisation ability as much as knowledge of the subject matter; specification of k is required for valid comparison.

  Leaderboard saturation: the phenomenon in which state-of-the-art models cluster at near-ceiling performance on a benchmark, eliminating its discriminative power. Saturation signals that the benchmark has been surpassed by model capability and requires replacement with a harder evaluation. MMLU saturated in 2024-2025; GPQA Diamond is approaching saturation at the frontier in 2026.

  Benchmark suite vs. single benchmark: a single benchmark measures one capability in one context (e.g. GSM8K measures mathematical word problem solving). A benchmark suite aggregates multiple benchmarks across diverse capabilities and scenarios, providing a more holistic picture of model capability. HELM covers 42 scenarios across accuracy, calibration, robustness, fairness, and efficiency; BIG-bench covers 204 diverse tasks. Suite-level aggregation requires a principled method for combining scores across heterogeneous metrics, which is itself a methodological challenge.

  Held-out test set: the portion of a benchmark’s data that is used exclusively for final evaluation, not for training or hyperparameter selection. The integrity of the held-out set depends on strict data governance — any leakage of test items into training corpora invalidates it as a measure of generalisation.

  Ground truth labels: the reference outputs against which model predictions are compared. In factual tasks (multiple-choice, extractive QA), ground truth is unambiguous. In generative tasks (open-ended QA, code generation, summarisation), ground truth is a set of acceptable outputs or a human rating, introducing annotation noise. The quality of ground truth labels is directly determined by the annotation pipeline, including annotator expertise, instruction quality, inter-annotator agreement, and quality control protocols.

  Goodhart’s Law: the principle that when a measure becomes a target, it ceases to be a good measure. In the benchmark context, this manifests as models optimised specifically to perform well on a benchmark without improving the underlying capability the benchmark was intended to probe. Detection requires: external validation by independent evaluators, comparison of benchmark scores to human preference judgements, and analysis of whether benchmark score improvements transfer to downstream tasks.

  Human evaluation: direct assessment of model outputs by human raters, used as a gold standard to validate automated metrics or to evaluate tasks (dialogue, creative writing, medical advice) where automated metrics are insufficient. Human evaluation is expensive, slow, and susceptible to rater biases, but provides the most deployment-relevant quality signal. Chatbot Arena / LMArena operationalises human evaluation at scale via pairwise preference voting, providing a continuously updated Elo leaderboard.

  Agentic evaluation: evaluation of AI agents that take sequences of actions to complete tasks, rather than single-turn predictions. Agentic benchmarks (GAIA, SWE-bench, OSWorld, AgentBench) require the model to use tools, maintain state, recover from errors, and execute multi-step plans — capabilities not measured by standard single-turn benchmarks. Scoring agentic tasks requires specifying success criteria for complete task execution, not just individual predictions.

  Model card: a structured documentation standard (Mitchell et al., 2019) that accompanies AI models, recording benchmark results, training details, intended use, out-of-scope uses, and ethical considerations. Model cards serve as the primary mechanism for communicating benchmark evidence to users and regulators; their adoption is mandated or strongly recommended by EU AI Act guidance, NIST AI RMF, and responsible AI frameworks from major AI labs.

  Statistical Foundations

  Responsible interpretation of benchmark results requires statistical rigour that the community has historically under-applied:

  Confidence intervals and significance testing: a model’s benchmark score is an estimate of its expected performance on the underlying task distribution, estimated from a finite test sample. The uncertainty of this estimate depends on test set size — GPQA Diamond (198 items), HumanEval (164 items), and GSM8K (1,319 items) have very different levels of precision. A 2% accuracy difference on HumanEval corresponds to approximately 3 problems and is not statistically distinguishable from noise at standard significance levels (p < 0.05). Results from small test sets should be accompanied by binomial confidence intervals or bootstrap confidence intervals to communicate uncertainty.

  Multiple comparisons: when a model is evaluated on dozens of benchmarks, some benchmark-specific improvements are expected by chance even for a model with no genuine capability improvement. Correction procedures (Bonferroni, Benjamini-Hochberg) are standard in empirical science but rarely applied in ML benchmark reporting. The result is systematic overstatement of the number of benchmarks on which a new model is genuinely better than its predecessor.

  Paired vs. unpaired tests: when comparing two models on the same benchmark, paired tests (comparing item-by-item correctness vectors) are more powerful than unpaired tests (comparing aggregate accuracy scores). McNemar’s test is the standard for paired binary outcomes. Paired significance testing is almost never reported in benchmark papers or model cards, despite being trivially implementable with access to per-item results.

  Calibration of reported scores: models can be miscalibrated on benchmark metrics just as they can be miscalibrated in confidence scores. A model that achieves 80% accuracy on a benchmark by correctly predicting the same 80% of easy items is behaviorally different from a model that achieves 80% by correctly predicting a random 80% of items. Disaggregated analysis — reporting performance by difficulty, topic, and demographic group — is required to distinguish these cases but is rarely provided in standard leaderboard submissions.

  Power analysis: the minimal test set size required to detect a performance difference of a given effect size at a given significance level depends on the effect size and the base rate accuracy. For typical benchmark accuracy levels (75-90%) and 2% effect sizes at 80% power and p < 0.05, approximately 2,000-5,000 test items are required. Most benchmarks fall below this threshold, limiting their ability to detect small but genuine performance differences.

  Benchmark Infrastructure and Tooling

  The benchmark ecosystem depends on a set of shared infrastructure tools that standardise evaluation, aggregate results, and lower the barrier to running reproducible assessments:

  EleutherAI Language Model Evaluation Harness (lm-evaluation-harness): the de facto open-source standard for evaluating language models on hundreds of benchmarks from a single unified interface. The harness handles prompt formatting, few-shot example selection, batched inference, metric computation, and result storage, eliminating many sources of implementation variance. It supports HuggingFace models natively and has been extended to support proprietary API models. Used by Hugging Face for the Open LLM Leaderboard, EleutherAI’s own evaluations, and thousands of independent researchers. Version-locked citations are essential when reporting results via this harness, as prompt templates and metric computations change between versions.

  Hugging Face Open LLM Leaderboard: the community standard for public model ranking on open-weight models. In 2024-2025, it transitioned from the first-generation suite (MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K) to a second-generation suite including MMLU-Pro, GPQA Diamond, MuSR, Math-Hard, and MBPP+ — reflecting the saturation of the first-generation benchmarks. The leaderboard accepts submissions from anyone with a public HuggingFace model repository and runs evaluations on standard hardware, providing hundreds of thousands of comparable data points.

  Papers With Code Leaderboard: a community-maintained database linking benchmark results to their source papers, providing historical tracking of state-of-the-art performance over time. It covers thousands of benchmarks across computer vision, NLP, audio, and reinforcement learning, enabling meta-analyses of research progress. The trend visualisations on Papers With Code provide the clearest high-level view of how capabilities have progressed across benchmark types over the past decade, connecting benchmark scores to the papers that achieved them and the code that implements them.

  HELM (Holistic Evaluation of Language Models): Stanford CRFM’s evaluation framework that goes beyond single-metric leaderboards to report multi-dimensional profiles including accuracy, calibration, robustness, fairness, efficiency, and disinformation resistance across 42 evaluation scenarios. HELM’s emphasis on comprehensive multi-metric reporting rather than a single aggregated score represents a methodological stance: that no single number adequately characterises model quality, and that trade-offs across dimensions must be made visible. HELM-Lite provides a faster evaluation subset for rapid iteration.

  LMArena (formerly LMSYS Chatbot Arena): the human-preference-based leaderboard maintained by UC Berkeley SkyLab, combining millions of anonymous pairwise preference votes to produce Elo rankings via the Bradley-Terry model. LMArena complements automated benchmarks by measuring what users actually prefer in open-ended conversation — the most ecologically valid evaluation of deployed model quality but also the hardest to reproduce and most susceptible to population effects, since the voting public may not represent target deployment populations. The platform has accumulated over 1 million human preference votes and is widely cited as the most reliable signal for conversational quality.

  BIG-bench Collaboration and BIG-bench Hard (BBH): the 204-task benchmark compiled by a 444-author community effort, designed to probe emergent capabilities of language models beyond few-shot prompting. The BIG-bench Hard (BBH) subset of 23 particularly challenging tasks requiring multi-step reasoning has become more widely used than the full suite, as it retains discrimination as models improve. The collaborative design process produces unusual task diversity but also uneven quality control, making careful task-level analysis important when using BIG-bench results.

  MLCommons Submission Ecosystem: the infrastructure supporting MLPerf benchmark submissions, including standard measurement methodologies, auditing procedures, and the LoadGen inference benchmark tool. The consortium’s open-submission model, with results audited by independent technical committees, provides governance structure for hardware benchmark integrity. Submission rules specify precise measurement conditions, power measurement protocols, and disclosure requirements for submitted systems.

  Research and Literature

  1. Deng, J., Dong, W., Socher, R., et al. (2009). ImageNet: A large-scale hierarchical image database. CVPR 2009, pp. 248-255.
  2. Russakovsky, O., Deng, J., Su, H., et al. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 211-252.
  3. Wang, A., Singh, A., Michael, J., et al. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR 2019. https://arxiv.org/abs/1804.07461
  4. Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. EMNLP 2016. https://arxiv.org/abs/1606.05250
  5. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019. https://arxiv.org/abs/1810.04805
  6. Brown, T. B., et al. (2020). Language models are few-shot learners (GPT-3). NeurIPS 2020. https://arxiv.org/abs/2005.14165
  7. Hendrycks, D., Burns, C., Basart, S., et al. (2020). Measuring massive multitask language understanding (MMLU). ICLR 2021. https://arxiv.org/abs/2009.03300
  8. Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating large language models trained on code (HumanEval). arXiv:2107.03374. https://arxiv.org/abs/2107.03374
  9. Cobbe, K., Kosaraju, V., Bavarian, M., et al. (2021). Training verifiers to solve math word problems (GSM8K). arXiv:2110.14168. https://arxiv.org/abs/2110.14168
  10. Srivastava, A., Rastogi, A., Rao, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models (BIG-Bench). Transactions on Machine Learning Research. https://arxiv.org/abs/2206.04615
  11. Liang, P., Bommasani, R., Lee, T., et al. (2022). Holistic evaluation of language models (HELM). Transactions on Machine Learning Research. https://arxiv.org/abs/2211.09110
  12. Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. ACL 2022. https://arxiv.org/abs/2109.07958
  13. Mitchell, M., Wu, S., Zaldivar, A., et al. (2019). Model cards for model reporting. FAccT 2019. https://arxiv.org/abs/1810.03993
  14. Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can language models resolve real-world GitHub issues? ICLR 2024. https://arxiv.org/abs/2310.06770
  15. Mialon, G., Dessi, R., Lomeli, M., et al. (2023). GAIA: A benchmark for general AI assistants. ICLR 2024. https://arxiv.org/abs/2311.12983
  16. Rein, D., Hou, B. L., Stickland, A. C., et al. (2023). GPQA: A graduate-level Google-proof Q&A benchmark. arXiv:2311.12022. https://arxiv.org/abs/2311.12022
  17. Jain, N., Han, K., Gu, A., et al. (2024). LiveCodeBench: Holistic and contamination free evaluation of LLMs for code. ICLR 2025. https://openreview.net/forum?id=chfJJYC3iL
  18. Golchin, S., & Surdeanu, M. (2023). Time travel in LLMs: Tracing data contamination in large language models. arXiv:2308.08493.
  19. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685
  20. MLCommons. (2026). MLPerf Training v6.0 Results. https://mlcommons.org/2026/mlperf-training-v6-0-results/
  21. MLCommons. (2026). MLPerf Inference v6.0 Results. https://mlcommons.org/2026/04/mlperf-inference-v6-0-results/
  22. AISI. (2025). Frontier AI Trends Report 2025. UK AI Security Institute. https://www.aisi.gov.uk/research/aisi-frontier-ai-trends-report-2025
  23. Stanford HAI. (2025). AI Index Report 2025 — Technical Performance. https://hai.stanford.edu/ai-index/2025-ai-index-report/technical-performance
  24. Bommasani, R., et al. (2023). COMPL-AI: A technical interpretation and LLM benchmarking suite for the EU Artificial Intelligence Act. arXiv:2410.07959. https://arxiv.org/abs/2410.07959
  25. Wang, A., Pruksachatkun, Y., Nangia, N., et al. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS 2019. https://arxiv.org/abs/1905.00537
  26. Chollet, F. (2024). ARC-AGI-2: A new evaluation for AGI. ARC Prize Foundation.
  27. Collins, E., et al. (2025). HLE: Humanity’s Last Exam. Scale AI / ARC Prize.

Provenance