Dataset

A Dataset is a structured collection of data records sharing a common schema, gathered for a specific purpose such as training machine learning models, conducting research, or supporting analytics. Datasets are characterised by their size, modality (text, image, tabular, audio, video, graph, etc.), provenance, and licensing terms, all of which affect their fitness for use. Data quality, curation methodology, and bias documentation are critical attributes that determine the reliability of downstream AI systems. The movement from model-centric to data-centric AI has elevated dataset engineering to a first-class research and engineering discipline, with dataset documentation frameworks such as Datasheets for Datasets, Dataset Nutrition Labels, and the Croissant machine-readable format formalising the metadata contract between dataset creators and downstream consumers.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:DataRecord))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:Schema))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:GroundTruthLabels))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:Metadata))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:DataSplit))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:hasPart ai:EvaluationMetric))
SubClassOf(ai:LabelledDataset
  ObjectSomeValuesFrom(ai:hasPart ai:Annotation))
SubClassOf(ai:MultimodalDataset
  ObjectSomeValuesFrom(ai:hasPart ai:ModalityPair))

Dependency Relationships

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:requires ai:DataCollection))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:requires ai:DataCleaning))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:requires ai:DataAnnotation))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:requires ai:DataGovernance))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:requires ai:DataLineage))
SubClassOf(ai:BenchmarkDataset
  ObjectSomeValuesFrom(ai:requires ai:GroundTruthLabels))
SubClassOf(ai:TrainingDataset
  ObjectSomeValuesFrom(ai:requires ai:DataLabelling))
SubClassOf(ai:SyntheticDataset
  ObjectSomeValuesFrom(ai:requires ai:GenerativeModel))

Capability Relationships

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:enables ai:MachineLearning))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:enables ai:ModelEvaluation))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:enables ai:Benchmarking))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:enables ai:DataAnalysis))
SubClassOf(ai:TrainingDataset
  ObjectSomeValuesFrom(ai:enables ai:SupervisedLearning))
SubClassOf(ai:LargeScaleDataset
  ObjectSomeValuesFrom(ai:enables ai:FoundationModel))
SubClassOf(ai:BenchmarkDataset
  ObjectSomeValuesFrom(ai:enables ai:Reproducibility))

Implementation Relationships

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:implements ai:DataGovernanceFramework))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:implements ai:Reproducibility))
SubClassOf(ai:DocumentedDataset
  ObjectSomeValuesFrom(ai:implements ai:DatasheetsForDatasets))
SubClassOf(ai:CuratedDataset
  ObjectSomeValuesFrom(ai:implements ai:DataQualityStandard))

Reduction Relationships

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:reducesTo ai:DataRecord))
SubClassOf(ai:MultimodalDataset
  ObjectSomeValuesFrom(ai:reducesTo ai:UnimodalDataset))
SubClassOf(ai:LargeScaleDataset
  ObjectSomeValuesFrom(ai:reducesTo ai:SampledDataset))
SubClassOf(ai:AnnotatedDataset
  ObjectSomeValuesFrom(ai:reducesTo ai:RawDataset))
SubClassOf(ai:CuratedDataset
  ObjectSomeValuesFrom(ai:reducesTo ai:UnprocessedCorpus))

Support Relationships

SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:supports ai:NaturalLanguageProcessing))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:supports ai:ComputerVision))
SubClassOf(ai:Dataset
  ObjectSomeValuesFrom(ai:supports ai:ExplainableAI))
SubClassOf(ai:LargeScaleDataset
  ObjectSomeValuesFrom(ai:supports ai:LargeLanguageModels))
SubClassOf(ai:FederatedDataset
  ObjectSomeValuesFrom(ai:supports ai:FederatedLearning))

About

A Dataset is the foundational unit of empirical artificial intelligence and machine learning research. In the supervised learning paradigm, a dataset pairs each input example with one or more target labels that a trained model should reproduce for unseen inputs; in unsupervised settings, inputs are provided without labels so the model must discover latent structure through dimensionality reduction, clustering, or density estimation; in reinforcement learning, the “dataset” is an experience replay buffer of state-action-reward-next-state tuples accumulated through environmental interaction. Regardless of paradigm, the statistical properties of the dataset—class balance and marginal distributions, feature range and covariance structure, sample diversity and geographic or demographic coverage, noise level and corruption patterns, and label consistency across annotators—are the dominant determinants of model quality in practical settings, often outweighing architectural choices for sufficiently large training regimes (Sun et al., 2017; Zha et al., 2023).

The field has undergone a fundamental shift from hand-curated, domain-specific datasets to web-scale corpora assembled through automated crawling and filtering. ImageNet (Deng et al., 2009), with 14 million labelled images across 21,000 WordNet synsets (a 1,000-class subset used for ILSVRC), became the benchmark that catalysed the deep learning revolution; its annual competition (ILSVRC, 2010–2017) drove successive accuracy improvements—from 26.2% top-5 error in 2011 to 2.25% in 2017, surpassing human-level performance—that transformed computer vision from an academic curiosity into a production technology. The transition to foundation model training has escalated dataset scale by orders of magnitude: Common Crawl provides over 300 billion web pages and grows by 3–5 billion pages monthly, representing hundreds of trillions of tokens. LAION-5B, released in 2022 by the LAION association, contains 5.85 billion image-text pairs derived from Common Crawl and was used to train open multimodal models including Stable Diffusion and OpenCLIP. At this scale, data quality cannot be enforced by human inspection; instead, automated filtering pipelines using classifier models (e.g., a CLIP ViT-L/14 model used to filter LAION by image-text alignment score), deduplication hashing (MinHash, exact URL deduplication), toxicity classifiers, and quality scoring are applied, with the trade-off that aggressive filtering may remove linguistically marginal or culturally non-Western but legitimate examples.

The concept of dataset provenance—systematic documentation of where, when, how, and by whom data was collected, processed, and annotated—has emerged as a first-class research and engineering concern. The Datasheets for Datasets framework (Gebru et al., 2018, published 2021 in CACM) provides a structured questionnaire covering seven dimensions: motivation, composition, collection process, preprocessing and cleaning, uses, distribution, and maintenance. A 2024 assessment of 60 NeurIPS Datasets and Benchmarks track submissions found significant variation in documentation quality, with many datasets omitting critical information about data sources, annotation procedures, and known limitations. The trend toward machine-readable documentation—Open Datasheets (2024), Croissant metadata format (Akhtar et al., 2024, adopted by Hugging Face, Kaggle, and OpenML)—aims to automate dataset discovery, compatibility checking, and regulatory compliance verification across repositories.

Components / Architecture

A fully specified dataset comprises several distinct components that together determine its fitness for a given downstream use:

• Data records: The atomic units of a dataset—images, text passages, tabular rows, graph triples, audio waveforms, sensor readings—constituting its content. Record format determines storage layout (CSV, Parquet, HDF5, TFRecord, WebDataset shards, Arrow) and access patterns during training. Large-scale datasets are typically sharded across many files to support parallel loading and avoid I/O bottlenecks during Deep Learning training runs on distributed GPU clusters.

• Schema: The formal specification of field names, data types, value ranges, cardinality constraints, and semantic meanings. A schema enables automated validation, prevents silent data corruption during pipeline transforms, and documents the mapping from raw data to model-ready features. Schema registries (Apache Avro, Protocol Buffers, JSON Schema) support schema evolution across dataset versions.

• Annotations and ground-truth labels: For supervised datasets, the target values paired with each input record—class labels, bounding boxes, segmentation masks, transcriptions, preference rankings. Ground Truth Labels quality is the single most impactful quality dimension: Northcutt et al. (2021) estimated error rates of 3.3%–5.8% in 10 canonical image classification benchmarks, with some benchmarks showing over 10% mislabelling. Consensus annotation protocols (majority vote, Dawid-Skene model, MACE) and active disagreement flagging are used to manage inter-annotator disagreement in tasks with subjective ground truth.

• Metadata: Dataset-level descriptors covering size, creation date, language, geographic coverage, collection methodology, licensing terms (open Creative Commons, restricted research-only, commercial, or proprietary), and version history. Metadata enables catalogue-based discovery via tools like Hugging Face Datasets Hub, Papers With Code, and OpenML. Well-structured metadata is the prerequisite for regulatory compliance documentation under the EU AI Act’s Article 10.

• Data splits: Predefined partitions into training, validation, and test subsets, with held-out test sets sometimes withheld from public release (as in SuperGLUE and MMLU) to prevent benchmark contamination—the phenomenon where model pre-training inadvertently exposes the model to test examples, inflating apparent performance. The standard train/val/test split hides a subtle distribution assumption: all splits must be drawn from the same underlying distribution as deployment inputs, which fails when dataset collection is geographically, temporally, or demographically stratified.

• Evaluation metrics and protocols: Defined scoring functions—accuracy, macro-F1, BLEU, FID (Fréchet Inception Distance), AUC-ROC, NDCG—and evaluation harnesses (EleutherAI lm-evaluation-harness, Stanford HELM) that standardise comparison across models and prevent cherry-picking of favourable metrics.

• Data cards and datasheets: Structured documentation artefacts accompanying the dataset, increasingly mandatory for dataset submissions to major venues (NeurIPS, ICML) and for high-risk AI systems under the EU AI Act. Data cards specify intended use cases, known limitations, demographic composition, and bias characterisation results.

• Data pipeline and versioning: The software artefacts responsible for downloading, preprocessing, transforming, and loading the dataset. Data Pipeline code is increasingly version-controlled alongside dataset releases (DVC—Data Version Control, MLflow Artifacts) to ensure that model training is reproducible across compute environments. Data Versioning tools track changes in dataset content over time, enabling regression analysis when data quality changes cause model performance shifts.

  Use Cases / Major Families

  Datasets divide into several major families by modality, scale, and intended use:

• Image classification: ImageNet-1k/21k (ILSVRC), CIFAR-10/100, iNaturalist, EuroSAT—benchmarks for visual recognition models including CNNs and Vision Transformers. ImageNet-1k with 1.2 million training images in 1,000 classes remains the canonical transfer learning source for Computer Vision models.

• Object detection and segmentation: MS-COCO (330,000 images, 80 categories), Pascal VOC, Open Images V7 (9 million images)—multi-class detection with bounding box and pixel-level annotations used for autonomous driving, surveillance, and medical imaging applications.

• Natural language pre-training corpora: C4 (Colossal Clean Crawled Corpus, 180 billion tokens), The Pile (825 GB, EleutherAI), RedPajama (1.2 trillion tokens), FineWeb (2024, 15 trillion tokens after quality filtering)—the primary training data for Large Language Models. Quality filtering strategies using perplexity scoring, n-gram deduplication, and content classification have substantial effects on downstream model performance.

• Natural language evaluation benchmarks: SQuAD (reading comprehension), GLUE and SuperGLUE (multi-task NLP), BIG-Bench (challenging NLP tasks), MMLU (57-subject multiple choice), HumanEval (code generation)—structured evaluation sets for Natural Language Processing models with standardised metrics.

• Multimodal image-text datasets: LAION-5B (5.85 billion image-caption pairs), CC3M (Conceptual Captions), YFCC100M, WIT (Wikipedia-based Image Text)—large-scale noisy corpora for training multimodal Foundation Model architectures including CLIP, DALL-E, and Stable Diffusion. DataComp (2023) provides a controlled framework for comparing dataset curation strategies on a fixed compute budget.

• Audio and speech: LibriSpeech (960 hours, audiobook), Common Voice (Mozilla, 100+ languages, 20,000+ hours), VoxCeleb (speaker verification), AudioSet (2 million YouTube clips with sound event labels)—for speech recognition, speaker diarisation, and audio classification.

• Medical and clinical: MIMIC-III/IV (ICU records for 40,000+ patients, PhysioNet), UK Biobank (genetic and imaging data from 500,000 UK participants), ChestX-ray14 (100,000+ chest X-rays), TCGA (The Cancer Genome Atlas)—highly regulated datasets with strict access controls, data use agreements, and IRB/ethics requirements. Federated Learning is the primary mechanism for accessing distributed medical datasets that cannot be centralised.

• Tabular and structured: UCI ML Repository (500+ datasets), OpenML (4,000+ datasets), Kaggle competition datasets—heterogeneous collections for classical Machine Learning benchmarking across domains including finance, chemistry, genomics, and social science.

• Graph datasets: OGB (Open Graph Benchmark, molecular and citation graphs), TUDatasets (60+ graph classification datasets), SNAP Datasets (social network graphs)—used for graph neural network Benchmarking.

• Synthetic datasets: Generated via GANs, diffusion models (Stable Diffusion, DALL-E 3), or physics simulation engines to supplement real-world data in scarce or privacy-sensitive domains. Synthetic Data can be used for data augmentation, domain randomisation (robotics), or privacy-preserving surrogates, but introduces the risk of model collapse when models are iteratively trained on self-generated data (Shumailov et al., 2023).

• Federated learning benchmarks: LEAF (Caldas et al., 2018—Shakespeare, FEMNIST, CelebA), FLamby (du Terrail et al., 2022—medical imaging across institutions)—designed for Federated Learning evaluation where data is distributed across heterogeneous client partitions without centralisation.

• Reinforcement learning environments: Atari Learning Environment (57 Atari games, Bellemare et al., 2013), MuJoCo continuous control tasks (OpenAI Gym), D4RL (offline RL datasets from expert and sub-expert policies)—datasets generated from environment interactions for Reinforcement Learning training and evaluation.

  Formal Analysis

  A dataset can be formally characterised as a finite sample D = {(x_i, y_i)}_{i=1}^{N} drawn from an underlying joint distribution P(X, Y), where X is the input space and Y is the label space (with Y = ∅ for unsupervised tasks). The quality of the dataset as a learning resource is determined by several properties:

  Representativeness: The empirical distribution P̂(X) = (1/N) Σ δ(x - x_i) should approximate P(X) such that the expected loss under P̂ approximates the expected loss under P. When D is an unbiased sample from P, the generalisation gap—the difference between training and test error—decreases as O(1/√N) under uniform convergence bounds (Vapnik, 1998). However, real datasets violate the i.i.d. assumption through systematic collection biases, geographic imbalances, and temporal drift, causing the empirical distribution to diverge from the deployment distribution.

  Label quality: In the noisy label setting, observed labels ỹ are related to true labels y through a noise transition matrix T where T_{jk} = P(ỹ = k | y = j). Confident Learning (Northcutt et al., 2021) estimates T from out-of-sample predicted probabilities, enabling identification and removal of mislabelled examples. The noise rate for class j is ε_j = Σ_{k≠j} T_{jk}. Learning with label noise reduces to minimising a noise-corrected loss L̃(f) = Σ_{j,k} T_{jk}^{-1} P̂(ỹ=k) L(f(x), j), which is unbiased for the clean distribution under mild conditions.

  Diversity and coverage: Effective datasets must cover the support of the deployment distribution. Diversity metrics such as dataset coverage (fraction of the input space manifold covered), core-set diversity (maximum minimum pairwise distance), and task-specific diversity scores (linguistic diversity for NLP, pose diversity for vision) quantify how well the dataset spans the variation expected at inference time. Active learning and core-set selection algorithms optimise diversity under annotation budget constraints.

  Benchmark validity: The Goodhart’s Law problem in benchmarking—once a measure becomes a target, it ceases to be a good measure—is acute for ML datasets. As models are specifically optimised for benchmark performance, benchmark accuracy becomes an increasingly poor proxy for genuine capability. Dynamic benchmarks (Dynabench, HELM Living), contamination detection methods, and held-out evaluation protocols are responses to this validity threat.

  Academic Context

  The academic study of datasets as first-class research objects—rather than incidental infrastructure for model research—is associated with the data-centric AI movement and the NeurIPS Datasets and Benchmarks track (inaugurated 2021 as a dedicated track, previously a workshop). This shift was motivated by the observation that many real-world ML deployment failures trace to data problems rather than model architecture choices (Sambasivan et al., 2021, “Everyone wants to do the model work, not the data work”), and that the academic community had systematically underinvested in dataset documentation, quality assurance, and provenance tracking.

  Seminal works defining the field include the Datasheets for Datasets framework (Gebru et al., 2018/2021, CACM), which provided the first systematic framework for dataset documentation; the Dataset Nutrition Label (Holland et al., 2020), which used the nutritional metaphor to make dataset properties legible to non-technical stakeholders; Data Statements for NLP (Bender & Friedman, 2018, TACL), addressing the documentation gap specifically for language data; and Pervasive Label Errors in Test Sets (Northcutt et al., 2021, NeurIPS D&B), which quantified annotation errors across 10 canonical benchmarks and motivated systematic label quality auditing. The FAccT conference (ACM Fairness, Accountability, and Transparency) has published extensively on Bias in training datasets and the downstream fairness implications of dataset composition choices.

  Research on large-scale data curation includes DataComp (Gadre et al., 2023), which provided controlled comparisons of filtering strategies across fixed model and compute budgets; the Data Provenance Initiative (Longpre et al., 2023), which conducted a large-scale audit of dataset licensing and attribution; and DCLM (Li et al., 2024), which established a systematic approach to web data curation for language model pre-training. The Croissant metadata format (Akhtar et al., 2024) represents the current state of the art in machine-readable dataset documentation.

  Key research groups include the ML Collective (Northcutt, Mu), CMU’s data science group, Stanford’s CRFM (Center for Research on Foundation Models, Bommasani et al.), the MIT Data Provenance Initiative (Longpre et al.), and the Alan Turing Institute’s Data-Centric Engineering programme (UK). The LAION association (Germany) has produced several of the most widely used open multimodal pre-training datasets.

  Current Landscape (2026)

  By 2026, the dataset landscape is shaped by several intersecting trends that reflect the maturation of machine learning from academic research into production-scale infrastructure:

  Scale and foundation model demands: GPT-4 class models were trained on datasets exceeding 10 trillion tokens; recent models reportedly use 15–30 trillion token datasets. The transition to multi-modal foundation models requires similarly scaled image-text, audio-text, and video-text corpora. Common Crawl remains the dominant text source, processed through curation pipelines (C4, FineWeb, DCLM) applying quality filters, deduplication, and language detection at petabyte scale. Data scarcity for specific domains—medical imaging, low-resource languages, specialised scientific literature—continues to drive Active Learning, Synthetic Data generation, and Federated Learning approaches as complements to web-scale crawling.

  Data quality as first-class research: The data-centric AI movement has shifted the field’s attention from model architecture to dataset quality. Automated data cleaning tools (Cleanlab Studio, Great Expectations, Monte Carlo Data) and dataset curation agents benchmarked by DCA-Bench (Guo et al., 2024, KDD 2025) are entering production pipelines. A 2024 analysis of 60 NeurIPS D&B track submissions found labelling error rate estimates of 3%–50% across commonly used benchmarks, motivating systematic cleaning and consensus annotation. The Croissant machine-readable metadata format, adopted by Hugging Face, Kaggle, and OpenML in 2024, enables automated compatibility checking and regulatory compliance documentation.

  Legal and regulatory pressure: The EU AI Act (2024, entering full force 2026) mandates data governance documentation for high-risk AI systems under Article 10, requiring that training, validation, and test datasets meet data quality criteria, be characterised for representational biases, and be subject to appropriate data management practices. The UK Data (Use and Access) Act 2025 introduces the National Data Library concept and reforms data-sharing frameworks, while the ICO’s updated AI and data protection guidance addresses consent for model training and bias documentation requirements. Copyright litigation against AI companies over training data—Getty Images v. Stability AI, The New York Times v. OpenAI—is reshaping the legal landscape for web-scraped datasets and has caused several dataset releases to be retracted or restricted post-publication.

  Benchmark saturation and contamination: Many established academic benchmarks show near-ceiling performance—state-of-the-art models score above 90% on GLUE and many SuperGLUE tasks—driving investment in harder, dynamic, and contamination-resistant benchmarks (BIG-Bench Hard, MMLU-Pro, HELM Living, LiveBench 2025). Test set contamination—where benchmark examples appear verbatim or paraphrased in model pre-training corpora—is an unresolved methodological problem affecting fair comparison of models trained on different data mixtures; contamination detection tools (Min-K% Prob, membership inference tests) are active areas of development.

  Synthetic data proliferation and model collapse risk: Diffusion models (Stable Diffusion, DALL-E 3, Midjourney) and instruction-tuned LLMs are increasingly used to generate synthetic training data for tasks where real annotation is expensive (medical imaging, robot manipulation, low-resource languages). However, model collapse—progressive performance degradation when models are trained on iteratively self-generated synthetic data—has been demonstrated empirically (Shumailov et al., 2023) and is a recognised risk in synthetic data pipelines. Hybrid approaches combining small amounts of high-quality real data with large amounts of carefully filtered synthetic data are emerging as the practical compromise.

  UK Context

  The United Kingdom has significant national infrastructure, academic capacity, and regulatory activity around dataset creation, curation, and governance.

  National institutions and research programmes: The Alan Turing Institute (ATI), as the UK’s national institute for data science and AI, hosts a Data-Centric Engineering programme funded in partnership with Lloyd’s Register Foundation since 2015, focused on engineering applications of data science including dataset quality for safety-critical systems. The ATI has partnered with the University of Edinburgh, UCL, and the Arts and Humanities Research Council on AI research incorporating humanities data. The ATI also leads the UK’s participation in international data governance initiatives and has published guidance on responsible AI development practices including dataset documentation requirements.

  UK Biobank: One of the world’s most valuable longitudinal research datasets, containing genetic (whole-genome sequencing and genotyping arrays), imaging (brain MRI, cardiac MRI, DEXA scans), health record linkage, and accelerometry data from approximately 500,000 UK participants aged 40–69 at recruitment. Based in Stockport (Greater Manchester) with operations at the Wellcome Centre for Human Genetics in Oxford, UK Biobank is a globally critical resource for medical AI research, with datasets licensed to over 30,000 registered researchers at 6,000+ institutions in 100+ countries. Access requires a data access application process reviewed by UK Biobank’s Access Committee, reflecting the sensitivity of health data.

  NHS and population health data: NHS England manages large-scale linked health datasets including the General Practice Extraction Service (GPES) data (covering 60+ million patients), Secondary Uses Service (SUS) hospital records, and the National Cancer Registration and Analysis Service. The NHS generates approximately 65 million patient records and is a primary motivation for Federated Learning approaches—NHS DigiTrials and the ODAP (OpenSAFELY Approved Datasets Programme) provide secure remote access to de-identified NHS data for approved researchers without data leaving NHS infrastructure. The COVID-19 pandemic accelerated NHS data sharing through the COPI (Control of Patient Information) notices, generating precedent for emergency data access frameworks.

  Academic dataset creation: The University of Edinburgh’s ILCC (Institute for Language, Cognition and Computation) and CDT in Natural Language Processing produce major NLP datasets and have contributed to multilingual evaluation benchmarks. Imperial College London and UCL contribute biomedical and clinical imaging datasets, with Imperial’s BioMedIA group creating several open medical image segmentation benchmarks. The University of Oxford’s Torr Vision Group produces robotics and video understanding datasets. The University of Manchester, with its history of knowledge representation and linked data research (including the original Semantic Web work), contributes ontology and knowledge graph datasets.

  Industrial data assets: Northern England manufacturing and logistics sectors—including automotive suppliers in the West Midlands, steel production in Sheffield and Scunthorpe, and port logistics at Grimsby and Immingham—generate industrial IoT time-series datasets increasingly used for predictive maintenance AI. The Hartree Centre (Daresbury, Cheshire, STFC) provides national HPC infrastructure used for large-scale dataset preprocessing and foundation model training. Energy sector datasets from National Grid ESO and Ofgem are being used for demand forecasting and grid stability AI models.

  Regulatory governance: The UK Information Commissioner’s Office (ICO) published detailed guidance on AI and data protection in 2023–2024, including requirements for dataset bias characterisation, documentation of data sources, and consent mechanisms for model training. The UK AI Safety Institute (AISI), established in October 2023, conducts evaluations of frontier AI models’ training data practices as part of its safety assessment framework. The UK’s post-Brexit divergence from EU data governance creates regulatory complexity for organisations deploying datasets across UK/EU boundaries.

  Future Directions (2026–2030)

• Autonomous dataset curation: LLM-powered curation agents (benchmarked by DCA-Bench, 2024) will automate quality assessment, deduplication, bias detection, and documentation generation, dramatically reducing the human annotation bottleneck. Agents will iteratively sample from candidate corpora, assess quality using learned quality signals, and build curated datasets to specified quality targets within compute budgets.

• Dynamic and adversarial benchmarks: Static test sets will increasingly be replaced by dynamically generated evaluation tasks that cannot be memorised during pre-training. Dynabench (Kiela et al., 2021) and LiveBench (White et al., 2024) represent early implementations; by 2028, most competitive NLP and vision benchmarks will use dynamic test generation or frequent dataset refresh cycles to prevent contamination.

• Cryptographic provenance and data lineage: Blockchain-anchored or cryptographically signed provenance registries will emerge to address legal uncertainty around web-scraped training data challenged by copyright litigation. Watermarking techniques applied at data collection time will enable tracing of specific content through training pipelines, supporting both copyright enforcement and contamination detection.

• Federated data markets: Formal mechanisms for pricing, accessing, and trading data assets—the European Health Data Space, the UK National Data Library, and commercial data trusts—will enable controlled data sharing across organisational boundaries while preserving Privacy through Differential Privacy guarantees and federated access protocols.

• Croissant and machine-readable dataset metadata at scale: The Croissant metadata format will become the universal standard for dataset description, enabling automated dataset discovery, compatibility matching, compliance checking, and integration into data pipelines without manual configuration.

• Multi-modal and embodied datasets: Robotics and embodied AI research will drive demand for large-scale datasets integrating egocentric vision, proprioception, language instructions, and action sequences (Open X-Embodiment, DROID). Simulation-to-real transfer will require paired sim and real datasets with domain gap characterisation.

• Model collapse mitigation: As synthetic data proliferates, techniques for detecting and preventing model collapse—mixing requirements, provenance-aware training curricula, and collapse detection metrics—will become standard components of large-scale training pipelines.

  Data-Centric AI: Principles and Practice

  The data-centric AI paradigm—attributed primarily to Andrew Ng (2021) but with intellectual antecedents in the database systems community’s work on data quality and data cleaning—asserts that for most practical ML deployments, the greatest leverage comes from improving training data quality rather than model architecture. This represents a significant shift from the model-centric paradigm that dominated academic ML from 2012–2020, in which standard benchmark datasets were held fixed while researchers competed to develop better model architectures and training algorithms.

  Data-centric AI encompasses several distinct practices that collectively constitute the engineering of a high-quality training dataset: (1) systematic label quality auditing and correction using Confident Learning or consensus annotation; (2) deduplication at both exact-match and near-duplicate levels to prevent memorisation and evaluation contamination; (3) class balance management through stratified sampling, oversampling, or class-weighted loss functions; (4) coverage analysis to identify underrepresented subpopulations that will be poorly served by the trained model; (5) feature relevance analysis to identify and remove spuriously correlated features that will degrade out-of-distribution generalisation; (6) distributional shift monitoring between training data collection periods and deployment contexts; and (7) Data Augmentation to synthetically expand the effective training set size through structure-preserving transformations (image flips, crops, colour jitter; text paraphrasing, back-translation; audio pitch shifting, time stretching).

  The Active Learning paradigm provides a data-centric approach to annotation budget allocation: rather than labelling a random sample, an active learning system iteratively selects the examples most informative for the current model state—typically those near the current decision boundary or those for which model uncertainty is highest—and requests labels only for those examples. This can achieve equivalent model performance with substantially fewer labelled examples, typically reducing annotation requirements by 50–90% for classification tasks. Active learning is particularly valuable in domains where annotation is expensive—medical imaging requiring radiologist review, audio transcription requiring expert linguists, scientific data requiring domain specialist labelling.

  The Data Pipeline infrastructure supporting dataset creation and consumption has become a major engineering discipline. Data pipelines automate the flow from raw source data through preprocessing, validation, feature computation, splitting, and loading into training frameworks. Modern data pipeline frameworks (Apache Beam, Spark, dbt, Prefect, Airflow) provide scalable, fault-tolerant execution with monitoring and lineage tracking. The ML-specific extensions (TFX—TensorFlow Extended, Metaflow, ZenML) add model training and evaluation steps while maintaining end-to-end data lineage from raw source to trained model artefact. This lineage is the technical substrate for Data Provenance documentation required by regulators and dataset users.

  Dataset Quality Assessment and Measurement

  Quantifying dataset quality is a prerequisite for data-centric AI improvement cycles. Quality assessment operates at multiple levels:

  Label quality metrics measure the accuracy, consistency, and completeness of annotations. Inter-annotator agreement (Cohen’s Kappa, Fleiss’ Kappa, Krippendorff’s Alpha) quantifies annotation consistency across multiple annotators for the same example. Kappa values above 0.80 indicate strong agreement; values below 0.60 suggest annotation guidelines are insufficient or the task is inherently ambiguous. Confident Learning estimates of mislabelling rates provide dataset-level quality scores that can be tracked across dataset versions. High Imbalanced Data ratios (>100:1 between majority and minority class) indicate collection biases that require targeted supplementation.

  Diversity and coverage metrics assess how well the dataset spans the input space. Coverage is often estimated via clustering: the dataset is clustered into K clusters (e.g., K-means over feature embeddings), and coverage is the fraction of clusters with at least a minimum number of examples. Low-coverage clusters correspond to underrepresented subpopulations. For NLP datasets, diversity metrics include vocabulary diversity (type-token ratio, distinct-n), syntactic diversity (parse tree depth distribution), and topic diversity (LDA-derived topic proportions). For image datasets, semantic diversity metrics using CLIP embeddings assess visual concept coverage.

  Distributional shift detection compares the training data distribution to validation, test, or production distributions using statistical tests (two-sample Kolmogorov-Smirnov, MMD—Maximum Mean Discrepancy) or divergence measures (KL divergence, Jensen-Shannon divergence estimated from histogram approximations or kernel density estimates). When distributional shift exceeds thresholds, model performance degradation is predictable and data collection should be directed at the shifted distribution.

  Benchmark validity is increasingly assessed through contamination detection, testing whether test examples or close paraphrases appear in model pre-training data. Min-K% Probability (Shi et al., 2024) estimates membership of a text example in a language model’s training data using the observation that memorised sequences have anomalously high token-level probabilities. Contamination rates above 5% in test sets raise serious concerns about the validity of reported benchmark performance.

  Data efficiency curves plot model performance as a function of training set size, estimating the marginal value of additional data and identifying the regime where data collection investment has diminishing returns. Power-law scaling relationships between dataset size and model performance (Kaplan et al., 2020; Hoffmann et al., 2022—Chinchilla) enable data-compute trade-off optimisation at foundation model scale.

  Large-Scale Pre-Training Corpora: Architecture and Curation

  The construction of pre-training corpora for Large Language Models and Foundation Model systems represents the most technically demanding and consequential instance of dataset engineering. The scale involved—trillions of tokens, petabytes of raw data—requires fully automated curation pipelines with multiple quality filtering stages, and the downstream consequences of curation decisions propagate through every model trained on the corpus.

  The Common Crawl corpus, the primary source for most LLM pre-training data, is a monthly snapshot of 3–5 billion web pages, accumulated over years into a multi-petabyte archive. Raw Common Crawl data has estimated quality of approximately 65% usable content after removing blank pages, near-duplicates, and pages with severe formatting errors. Curation pipelines applied to Common Crawl to produce model-ready datasets include: C4 (Colossal Clean Crawled Corpus, Raffel et al., 2020)—produced by removing pages not passing language identification, minimum sentence length thresholds, and profanity filters; The Pile (Gao et al., 2020)—a diverse 825 GB corpus combining Common Crawl with domain-specific high-quality sources (PubMed, arXiv, GitHub, Books3); FineWeb (Penedo et al., 2024, Hugging Face)—a 15 trillion token corpus produced by applying extensive quality filtering including URL filtering, text quality scoring, and aggressive near-deduplication, achieving substantially better downstream model quality than C4 per token.

  Near-deduplication—removing near-duplicate documents sharing substantial content but not byte-identical—is a critical step that substantially improves generalisation and prevents memorisation of frequently repeated content. MinHash-based deduplication provides efficient approximate set similarity detection at scale; deduplication reduces Common Crawl corpora by 15–30% and can improve downstream model perplexity by 5–10%. Content filtering—removing harmful, offensive, or legally risky content—is technically challenging: ML-based content classifiers trained on human-labelled examples achieve better precision-recall trade-offs than keyword blocklists but require ongoing update. The Data Composition Mixture problem—determining the optimal mixture of data sources (web, code, books, scientific papers, multilingual text) for pre-training—is an active research area with significant practical implications; DataComp-LM (2024) and DCLM (Li et al., 2024) provide controlled benchmarks for comparing curation strategies at LLM pre-training scale.

  Scaling law research (Kaplan et al., 2020; Hoffmann et al., 2022—Chinchilla scaling laws) has established that optimal compute allocation between model size and dataset size follows a roughly equal split: a model trained optimally for a given compute budget should scale model parameters and training token count approximately equally. The Chinchilla result suggested that most large models in 2020–2021 were significantly under-trained relative to model size, and that training a smaller model on more tokens can outperform a larger model trained on fewer tokens for the same compute budget. This insight has shifted pre-training dataset scale upward dramatically, increasing demand for high-quality, large-scale web-crawled and curated corpora and intensifying the data-compute trade-off optimisation problem.

  Privacy and copyright constraints are the dominant legal challenges for large-scale dataset construction. Web-scraped corpora inevitably contain personal information (names, addresses, health information, financial details shared publicly), which raises GDPR and UK GDPR compliance issues for European users. Copyright litigation from authors (Sarah Silverman et al. v. Meta), publishers (The New York Times v. OpenAI), and image rights holders (Getty Images v. Stability AI) has created legal uncertainty about the permissibility of using copyrighted content for model training without consent or licence. The EU AI Act (2024) requires providers of general-purpose AI models (GPAI) with systemic risk to publish detailed documentation of their training data including copyright compliance measures. Differential Privacy techniques applied at training time can provide formal guarantees that the trained model does not memorise specific training examples, providing some protection against individual privacy claims while incurring a performance cost.

  Ethical Dimensions of Dataset Construction

  Dataset construction involves ethical choices that propagate into the values and biases of models trained on the data, with real-world consequences for the populations those models serve. Several ethical dimensions require explicit attention during dataset design and curation:

  Representation and inclusion: Datasets assembled from digitised text and images over-represent English, Western European, and North American perspectives and underrepresent the global majority of the world’s population. Languages spoken by billions of people—Hindi, Swahili, Bengali, many African and Southeast Asian languages—are dramatically underrepresented in Common Crawl relative to their speaker populations. This creates models that perform substantially worse for speakers of underrepresented languages, perpetuating existing digital divides. The Common Voice project (Mozilla) specifically targets representation gaps in speech datasets by crowdsourcing voice recordings across 100+ languages; the Masakhane NLP project addresses African languages specifically.

  Consent and participation: The people whose text, images, and voices constitute training datasets rarely gave explicit consent for this use. Scraping public social media posts, web pages, and image repositories creates training data without the knowledge of the people whose expressions are captured. This is particularly acute for medical data, where patients have strong expectations of privacy and data protection law requires explicit consent or lawful basis for processing. The Data Provenance Initiative (Longpre et al., 2023) found that the majority of widely used NLP datasets lack clear consent documentation for their source content.

  Data Ethics and harm: Training datasets may contain hate speech, misinformation, harassment, and content that could teach models harmful behaviours or reinforce stereotypes. Birhane et al. (2021) found disturbing content in LAION-400M including non-consensual intimate imagery and stereotyping. Removing such content requires clear ethical frameworks for what constitutes harmful content—which are themselves contested—and creates trade-offs between comprehensiveness and safety. The field is moving toward explicit harm taxonomies and red-teaming processes that systematically test dataset content for the presence of defined harm categories.

  Worker welfare in annotation: Many large annotated datasets are produced through crowdsourcing platforms (Amazon Mechanical Turk, Scale AI, Sama, Appen) using workers in developing countries paid below living wages for cognitively and emotionally demanding annotation work. Reviewing content moderation datasets for social media, medical imaging datasets for pathology, and instruction-following datasets for alignment training frequently exposes annotators to disturbing material without adequate psychological support. The ethical obligations of dataset creators toward annotation workers are an emerging area of research and advocacy.

  Dataset Versioning, Lineage, and Lifecycle Management

  The lifecycle of a dataset—from initial collection through multiple versions, curation passes, usage across many models, and eventual deprecation—requires systematic management infrastructure analogous to the software development lifecycle. Without rigorous dataset versioning and lineage tracking, reproducing past experiments becomes impossible, debugging dataset-induced model failures becomes intractable, and compliance with Regulation requiring audit trails of training data becomes unachievable.

  Dataset versioning tracks changes to a dataset’s content across releases. Semantic versioning conventions (MAJOR.MINOR.PATCH) applied to datasets convey the nature of changes: a patch release corrects identified label errors without adding or removing records; a minor release adds new records or annotations; a major release changes the schema, splits, or collection methodology in ways that make the dataset non-backward-compatible. Tools including DVC (Data Version Control), Pachyderm, LakeFS, and Delta Lake provide git-like versioning semantics for large datasets, enabling checkout of arbitrary historical versions, diff computation between versions, and branching for experimental curation changes.

  Data Lineage tracking records the provenance chain from raw source data through every transformation step to the final dataset version consumed by a model. A complete lineage record enables answering questions such as: which model versions were trained on which dataset versions; which records in the current dataset originated from which source documents; what preprocessing transformations were applied and with what parameters; and whether a specific record flagged as problematic during model incident investigation should be removed from the training set. Data Lineage is the technical implementation of the accountability principle under GDPR and the traceability requirements under the EU AI Act.

  Data cataloguing: Large organisations managing many datasets require Data Catalogue infrastructure—a metadata repository enabling dataset discovery, quality assessment, access control, and governance documentation. Enterprise data catalogue platforms (Alation, Collibra, Atlan, Google Data Catalog, Azure Purview) provide search, data profiling, business glossary mapping, and data stewardship workflows. ML-specific catalogues (MLflow Artifacts, Weights & Biases Artifacts, Hugging Face Datasets Hub) add model-dataset linkage, enabling tracking of which model was trained on which dataset version.

  Dataset deprecation: As better-curated alternatives become available or as known quality issues make a dataset unsuitable for continued use, datasets must be deprecated in a way that does not break downstream users who have hard-coded dependencies. Dataset deprecation policies should include: announcement timelines giving downstream users adequate time to migrate; documentation of the reasons for deprecation; pointers to recommended replacement datasets; and archival of the deprecated dataset for reproducibility purposes. The Hugging Face Datasets Hub provides deprecation markers and structured migration guidance for deprecated datasets.

  Experiment tracking and reproducibility: Linking dataset versions to model training runs and evaluation results is the prerequisite for Reproducibility in ML research. Experiment tracking platforms (MLflow, Weights & Biases, Comet ML, Neptune) record dataset version identifiers alongside hyperparameters, model architecture specifications, training compute, and evaluation metrics, enabling reconstruction of the full experimental context for any reported result. Reproducibility requirements are increasingly enforced by top ML conferences (NeurIPS, ICML, ICLR) through supplementary material checklists and compute resource disclosure requirements.

  Dataset Interoperability and Standards

  The Data Architecture ecosystem has developed several standards and interchange formats that promote dataset interoperability across frameworks, organisations, and jurisdictions:

• Apache Parquet: The dominant columnar storage format for large tabular datasets, widely supported by Spark, Pandas, Arrow, and cloud data warehouses. Parquet’s columnar layout enables efficient predicate pushdown and projection pruning, dramatically reducing I/O for analytical queries over large datasets.

• Apache Arrow: An in-memory columnar data format providing zero-copy data exchange between analytics frameworks (Pandas, Spark, DuckDB, database connectors). The Hugging Face Datasets library uses Arrow as its internal format, enabling efficient loading of large NLP datasets with memory-mapped access.

• TFRecord: TensorFlow’s binary record format, optimised for reading sequential mini-batches during neural network training. TFRecords support both fixed-length and variable-length features and are efficient for image, text, and audio data at scale.

• WebDataset: A format wrapping TAR archives as streaming datasets, enabling efficient reading from cloud object storage (S3, GCS, Azure Blob) during training without full local materialisation. Used extensively for LAION-5B and other large-scale vision datasets.

• Croissant: A machine-readable metadata format (2024, adopted by Hugging Face, Kaggle, and OpenML) describing dataset structure, features, splits, and licensing in JSON-LD. Croissant metadata enables automated tooling for dataset loading, compatibility checking, and compliance documentation generation.

• DCAT (Data Catalog Vocabulary): W3C standard for describing datasets in RDF, enabling Knowledge Graph integration and semantic search over dataset metadata. Used by open government data portals and research data repositories.

• Schema.org Dataset: JSON-LD markup standard enabling dataset metadata embedding in web pages, supporting search engine indexing of datasets for discovery. Google Dataset Search indexes Schema.org Dataset markup from publisher websites.

• ISO 8000 / ISO 25012: International standards for data quality specification and measurement, providing a formal framework for the quality dimensions (accuracy, completeness, consistency, timeliness) applicable to datasets used in Machine Learning and analytics contexts.

  Dataset Regulation and Legal Landscape

  The legal environment governing dataset creation, distribution, and use for AI training is among the most rapidly evolving areas of technology law. Multiple distinct legal frameworks intersect:

  Copyright and intellectual property: Copyright protects the expression of ideas in creative works—text, images, music, code—but not the underlying facts or information. Training a model on copyrighted content may or may not constitute infringement depending on jurisdiction, the nature of the use, and whether the trained model can reproduce the copyrighted expression. In the United States, fair use doctrine (§107 of the 17 USC Copyright Act) provides a flexible four-factor test; the case Authors Guild v. Google (2015) established that creating searchable indices of books constitutes fair use. In the UK, a text and data mining exception (§29A CDPA 1988) allows copyright works to be copied for computational analysis without permission, with post-Brexit amendments broadening this exception in the Enterprise and Regulatory Reform Act 2023. The EU AI Act (2024) requires GPAI model providers to document copyright compliance, including opt-out mechanisms for rights holders who object to their works being used for training.

  Data protection law: Personal data processed for model training is subject to GDPR (EU), UK GDPR (UK), and equivalent legislation in other jurisdictions. Key obligations include identifying a lawful basis for processing (Article 6, e.g., legitimate interests, consent, public task), ensuring that special category data (health, biometric, racial/ethnic origin) meets the higher threshold of Article 9 lawful bases, implementing data minimisation and purpose limitation, and providing transparency to data subjects about how their data is used. The use of web-scraped personal data for model training has been challenged by data protection authorities: Italy’s Garante blocked ChatGPT in March 2023; France’s CNIL investigated several AI companies; the ICO has published guidance on the lawful bases for scraping and training.

  AI-specific regulation: The EU AI Act (2024) introduces the first comprehensive regulation specifically addressing AI training data quality. Article 10 (Data and Data Governance) applies to providers of high-risk AI systems and requires: (a) training, validation, and test datasets meeting “appropriate data governance and management practices” including examination for biases and errors; (b) characterisation of the datasets with respect to their purpose, geographic, functional, or behavioural setting; (c) assessment of the availability, quantity, and suitability of the data; and (d) examination of the datasets for possible biases that could affect health, safety, or fundamental rights. The EU AI Act’s GPAI provisions additionally require that foundation model providers publish detailed summaries of their training data including copyright compliance measures.

  Research exemptions and open data policies: Many jurisdictions provide research exemptions from data protection law (GDPR Article 89, UK GDPR Schedule 2 para 27) enabling processing of personal data for scientific research without individual consent, subject to appropriate safeguards. These exemptions underpin the academic research datasets in health (UK Biobank, MIMIC) and social science (UK Data Service, UK Data Archive). Government Open Data policies—the G8 Open Data Charter, the UK’s National Data Strategy, the EU Open Data Directive—require public sector organisations to publish datasets as open data by default, creating a significant resource for AI research and innovation.

  Export controls on AI datasets: Certain datasets containing sensitive information—satellite imagery of military installations, dual-use technical specifications, controlled biological or chemical data—may be subject to export control regulations (EAR in the US, analogous controls in the UK and EU) that restrict sharing with foreign persons or entities. AI-generated datasets derived from controlled source data may themselves be subject to export restrictions, creating compliance challenges for international research collaborations.

  Key Challenges and Open Problems

  The field of dataset engineering faces several persistent challenges that represent active research frontiers as of 2026:

  The annotation scalability challenge: High-quality human annotation does not scale to the volume required for foundation model training. Weak supervision (Ratner et al., 2020, Snorkel), programmatic labelling through labelling functions, and LLM-assisted annotation are partial solutions, but all introduce systematic errors. The development of robust methods for learning from noisy, weakly supervised, and partially labelled data remains a central challenge.

  The distribution shift detection challenge: Real-world deployment populations shift over time as user behaviour, product offerings, and macroeconomic conditions change, causing trained models to become miscalibrated. Efficient, reliable detection of distribution shift without labels—using only input feature statistics—is an unsolved problem. Methods including Maximum Mean Discrepancy testing, adversarial discriminators, and density ratio estimation provide partial solutions but require careful calibration to avoid excessive false alarms.

  The benchmark saturation and Goodhart’s law challenge: As models are specifically optimised for benchmark performance, benchmark scores increasingly fail to reflect genuine capability. Addressing this requires continuous development of new, harder benchmarks; dynamic evaluation that cannot be gamed through test set memorisation; and evaluation methodologies focused on capability rather than benchmark score.

  The catastrophic forgetting challenge in dataset streams: When models are continuously trained on new data batches (continual learning), they tend to forget previously learned information from older batches—catastrophic forgetting (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017 Elastic Weight Consolidation). Maintaining performance across the full distribution of past and present data while absorbing new information is a fundamental challenge for continually updated AI systems.

  The privacy-utility trade-off: Differential Privacy mechanisms that prevent training data memorisation degrade model performance, with the trade-off depending on the privacy budget ε. At privacy budgets strong enough to provide meaningful protection (ε ≤ 8), utility degradation is substantial for many tasks. Developing techniques that achieve better privacy-utility trade-offs—through improved noise mechanisms, better accounting methods, or architectural choices that reduce sensitivity to individual examples—is an active research area.

  The multimodal alignment problem: In multimodal datasets, ensuring that paired records across modalities (image-text, video-audio, document-summary) genuinely correspond semantically is non-trivial at web scale. Automated alignment quality filtering using cross-modal similarity models (CLIP for image-text) achieves moderate precision but misses subtle misalignments. Improving multimodal alignment quality at scale without prohibitive annotation cost remains an open challenge.

  Research & Literature

  1. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database. CVPR 2009. https://doi.org/10.1109/CVPR.2009.5206848
  2. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723
  3. Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2020). The Dataset Nutrition Label. Data Protection and Privacy, 12, 1–33.
  4. Northcutt, C. G., Athalye, A., & Mueller, J. (2021). Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS 2021 Datasets and Benchmarks Track. https://arxiv.org/abs/2103.14749
  5. Schuhmann, C., et al. (2022). LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. NeurIPS 2022. https://arxiv.org/abs/2210.08402
  6. Bender, E. M., & Friedman, B. (2018). Data Statements for Natural Language Processing. TACL, 6, 587–604. https://doi.org/10.1162/tacl_a_00041
  7. Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017). Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. ICCV 2017. https://arxiv.org/abs/1708.02862
  8. Zha, D., et al. (2023). Data-Centric Artificial Intelligence: A Survey. arXiv. https://arxiv.org/abs/2303.10158
  9. Akhtar, M., et al. (2024). Croissant: A Metadata Format for ML-Ready Datasets. arXiv. https://arxiv.org/abs/2403.19546
  10. Lhoest, Q., et al. (2021). Datasets: A Community Library for Natural Language Processing. EMNLP 2021. https://arxiv.org/abs/2109.02846
  11. Wang, A., et al. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. NeurIPS 2019. https://arxiv.org/abs/1905.00537
  12. Lin, T.-Y., et al. (2014). Microsoft COCO: Common Objects in Context. ECCV 2014. https://arxiv.org/abs/1405.0312
  13. Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140), 1–67.
  14. Shumailov, I., et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv. https://arxiv.org/abs/2305.17493
  15. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. https://doi.org/10.1145/3442188.3445922
  16. Birhane, A., et al. (2021). Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes. arXiv. https://arxiv.org/abs/2110.01963
  17. Dodge, J., et al. (2021). Documenting Large Webtext Corpora. EMNLP 2021. https://arxiv.org/abs/2104.08758
  18. Hutchinson, B., et al. (2021). Towards Accountability for Machine Learning Datasets. FAccT 2021. https://doi.org/10.1145/3442188.3445918
  19. Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021). Data and Its (Dis)Contents. Patterns, 2(11). https://doi.org/10.1016/j.patter.2021.100336
  20. Caldas, S., et al. (2018). LEAF: A Benchmark for Federated Settings. arXiv. https://arxiv.org/abs/1812.01097
  21. Bommasani, R., et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM. https://arxiv.org/abs/2108.07258
  22. Guo, C., et al. (2024). DCA-Bench: A Benchmark for Dataset Curation Agents. Presented at KDD 2025. https://arxiv.org/abs/2406.07275
  23. Longpre, S., et al. (2023). The Data Provenance Initiative: A Large-Scale Audit of Dataset Licensing and Attribution in AI. arXiv. https://arxiv.org/abs/2310.16787
  24. Sambasivan, N., et al. (2021). “Everyone Wants to Do the Model Work, Not the Data Work.” CHI 2021. https://doi.org/10.1145/3411764.3445518
  25. Koch, B., Denton, E., Hanna, A., & Foster, J. G. (2021). Reduced, Reused, and Recycled: The Life of a Dataset in Machine Learning Research. NeurIPS 2021 Datasets and Benchmarks. https://arxiv.org/abs/2112.01716
  26. Gadre, S. Y., et al. (2023). DataComp: In Search of the Next Generation of Multimodal Datasets. NeurIPS 2023. https://arxiv.org/abs/2304.14108
  27. Moor, M., et al. (2023). Foundation Models for Generalizable Disease Segmentation. Nature Medicine. [The NCI ISIC, UK Biobank, and multi-site dataset context.] https://doi.org/10.1038/s41591-023-02625-3

  Variant Taxonomy

  The Dataset concept admits a rich taxonomy of specialised sub-types differentiated by construction methodology, access regime, and intended application:

• Training Data / training split: The largest partition of a labelled dataset, used to fit model parameters. The training set size is the primary determinant of model capacity utilisation, and the curation quality of the training set has outsized influence on generalisation performance relative to the validation or test partitions.

• Benchmark Dataset: A fixed, publicly released dataset paired with a standardised evaluation protocol and metric, used to compare model performance across research groups. Benchmark datasets serve a dual function as evaluation standards and as implicit training targets—once a benchmark becomes widely known, researchers implicitly optimise for it. Key benchmarks include MNIST (digit recognition), ImageNet (image classification), SQuAD (reading comprehension), GLUE/SuperGLUE (NLP multi-task), MMLU (knowledge-intensive multiple choice), and HumanEval (code synthesis).

• Open Data: Datasets released under permissive licences (Creative Commons CC-BY, CC0, Open Database License) enabling reuse, modification, and redistribution without restriction. Open datasets accelerate research but create legal risk when used for commercial model training if the licence terms are ambiguous or if the underlying data was itself generated from non-open sources.

• Proprietary dataset: A dataset held as a competitive asset by an organisation, not publicly released. Large proprietary datasets—Google’s internal search query logs, Meta’s social graph interaction data, Amazon’s purchase history corpus—confer significant advantage in training specialised models and are the subject of antitrust scrutiny in multiple jurisdictions.

• Web-scraped corpus: A dataset assembled by automated crawling of publicly accessible web content. Common Crawl is the canonical example; LAION-5B, The Pile, and FineWeb are curated subsets. Web-scraped corpora are subject to copyright uncertainty, as the legal status of model training on copyrighted web content is contested globally.

• Curated research dataset: A carefully assembled, human-reviewed dataset targeting a specific research task. Curated datasets typically have higher label quality and better documentation than web-scraped corpora but are orders of magnitude smaller. Examples include MNIST (60,000 images), CIFAR-10 (60,000 images), and SQuAD v2 (150,000 question-answer pairs).

• Longitudinal dataset: A dataset tracking the same subjects or entities over time, enabling analysis of temporal dynamics, treatment effects, and developmental trajectories. UK Biobank with its 500,000-participant longitudinal health tracking cohort and MIMIC-IV with longitudinal ICU patient records are examples. Longitudinal datasets enable causal inference analyses that cross-sectional snapshots cannot support.

• Synthetic dataset: A dataset generated programmatically through simulation, generative modelling (GANs, diffusion models, LLMs), or rule-based data generation, rather than collected from real-world observations. Synthetic datasets can be generated at arbitrary scale, preserve Privacy by construction, and can be engineered to have specific statistical properties, but may not fully capture the distribution of real-world data and carry model collapse risk.

• Federated dataset: A collection of locally held, non-centralised data partitions across multiple organisations or devices, used in Federated Learning settings where data cannot be shared due to privacy, regulatory, or competitive constraints. Federated datasets are characterised by non-IID (independently and identically distributed) partition structure, as different parties hold different subpopulations or measurement protocols.

• Multi-modal dataset: A dataset containing aligned records across multiple data modalities—image-text pairs (LAION-5B), video-audio-transcript triples (AudioSet, HowTo100M), or document-table pairs. Multi-modal datasets enable training of joint embedding models and cross-modal retrieval systems. Modality alignment—ensuring that the paired modalities genuinely correspond semantically—is a significant curation challenge.

• Instruction tuning / alignment dataset: A dataset of (instruction, response) pairs used for supervised fine-tuning of pre-trained Large Language Models to follow natural language instructions. Examples include FLAN (fine-tuned language model collection), Alpaca (generated from GPT-3), and high-quality human-curated sets such as OpenAssistant. The quality and diversity of instruction tuning data has a disproportionate impact on model helpfulness and safety relative to its size.

  Key Terminology

• Training set: The partition of a dataset used to fit model parameters during the learning algorithm. Comprises the majority (typically 60–80%) of labelled examples.

• Validation set: A held-out partition used to tune hyperparameters and monitor for Overfitting during training. Distinct from the test set to avoid implicit optimisation pressure on evaluation metrics.

• Test set: A held-out partition used exclusively for final evaluation of trained models; should be used only once to avoid test set contamination through hyperparameter tuning cycles.

• Data split: The division of a dataset into train/validation/test partitions, stratified to preserve class distribution. Cross-validation generalises this by rotating the validation set across k folds.

• Imbalanced Data: A dataset in which class labels are distributed non-uniformly, often severely so (e.g., fraud detection datasets where fraudulent transactions are 0.1% of samples). Imbalance causes naive classifiers to default to the majority class and is addressed through oversampling (SMOTE), undersampling, or class-weighted loss functions.

• Data leakage: The contamination of model training or validation with information that would not be available at inference time, including test set examples appearing in training data, future-looking features in time series, or label-derived features.

• Annotation artefact: A spurious statistical pattern in the dataset that correlates with labels due to annotation methodology rather than genuine signal—e.g., presence of camera watermarks as a proxy for certain classes. Models exploit annotation artefacts, producing high benchmark performance that fails to generalise.

• Datasheet: A structured documentation artefact (Gebru et al., 2021) accompanying a dataset, covering motivation, composition, collection process, preprocessing, uses, distribution, and maintenance. Analogous to a product specification sheet.

• Data card: Google’s variant of the datasheet framework, covering dataset description, intended use, source information, and known limitations. Used for documentation of Google’s publicly released datasets.

• Croissant: A machine-readable metadata format (2024) for ML-ready datasets, adopted by Hugging Face, Kaggle, and OpenML. Enables automated tooling for dataset discovery, loading, and compliance verification.

• Data Provenance: The documented lineage of data from collection source through all transformations to the final dataset. Critical for copyright compliance, regulatory accountability, and debugging of dataset-induced model failures.

  Benchmark Dataset Families and Their Roles

  The following canonical benchmark datasets define performance standards across major AI research domains. Each represents a carefully constructed evaluation resource that has shaped the development of its field:

  Natural Language Processing benchmarks:

• GLUE (Wang et al., 2018): 9-task multi-task NLP benchmark covering entailment, similarity, sentiment, grammar. Saturated by 2020 with models exceeding human performance.

• SuperGLUE (Wang et al., 2019): Harder successor to GLUE; 8 tasks requiring reading comprehension, co-reference, causal reasoning. Near-saturated by 2022.

• MMLU (Hendrycks et al., 2021): 57-subject multiple-choice covering STEM, law, medicine, social sciences. 14,000 questions. Top models score above 90% as of 2025.

• BIG-Bench Hard (Suzgun et al., 2022): 23 challenging tasks from BIG-Bench where average human performance exceeded prior LLM performance. Chain-of-thought prompting provides substantial benefit.

• HumanEval (Chen et al., 2021): 164 programming problems for code generation. Top models achieve over 90% pass@1 as of 2025.

• SQuAD v2 (Rajpurkar et al., 2018): Reading comprehension with unanswerable questions; 150,000 question-answer pairs over Wikipedia passages.

  Computer Vision benchmarks:

• ImageNet-1k ILSVRC: 1.28M training, 50K validation, 1,000 classes. Top-5 error reduced from 26.2% (2011) to under 1.5% (2023) by the best models.

• MS-COCO 2017: 123,000 images, 80 object categories with bounding boxes, keypoints, and captions. Standard for object detection and image captioning research.

• ADE20K: 25,000 images with pixel-level semantic segmentation across 150 categories. Standard for scene understanding and semantic segmentation.

  Multimodal benchmarks:

• VQAv2 (Goyal et al., 2017): 265,000 images with 1.1M questions; evaluates visual question answering and visual grounding.

• MMMU (Yue et al., 2024): Massive Multidisciplinary Multimodal Understanding—college-level multimodal questions across 30 subjects. Top models score around 70%.

• DocVQA: Document-oriented visual question answering over scanned documents with OCR-dependent reasoning.

  Medical and scientific benchmarks:

• MedQA (Jin et al., 2021): USMLE-style medical licensing questions. Top models now exceed passing threshold; clinical deployment requires higher standards.

• PubMedQA (Jin et al., 2019): 1,000 biomedical research questions from PubMed abstracts.

• CAMELYON16: Whole slide imaging for breast cancer metastasis detection; standard for digital pathology AI evaluation.

  Code and mathematical reasoning:

• MATH (Hendrycks et al., 2021): 12,500 competition mathematics problems across 7 difficulty levels. Top models achieve over 80% as of 2025.

• LiveCodeBench (Jain et al., 2024): Dynamic, contamination-resistant coding benchmark with problems from recent competitive programming contests.

• SWE-Bench (Jimenez et al., 2024): Real GitHub issue resolution across 12 Python repositories; tests practical software engineering capability.

  Robustness and out-of-distribution benchmarks:

• WILDS (Koh et al., 2021): Distribution shift benchmark across 10 real-world datasets spanning genomics, satellite imagery, healthcare, and NLP.

• ImageNet-C (Hendrycks & Dietterich, 2019): ImageNet with 19 types of common corruptions (blur, noise, weather) at 5 severity levels.

• ANLI (Nie et al., 2020): Adversarially collected NLI dataset specifically designed to defeat current models through human-in-the-loop adversarial collection.

  Dataset Discovery and Access Infrastructure

  The infrastructure for finding, accessing, and loading datasets has matured significantly by 2026, with several major platforms providing centralised discovery and access:

  Hugging Face Datasets Hub: The dominant public repository for ML datasets, hosting over 100,000 datasets across all modalities with standardised loading APIs (datasets library), Croissant metadata, and Data Catalogue functionality including dataset cards, viewer previews, and community discussions. The Hub provides version-controlled dataset storage, DOI assignment for academic citation, and access control for gated research datasets.

  Papers With Code: Aggregates ML papers, code repositories, and associated datasets, providing leaderboards linking model performance to the datasets on which they were evaluated. Enables discovery of which datasets are used to evaluate models claiming state-of-the-art performance.

  OpenML: A collaborative platform for sharing datasets, machine learning tasks, flows (algorithms), and experiment runs. Provides standardised benchmarking infrastructure with task definitions, evaluation protocols, and result sharing, supporting Reproducibility through archived experimental metadata.

  Kaggle Datasets: Commercial competition and dataset hosting platform (Google). Provides structured, documented datasets across many domains with community notebooks, discussion forums, and competition leaderboards. Croissant metadata adoption in 2024 improved automated loading.

  UK Data Service / UK Data Archive: The UK’s national data service for social, economic, and population data, hosting academic and government datasets under the UKDS licence framework. Provides access to Understanding Society (BHPS successor, 40,000+ households), the British Social Attitudes Survey, and hundreds of other longitudinal and cross-sectional social research datasets.

  European Data Portal / data.europa.eu: EU open government data platform aggregating datasets from 36 European countries and EU institutions. Hosts geospatial, statistical, and administrative data under DCAT-AP metadata standard. Essential for EU-context Machine Learning and policy research.

  NASA Earthdata, CERN Open Data Portal, NASA/JPL: Scientific domain-specific repositories providing large-scale physics simulation data, astronomical observations, earth observation imagery, and experimental results. Used for scientific ML applications in climate modelling, particle physics, and space research.

  Google Dataset Search: Web search engine for datasets, indexing Schema.org Dataset markup from publisher websites. Provides discovery across geographically distributed dataset publishers without requiring centralised hosting.

  PhysioNet: Repository for biomedical research data, hosting the MIMIC series of ICU clinical datasets, the PTB-XL ECG dataset, and many other clinical research datasets under data use agreements requiring credentialing and ethics training.

  AWS Open Data Registry: Amazon Web Services registry of publicly available, large-scale datasets hosted on S3 at no egress cost to researchers, including Landsat satellite imagery, Common Crawl, OpenStreetMap, and genomic reference datasets. The free-egress model has democratised access to petabyte-scale datasets for academic and small-team researchers.

  Zenodo and Figshare: General-purpose research data repositories integrated with academic publishing workflows. Zenodo (CERN) provides DOI assignment, version control, and 50GB upload limits; Figshare provides similar capabilities with commercial institutional options. Both are extensively used for sharing supplementary datasets accompanying academic papers.

  DRYAD and OSF (Open Science Framework): Discipline-specific open data repositories targeting life sciences (DRYAD) and general research transparency (OSF). OSF’s pre-registration workflow links planned experimental methodology to resulting datasets and analysis, supporting Reproducibility auditing of published ML results.

  Socrata and CKAN: Open data portal platforms widely adopted by city and regional governments for publishing civic datasets (crime statistics, planning applications, transport usage, health metrics). UK local authorities including Greater Manchester Combined Authority, Leeds City Council, and Sheffield City Council publish datasets through CKAN-powered portals, providing locally grounded training data for urban AI applications.

  Domain-specific data commons: Federated data sharing consortia in genomics (GA4GH Data Commons), climate science (PANGAEA, NCAR Research Data Archive), and astronomy (CDS, VizieR, ESO Science Archive) provide domain-expert-curated, high-quality datasets with strong metadata standards and access governance frameworks appropriate for scientific Deep Learning research.

Provenance