Data Versioning

Data Versioning is the systematic practice of tracking, storing, and managing changes to datasets, model artefacts, and derived data products over time, providing reproducibility of experiments, auditability of data lineage, and governance of the full ML data supply chain. It applies software version-control semantics — branching, tagging, diffing, commit history, and rollback — to the large binary assets that underpin machine learning: raw datasets, processed feature stores, trained model weights, and pipeline configuration. Tools such as DVC (Data Version Control), lakeFS, and Apache Iceberg implement data versioning at different granularities, from file-level pointers through table-level snapshots to full environment branching over petabyte-scale data lakes. Data Versioning is a required foundation for reproducible science, regulatory compliance under frameworks such as the EU AI Act, and the operationalisation of mature MLOps practice.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:hasPart ai:ArtifactMetadata))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:hasPart ai:DataLineage))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:hasPart ai:DVC))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:hasPart ai:VersionControlPointer))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:hasPart ai:DataSnapshot))

Dependency Relationships

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:requires ai:ArtifactMetadata))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:requires ai:DataGovernance))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:requires ai:ObjectStorage))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:dependsOn ai:DataEngineering))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:dependsOn ai:VersionControl))

Capability Relationships

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:ModelTraining))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:ExperimentTracking))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:FeatureEngineering))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:Reproducibility))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:ContinuousTraining))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:RegulatoryAuditability))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:supports ai:AIGovernance))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:supports ai:DataGovernance))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:supports ai:MLOps))

Implementation Relationships

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:implements ai:DataPipeline))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:implements ai:WorkflowAutomation))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:partOf ai:MachineLearningPipeline))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:partOf ai:MLOps))

Reduction Relationships

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:reducesTo ai:VersionControl))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:reducesTo ai:ArtifactMetadata))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:reducesTo ai:DataManagement))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:reducesTo ai:DataGovernance))

Additional Axioms

SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:uses ai:ExperimentTracking))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:uses ai:FeatureStore))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:relatedTo ai:ModelRegistry))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:relatedTo ai:DataDrift))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:relatedTo ai:CICD))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:supports ai:Reproducibility))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:supports ai:RegulatoryCompliance))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:enables ai:DataQuality))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:contrastsWith ai:StaticDataset))
SubClassOf(ai:DataVersioning
  ObjectSomeValuesFrom(ai:standardizedBy ai:EUAIAct))

About

Data Versioning extends the version-control paradigm from source code to the data assets that machine learning pipelines consume and produce. The core insight is that an ML system is defined not only by its code but also by the specific snapshot of data it was trained on, the preprocessing transformations applied, the hyperparameter configuration used, and the model weights produced. To reproduce any historical result — or to debug why a deployed model’s behaviour changed — all four of these dimensions must be trackable and recoverable together. Data Versioning provides the infrastructure to make this possible. It is a necessary precondition for meaningful Experiment Tracking, which records hyperparameters and metrics but is useless without the ability to recover the exact dataset those metrics were computed on. Without data versioning, the Model Registry entries listing which model version is in production cannot be connected to the training data that produced them, breaking the audit chain that AI Governance and regulatory compliance demand.

The challenge relative to code versioning is primarily one of scale and binary format. A training dataset for a large vision model may occupy hundreds of gigabytes or terabytes of image files; a transformer language model checkpoint may be tens of gigabytes of binary floating-point tensors. Git’s text-oriented delta compression is unsuitable for such artefacts. Data versioning tools therefore separate concerns: a lightweight pointer record (a hash digest of the artefact’s content) is committed to a Git repository alongside the training code, while the actual binary data is stored in external object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. Checking out a particular Git commit and issuing a data pull command restores both the code and the exact data snapshot that commit corresponds to. This pointer-based approach, pioneered by DVC, has become the baseline data versioning pattern for individual practitioners and small teams. The pointer approach is tightly coupled to Data Engineering infrastructure: the quality of the versioned dataset ultimately depends on the quality of the upstream Data Pipeline that produced it, making Data Quality monitoring and validation a necessary companion discipline.

At enterprise scale, the lakehouse architecture introduces a complementary layer of versioning built into the table format itself. Apache Iceberg and Delta Lake use immutable snapshot isolation: each write operation produces a new snapshot entry in a metadata log, preserving all prior snapshots as fully accessible historical states. lakeFS extends this further by wrapping an entire data lake — including multiple tables, directories, and unstructured files — in a Git-like branching model, allowing teams to isolate experimental data transformations on feature branches, run CI tests against them, and merge them back only when validated. In November 2025, lakeFS acquired the DVC open-source project from Iterative.ai, marking a consolidation of the data versioning ecosystem and signalling the convergence of the two dominant paradigms: developer-oriented lightweight versioning and enterprise-oriented lakehouse branching. The interaction between data versioning and Data Drift monitoring is increasingly important at this level: production ML systems rely on versioned training datasets as the reference distribution against which drift detectors measure incoming data, and automated retraining via Continuous Training pipelines requires the ability to identify the freshest compliant dataset version for the next training cycle. This closes a feedback loop between versioned historical data and live production inference that is the hallmark of a mature MLOps practice. CD automation can then gate model promotion on passing data quality checks against the versioned training set, ensuring that only validated data provenance chains reach production.

The relationship between Data Versioning and Blockchain deserves specific treatment. Blockchain-based provenance ledgers have been proposed as an immutable, decentralised alternative to centralised version stores for high-assurance use cases — in particular for consortium ML settings where no single party should be trusted as the authoritative custodian of the data version history. While blockchain data versioning remains a research prototype pattern rather than mainstream practice, the conceptual connection is meaningful: both approaches share the property of append-only, cryptographically-authenticated history, and DVC’s content-addressed storage already exhibits hash-chain integrity properties analogous to blockchain block hashes. The relationship is therefore modelled as a bridges-to link rather than a full equivalence.

Components / Architecture

Data Versioning systems share a common architectural vocabulary across different implementation families:

• Pointer / metadata layer — a lightweight record (typically a content-addressed hash) stored in a Git repository or metadata catalogue that identifies the exact version of a data artefact. In DVC this is a .dvc file; in Iceberg it is a snapshot entry in the manifest tree; in lakeFS it is a commit object in the reference store. This layer constitutes the Artifact Metadata component of a data versioning system.

• Remote storage backend — object storage (S3, GCS, Azure Blob, HDFS, local filesystem) where the actual data bytes reside. The versioning layer manages access to specific object versions by key or content hash, never overwriting past versions. Storage backend selection has significant cost and performance implications: columnar formats (Parquet, ORC) compressed with Zstandard are standard for analytical datasets, while raw image and video datasets often use content-addressed blob stores.

• Version history log — an append-only journal of commits or snapshots, each recording who changed what, when, and why. Enables diff, blame, and audit queries across any two points in the version history. This log is the primary auditability mechanism that satisfies regulatory requirements for Data Lineage traceability.

• Branching and merging — the ability to create divergent versions of a dataset (for experimentation, A/B testing of preprocessing strategies, or regulatory sandboxing), develop them independently, and merge them under controlled conflict resolution. Feature branches for data mirror Git feature branches for code. Branch-level isolation is the key mechanism enabling safe experimentation within a production Data Pipeline without risking the integrity of the main training dataset.

• Pipeline stage definitions — declarative descriptions of how one data version is derived from another (DVC’s dvc.yaml, Airflow DAG definitions, lakeFS hooks). Pipeline definitions are themselves versioned, so the full transformation graph from raw ingestion through preprocessing to training-ready features is reproducible and auditable. This is the bridge to Workflow Automation — version-controlled pipelines are the executable form of the data transformation specification.

• Garbage collection — mechanisms to expire and delete old snapshots that are no longer referenced, reclaiming storage without breaking live references. Iceberg’s expire_snapshots procedure and lakeFS’s garbage collection policies are examples. GC policies must be aligned with regulatory retention requirements — in regulated industries, certain historical dataset versions may be legally required to be retained for audit purposes for defined periods.

• Integration layer — connectors to Experiment Tracking systems, Model Registry instances, Feature Store platforms, and CD pipelines that ensure data version metadata flows automatically through the MLOps toolchain. The integration layer is responsible for associating each model training run logged in the experiment tracker with the specific data version identifier that was used, so the full provenance chain is queryable.

• Data validation and quality gates — schema checks, statistical distribution validation, and referential integrity tests run against each data version before it is promoted from staging to production use. Tools such as Great Expectations, Soda Core, and dbt tests provide the validation layer. Integration with data versioning allows failed validations to block version promotion without discarding the version, enabling retrospective debugging.

• Access control and audit logging — data versioning systems in regulated industries must maintain fine-grained access logs recording who accessed which version of which dataset, when, and for what stated purpose. This audit log is distinct from the version history log and may be subject to different retention and security requirements under Data Governance frameworks.

  Use Cases / Major Families

  Data versioning implementations divide into four families distinguished by granularity and the organisational context they serve:

  File-level versioning (DVC, Git-LFS) The foundational pattern for individual practitioners and small teams. A .dvc pointer file tracks the content hash of each dataset file; the actual data is pushed to a remote store. Ideal for projects where datasets fit in hundreds of gigabytes and experimentation involves swapping whole dataset versions or splits. DVC additionally supports pipeline reproducibility through stage definitions that record dependencies and outputs, making it possible to rerun only the pipeline stages whose inputs have changed — analogous to Make-style build systems. DVC’s close integration with Git means that the full experimental history is navigable through standard Git tooling: git checkout plus dvc pull restores any historical experiment state. After the lakeFS acquisition in November 2025, DVC remains maintained as an open-source project and continues to serve its core constituency of data scientists working on individual or small-team projects.

  Table-level versioning (Apache Iceberg, Delta Lake, Apache Hudi) Open table formats that embed versioning semantics into the storage layer of lakehouse architectures. Each write is a new immutable snapshot; historical snapshots are fully queryable. Schema evolution (adding/removing/renaming columns) is tracked in the metadata layer without physical rewriting. Time-travel queries allow analysts to reconstruct the state of a table at any historical timestamp — a capability directly relevant to Empirical Experimental Design workflows that require retrospective analysis. As of 2025, Apache Iceberg has emerged as the de facto standard open table format, with Delta Lake and Iceberg communities aligning on shared roadmaps for catalogue interoperability under the OneTable / Apache XTable project, converging on a shared commit handling and schema evolution specification.

  Environment-level versioning (lakeFS) A Git-for-data layer wrapping an entire data lake or lakehouse environment. Creates full branches of the entire namespace — not just individual tables — allowing teams to test end-to-end Data Pipeline processing on isolated branches before merging changes to the main production branch. Supports CD gates: before merging a feature branch, automated data quality checks, schema validation, and statistical distribution tests run against the branch data. Particularly suited to regulated industries where data changes must be peer-reviewed before entering the production Model Training pipeline. lakeFS’s branching model is also used for safe data migrations, ingestion testing, and multi-tenant data isolation.

  Blockchain-backed provenance versioning An emerging pattern for consortium ML settings where no single party should be trusted as the authoritative custodian of the data version history. Distributed ledger technologies provide append-only, cryptographically authenticated version logs that are tamper-evident and independently auditable by all consortium members without requiring trust in a central version store. While this family remains predominantly in research and pilot stages, interest from pharmaceutical consortia (multi-party clinical trial data), financial risk modelling consortia, and defence intelligence fusion programmes has grown. The Blockchain bridges-to relationship in this ontology reflects this conceptual connection between data versioning and distributed ledger provenance.

  Scientific data versioning (Zenodo, Figshare, Dryad) Academic and research data repositories that implement persistent identifier-based versioning using Digital Object Identifiers (DOIs). Each published dataset version receives a unique DOI enabling citation and retrieval; the FAIR Guiding Principles (Wilkinson et al., 2016) frame versioning as a prerequisite for data reusability. The 2024 standardised versioning proposal (Sansone et al.) extends this with “major.minor.patch” semantics aligned with semantic versioning, enabling consumers of research datasets to assess the impact of dataset changes before updating their analyses. The Reproducibility and Data Management links are most salient in this context.

  Academic Context

  The intellectual foundations of data versioning trace to several distinct research threads, each contributing a different dimension of the concept as it is understood today.

  The most influential single thread is the reproducibility crisis in machine learning, documented by Sculley et al. at Google (2015) in their influential “Hidden Technical Debt in Machine Learning Systems” paper, which identified the conflation of data and code evolution as a primary source of silent system degradation. This paper gave data versioning a compelling engineering motivation beyond mere good practice: without data versioning, debugging why a model that performed well in experiments degrades in production is often impossible, because the exact training data used is irrecoverable. The paper influenced an entire generation of MLOps tooling designed to make Reproducibility tractable in industrial ML settings.

  Database provenance research provided the theoretical foundations. Cheney et al.’s survey “Provenance in Databases: Why, How, and Where” (2009) surveyed the landscape of data provenance — the problem of tracking how data values came to be what they are — and introduced the vocabulary of why-provenance, where-provenance, and how-provenance that continues to shape Data Lineage tooling. The W3C PROV data model (Moreau et al., 2011), developed partly through contributions from the University of Edinburgh and Southampton, provided a standard interchange format for provenance graphs that is widely used in scientific and enterprise data pipelines. The University of Manchester’s contribution to PROV through the e-Science programme was substantial, reflecting the UK’s historically strong position in provenance research.

  Vartak et al. at MIT CSAIL introduced ModelDB (2016), one of the first systems to jointly version models and the datasets they were trained on, establishing the conceptual link between Experiment Tracking and data versioning. This work directly influenced the design of MLflow (Zaharia et al., 2018), which integrated Model Registry, experiment logging, and dataset provenance references in a unified tracking server that became the most widely deployed open-source MLOps tracking tool.

  In the FAIR data movement (Wilkinson et al., 2016), the principle that data should be Findable, Accessible, Interoperable, and Reusable introduced versioning as a first-class requirement for scientific data management. The principle that each version of a dataset must have a persistent, unique identifier (the F in FAIR) aligns directly with the content-addressed hash approach used by DVC and similar tools, making FAIR and technical data versioning mutually reinforcing. A 2024 proposal for standardised dataset versioning (Sansone et al., Scientific Data) formalised a “major.minor.patch” versioning scheme analogous to semantic versioning in software engineering, providing a vocabulary for communicating the significance of dataset changes to downstream consumers and enabling automated compatibility checking. Major version increments indicate schema or population changes that break backward compatibility; minor increments indicate additive extensions; patch increments indicate quality fixes that do not change the population or schema.

  The UK Reproducibility Network (UKRN), established in 2019 under the leadership of Dorothy Bishop (Oxford) and Marcus Munafo (Bristol), has driven significant investment in open and transparent research practices across UK universities, with large-scale audits (published 2024) quantifying adoption of Version Control and FAIR data sharing practices across disciplines. Results showed significant variation: physical sciences and computational disciplines had high adoption rates, while clinical and social sciences lagged. The findings directly informed UK Research and Innovation (UKRI) open data policies, which now mandate FAIR data sharing for grant-funded research, with versioning requirements embedded in data management plans. The University of Edinburgh’s 2024 longitudinal study found that the fraction of bioscience publications sharing FAIR-compliant datasets increased from 7% in 2014 to 45% in 2023, with Data Pipeline automation and data versioning tools cited as key enablers of this increase.

  The intersection of data versioning with Data Quality is an active research area. The 2021 survey by Whang et al. in the VLDB Journal identified dataset versioning as one of the three primary mechanisms for addressing data collection and quality challenges in deep learning (alongside data augmentation and active learning). A data-centric AI paradigm — articulated by Andrew Ng and colleagues circa 2021 — frames model improvement as primarily a data improvement problem rather than a model architecture problem, elevating data versioning from infrastructure concern to central research topic. The Model Training loop in this paradigm involves iterating over dataset versions, measuring model performance, identifying data failure modes, and releasing improved dataset versions — a cycle that data versioning infrastructure must support with low friction.

  Current Landscape (2026)

  The data versioning landscape underwent significant consolidation in late 2025. In November 2025, lakeFS — backed by a $20 million growth funding round announced in July 2025 — acquired the DVC open-source project from Iterative.ai, bringing together the two dominant paradigms under one organisation. lakeFS has reported triple-digit growth in user adoption, with enterprise customers including ARM, Bosch, Lockheed Martin, NASA, Volvo, and the US Department of Energy. DVC continues as an independent open-source project focused on individual data scientists managing smaller-scale versioning needs, while lakeFS targets enterprise petabyte-scale data lake governance. The acquisition was accompanied by a clear vision for complementarity: DVC serves the individual data scientist and small team workflow, integrating with Git for lightweight file-level versioning, while lakeFS provides the organisational data platform layer for enterprise AI infrastructure. According to lakeFS, the combined ecosystem now serves thousands of organisations globally.

  At the table format level, Apache Iceberg has consolidated its position as the de facto open standard for data lakehouse environments. The Delta Lake and Iceberg communities are now aligned under a shared strategic direction, with converging commit handling, schema evolution, and versioning mechanics. The focus of the lakehouse community has shifted from format competition to enabling a shared catalogue interface across query engines, with OneTable / Apache XTable providing cross-format interoperability. This convergence simplifies the data versioning story for organisations that need to support multiple query engines (Spark, Flink, Trino, DuckDB) against the same versioned data assets.

  An EY survey in 2025 found that 83% of executives reported AI adoption would be faster with stronger data infrastructure, while 67% cited lack of data infrastructure as the primary barrier to AI progress — underscoring the growing strategic importance of data versioning as a business capability rather than a purely technical concern. The survey reflects a broader enterprise recognition that MLOps maturity is gated on data infrastructure maturity, and that data versioning is a prerequisite for the Continuous Training loops that make production AI systems self-improving over time.

  Regulatory pressure from the EU AI Act (fully applicable from August 2026 for high-risk systems) and the UK’s AI Governance Framework is accelerating enterprise adoption: high-risk AI system operators must maintain complete traceability of training data under Articles 10 and 11 of the Regulation, meaning data versioning is no longer optional for regulated industries including financial services, healthcare, and autonomous systems. The Act’s requirement for technical documentation covering data governance and management practices — including information about the data sets used for training — is creating a compliance-driven demand for data versioning tooling that produces structured, machine-readable audit artefacts alongside human-readable version histories. ISO/IEC 42001 (AI Management Systems Standard, 2023) independently mandates traceability of data used in AI system development, reinforcing the regulatory push for data versioning adoption.

  The data versioning ecosystem in 2026 also includes notable tools beyond DVC and lakeFS: Weights & Biases Artifacts provides versioning tightly integrated with experiment tracking for deep learning practitioners; Pachyderm provides a containerised pipeline versioning system that co-versions code and data transformations; Hugging Face Hub provides dataset versioning through Git-LFS for the NLP research community; and neptune.ai provides artefact versioning integrated with its experiment tracking platform. This diversity of tools reflects the breadth of use cases data versioning must serve, from research reproducibility to enterprise compliance to collaborative data science at web scale.

  UK Context

  The UK has a strong academic tradition in reproducible research and data provenance that underpins its data versioning research community, spanning formal provenance theory, open science practices, and industrial ML infrastructure.

  The Alan Turing Institute (London), as the UK’s national institute for data science and artificial intelligence, has published extensively on data lineage and provenance in ML systems, and its Research Engineering Group has developed and published open-source tooling for reproducible ML pipelines including data versioning integration guidance. The Institute’s collaborations with BT, HSBC, Rolls-Royce, and the NHS have produced practical frameworks for data versioning in industrial and healthcare AI contexts that are directly informed by UK regulatory requirements.

  The UK Reproducibility Network (UKRN), with institutional nodes at universities including Edinburgh, Bristol, Manchester, Leeds, Cardiff, and Birmingham, has been instrumental in surveying and improving Version Control and FAIR data adoption across disciplines, with landmark audit studies published in 2024. The UKRN’s institutional leads programme has placed reproducibility champions in over 50 UK universities, driving data versioning adoption as a practical open science skill. The Network collaborates with UK Research and Innovation (UKRI) on open data policy, and UKRI’s open data requirements now mandate FAIR data sharing — including versioned dataset releases with persistent identifiers — for funded research outputs.

  The University of Edinburgh’s School of Informatics has contributed foundational work on data provenance systems and reproducible pipelines, including contributions to the W3C PROV standard. Edinburgh’s national e-infrastructure role (hosting ARCHER2, the UK national supercomputer) means its research data management practices influence norms across the entire UK research community. The Centre for Doctoral Training in Robotics and Autonomous Systems (Edinburgh/Heriot-Watt) addresses data versioning for robotic learning datasets as a core curriculum component.

  The University of Manchester’s eScience Institute was a global pioneer in provenance research, contributing to the Open Provenance Model (Moreau et al., 2011) and the W3C PROV standard through the work of Carole Goble and Iain Buchan. The ELIXIR Research Infrastructure node at Manchester handles life sciences data versioning at national scale, managing versioned releases of genomics, proteomics, and metabolomics datasets used by UK and European research consortia. Manchester’s Biomedical Research Centre generates large-scale clinical datasets that must be versioned, governed, and shared in compliance with NHS Digital data governance requirements and the UK Policy Framework for Health and Social Care Research.

  Imperial College London’s Data Science Institute has published on data management pipelines for large-scale clinical AI, where dataset versioning is essential for regulatory compliance under MHRA guidelines and NHS AI governance requirements. Imperial’s involvement in NHS AI deployment — including radiology AI tools and sepsis prediction models — requires dataset versioning as a regulatory necessity for CE-marked medical devices under UK MDR 2002 (as retained from EU MDR 2017/745 post-Brexit).

  In the Northern English industrial context, Leeds, Sheffield, and Manchester host significant data engineering and MLOps teams within financial services (HSBC Leeds, Lloyds Banking Group, Yorkshire Building Society, First Direct), manufacturing and aerospace (AMRC Sheffield, BAE Systems Samlesbury, Rolls-Royce Derby), and the NHS. These organisations face practical data versioning challenges across three regulatory regimes: patient data governance under UK GDPR and the NHS Data Security and Protection Toolkit, clinical trial dataset traceability under ICH GCP (Good Clinical Practice) regulations enforced by the MHRA, and aerospace software certification requirements (DEF STAN 00-970, DO-178C Level A) that mandate full data provenance records for training datasets used in safety-critical AI systems.

  The UK’s AI Safety Institute (AISI), established in late 2023 at the Department for Science, Innovation and Technology, has produced technical reports on foundation model evaluation that directly engage with training data versioning and documentation requirements. The AISI’s assessment frameworks require AI developers to document and version their training datasets as part of model transparency obligations, creating a new regulatory driver for data versioning adoption in the UK AI industry. UK AI companies developing high-risk AI systems for financial services (FCA sandbox) or healthcare (NHS procurement) face parallel requirements from the Financial Conduct Authority and NHS England’s AI Assurance Framework, both of which reference dataset provenance documentation.

  Future Directions (2026-2030)

  The near-term trajectory of data versioning is towards tighter integration with the AI governance and compliance stack. As the EU AI Act’s technical documentation requirements take full effect for high-risk systems (August 2026), data versioning tooling will need to produce structured, machine-readable compliance artefacts — not just human-readable commit histories — linking dataset versions to risk assessments, bias evaluations, and conformity declarations. The Act’s Article 10 requirements for data governance and management effectively codify data versioning as a legal requirement for high-risk AI systems in the EU. The UK’s AI Safety Institute is developing analogous guidance for high-risk AI deployment in the UK market post-Brexit, creating parallel compliance drivers for British organisations.

  Semantic versioning for datasets (the “major.minor.patch” framework formalised by Sansone et al. in 2024 in Scientific Data) is expected to gain adoption as a shared vocabulary between data producers and consumers, allowing downstream consumers to programmatically assess the impact of dataset changes before updating their dependent Model Training pipelines. This could enable automated dependency management analogous to npm or pip for software packages, but applied to dataset dependencies — alerting model training pipelines when their training dataset has been patched (safe to update automatically), extended (update after review), or restructurally changed (manual migration required). The Experiment Tracking infrastructure would log dataset version identifiers using semantic versioning notation alongside model performance metrics, enabling causal analysis of the relationship between dataset version changes and model quality changes.

  Data contracts — formal specifications of the expected schema, statistical distribution properties (feature means, ranges, null rates), and quality guarantees of a dataset version — are emerging as a complementary mechanism to version history, making explicit what a dataset version promises to its consumers. Tools like Soda Core, Great Expectations, and dbt Contracts provide data contract infrastructure; the connection to data versioning is that each published version of a dataset should be accompanied by its contract, and consumers should validate their expectations against the contract before accepting a new version. This contractual model introduces a formal quality gate into the data versioning workflow that goes beyond content-addressed integrity verification to include statistical and semantic correctness guarantees.

  Convergence between data versioning and data mesh architectures will likely accelerate over the 2026-2030 period. In a data mesh, each domain team owns and versions its own data products as independently deployable units, and cross-domain Data Lineage requires federated versioning registries rather than centralised version stores. This architectural shift challenges the current paradigm of centralised data lakes with a single versioning layer, requiring instead a mesh of domain-level version registries connected by a data lineage federation layer implementing standards such as OpenLineage. Tools bridging DVC-style local versioning with lakeFS-style enterprise branching, and connecting both to the OpenLineage standard for cross-system lineage federation, represent the next frontier.

  Integration with foundation model training pipelines represents a particularly high-stakes application of data versioning. Models with hundreds of billions of parameters trained on trillions of tokens require unprecedented levels of dataset curation, deduplication, quality filtering, and provenance tracking. The legal and ethical questions surrounding training data — copyright, personal data under GDPR, bias and representativeness — are creating demand for training dataset registries with cryptographic provenance chains that can withstand legal scrutiny. The connection between data versioning and Blockchain may become practically significant in this context, as immutable, independently auditable provenance records for training data become a business and regulatory necessity for foundation model developers.

  LLM-assisted data versioning management is also emerging: systems that can automatically generate human-readable change descriptions for dataset version diffs, flag potentially problematic changes (e.g., a version that accidentally drops a protected demographic subgroup), and suggest validation tests to run before promoting a new dataset version to production. These AI-over-AI capabilities close a feedback loop in which Artificial Intelligence systems both produce and govern the data versioning lifecycle.

  Formal Analysis

  The formal treatment of data versioning can be grounded in the theory of multiversion databases and immutable data structures. A versioned dataset D can be modelled as a function V → I from a totally ordered set of version identifiers V (timestamps, content hashes, or sequence numbers) to a set of dataset instances I. For any v ∈ V, D(v) is the dataset state at version v. The versioning operations are:

• commit(D, Δ, m): given current dataset state D(head), delta Δ (a set of additions and deletions), and commit message m, produces a new version identifier v’ and stores D(v’) = D(head) ∪ additions(Δ) \ deletions(Δ), recording v’, m, and the identity of the committer in the version log.

• checkout(D, v): retrieves D(v) for any version identifier v in the version log, enabling point-in-time reconstruction of any historical dataset state.

• diff(D, v₁, v₂): computes the symmetric difference between D(v₁) and D(v₂), returning the set of additions and deletions needed to transform one into the other.

• merge(D, v₁, v₂, base): given two divergent versions v₁ and v₂ and their common ancestor base, computes a merged version that incorporates all non-conflicting changes from both branches. Conflict resolution policies (last-write-wins, application-defined merge functions, manual resolution) are applied for conflicting changes.

  Content-addressed hashing provides the foundational integrity guarantee: a version identifier is computed as a cryptographic hash (SHA-256 or SHA-512) of the dataset content, ensuring that two identical datasets always produce the same identifier and that any modification — however small — produces a different identifier. This property makes it computationally infeasible to tamper with a historical dataset version without detection, providing the same integrity guarantees as a Blockchain commit chain without the distributed consensus overhead.

  The Data Lineage graph of a versioned data processing pipeline is a directed acyclic graph (DAG) where nodes are dataset versions and edges are transformations. Formal provenance models such as W3C PROV represent this DAG using wasDerivedFrom and wasGeneratedBy edges, enabling precise causal queries: “which raw data records contributed to this model prediction?” (backward provenance / why-provenance) and “which downstream models are affected by a correction to this training example?” (forward provenance / where-provenance). The complexity of answering these provenance queries depends on the provenance semiring structure of the pipeline — a formalism developed by Green, Karvounarakis, and Tannen (2007) that assigns algebraic expressions to query outputs in terms of input tuple annotations, providing a unified framework for why, how, and lineage provenance.

  The relationship between data versioning and Empirical Experimental Design can be formalised as a combinatorial search problem over the space of dataset-hyperparameter-architecture triples. Each training run in a systematic ML experiment consumes a specific dataset version (identified by its version ID) and produces a model artefact (identified by its checkpoint hash). The Experiment Tracking system records the mapping from run ID to (dataset version, hyperparameter configuration, model checkpoint), enabling post-hoc analysis queries such as: “among all runs trained on dataset version v₃, which hyperparameter settings produced models that generalised best to the held-out test set?” Data versioning is the mechanism that makes these dataset version IDs stable and recoverable references rather than ephemeral pointers to mutable live data.

  The interaction between data versioning and Data Drift detection in production MLOps systems can be modelled as follows. Let D(v_train) be the training dataset version used to produce deployed model M. Let D(prod, t) be the distribution of inputs seen by M in production at time t. Data Drift is detected when a statistical distance metric δ(D(v_train), D(prod, t)) exceeds a threshold τ. The versioned training dataset D(v_train) is the reference distribution against which drift is measured; without data versioning, D(v_train) is undefined as a stable reference point — the drift detector would be comparing production data against an unknown, potentially mutable dataset, making the drift signal uninterpretable. Data versioning thus provides the stable reference distribution that makes Data Drift monitoring semantically meaningful, closing the loop between MLOps monitoring and the historical data record. When drift triggers Continuous Training, the new Model Training run must select a fresh dataset version that represents the current production distribution — requiring data versioning to manage both the historical reference and the current training snapshot as distinct, versioned entities.

  Standards and Governance Context

  Data versioning sits at the intersection of several governance and standards frameworks:

  W3C PROV Data Model (2013) — the international standard for data provenance representation, defining the Entity, Activity, and Agent types and the wasGeneratedBy, used, wasAttributedTo, wasDerivedFrom, and wasAssociatedWith relations. PROV provides the formal vocabulary for expressing Data Lineage chains, and data versioning tools that emit PROV-compliant lineage records enable cross-system provenance federation under the Open Lineage standard.

  FAIR Data Principles (2016) — established by Wilkinson et al. as four principles (Findable, Accessible, Interoperable, Reusable) for scientific data management. The Findability principle specifically requires that each dataset version carry a globally unique, persistent identifier — satisfied by content-addressed hashing in DVC and by Iceberg snapshot IDs. The Reusability principle requires data to carry a clear licence and provenance record, satisfied by data versioning tools that log Artifact Metadata including creator, timestamp, and transformation history.

  EU AI Act — Article 10 (Data and Data Governance) — requires providers of high-risk AI systems to implement data governance and management practices covering: the design choices for data sets; data collection processes; data preparation and analysis operations; formulation of relevant assumptions; assessment of availability, quantity, and suitability; and examination of possible biases. Article 11 requires technical documentation including information about the training, validation, and testing data sets used. These requirements collectively constitute a legal mandate for data versioning as a compliance infrastructure.

  ISO/IEC 42001:2023 (AI Management Systems) — requires organisations developing or using AI systems to implement processes for data management that include traceability of training data, data quality assurance, and documentation of data sources and transformations. Clause 6.2.3 on data requirements directly addresses the need for version-controlled training datasets.

  UK GDPR and the UK DSAR framework — require organisations to be able to demonstrate what personal data they hold, when it was collected, and what processing has been applied. For AI systems trained on personal data, this creates an obligation to version and document the training dataset with sufficient precision to respond to Data Subject Access Requests and Right to Erasure requests — including the ability to identify which historical training dataset versions contained data about a specific individual.

  Open Lineage Standard — an open-source standard for cross-system data lineage metadata, defining a common API and data model for lineage events emitted by diverse data processing systems (Spark, dbt, Airflow, Flink). Tools integrating OpenLineage emit lineage events that reference dataset version identifiers, enabling end-to-end lineage graphs spanning multiple systems to be reconstructed from a centralised lineage backend. This creates a federated Data Lineage layer above the individual data versioning systems used by each pipeline component.

  Research and Literature

  1. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., … & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems (NIPS), 28.
  2. Vartak, M., Subramanyam, H., Lee, W. E., Viswanathan, S., Huber, S., Bharioke, A., … & Madden, S. (2016). ModelDB: a system for machine learning model management. Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA), ACM.
  3. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., … & Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.
  4. Miao, H., Li, A., Davis, L. S., & Deshpande, A. (2017). Towards unified data and lifecycle management for deep learning. IEEE International Conference on Data Engineering (ICDE).
  5. Bose, R., & Frew, J. (2005). Lineage retrieval for scientific data processing: a survey. ACM Computing Surveys, 37(1), 1-28.
  6. Alvaro, P., Marczak, W., Conway, N., Hellerstein, J. M., Maier, D., & Sears, R. (2011). Dedalus: Datalog in time and space. Datalog Reloaded, LNCS.
  7. Cheney, J., Chiticariu, L., & Tan, W. C. (2009). Provenance in databases: Why, how, and where. Foundations and Trends in Databases, 1(4), 379-474.
  8. Amsterdamer, Y., Deutch, D., & Tannen, V. (2011). Provenance for aggregate queries. Proceedings of the 30th ACM SIGMOD-SIGACT-SIGAI symposium on Principles of Database Systems.
  9. Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., … & Zhao, J. (2011). The open provenance model — core specification (v1.1). Future Generation Computer Systems, 27(6), 743-756.
  10. Sansone, S. A., et al. (2024). Standardised Versioning of Datasets: a FAIR-compliant Proposal. Scientific Data, 11, 1-12.
  11. UK Reproducibility Network. (2024). Open and transparent research practices survey dataset. Scientific Data, 11, 939.
  12. Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., … & Talwalkar, A. (2018). Accelerating the machine learning lifecycle with MLflow. IEEE Data Engineering Bulletin, 41(4).
  13. Rukat, T., Ghassemi, M., Theis, F. J., & Stegle, O. (2020). Detecting and correcting for label shift with black box predictors. International Conference on Machine Learning (ICML).
  14. Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020). Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions. Information and Software Technology, 127, 106368.
  15. Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2022). Operationalizing machine learning: An interview study. arXiv preprint arXiv:2209.09125.
  16. Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11, 31866-31879.
  17. Renggli, C., Rimanic, L., Gürel, N. M., Karlas, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 44(4).
  18. lakeFS. (2025). The State of Data and AI Engineering 2025. Retrieved from lakefs.io/blog/the-state-of-data-ai-engineering-2025/.
  19. lakeFS / Iterative.ai. (2025). lakeFS Acquires DVC, Uniting Data Version Control Pioneers to Accelerate AI-ready Data. PR Newswire, November 2025.
  20. Dremio. (2025). Data Lakehouse Versioning: Nessie vs Iceberg vs lakeFS. Retrieved from dremio.com.
  21. European Commission. (2024). EU AI Act — Regulation (EU) 2024/1689 of the European Parliament and of the Council on Artificial Intelligence. Official Journal of the European Union.
  22. University of Edinburgh Biosciences. (2024). The rise of open data practices among bioscientists at the University of Edinburgh. PLOS ONE, 19(7).
  23. Klaise, J., Van Looveren, A., Cloete, G., & Coca, A. (2021). Alibi detect: Algorithms for outlier, adversarial and drift detection. Journal of Machine Learning Research, 22(147).
  24. Whang, S. E., Roh, Y., Song, H., & Lee, J. G. (2023). Data collection and quality challenges in deep learning: a data-centric AI perspective. The VLDB Journal, 32(4), 791-813.
  25. Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018). Data lifecycle challenges in production machine learning: A survey. ACM SIGMOD Record, 47(2), 17-28.
  26. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML test score: A rubric for ML production readiness and technical debt reduction. IEEE International Conference on Big Data.
  27. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., … & Zimmermann, T. (2019). Software engineering for machine learning: A case study. ICSE — Software Engineering in Practice Track.
  28. Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media, Sebastopol CA.

  Key Terminology

• Data snapshot — an immutable, point-in-time capture of a dataset’s contents, identified by a content hash or version identifier. The fundamental unit of the data versioning history, analogous to a Git commit for source code.

• Content-addressed storage — a storage system where data is retrieved by a cryptographic hash of its content rather than by name or path, ensuring that identical data always resolves to the same identifier and that any modification creates a new, distinct identifier. The basis of DVC’s file versioning and Iceberg’s manifest files.

• Data lineage — the tracked provenance chain from raw source data through all transformations, aggregations, and joins to a final model artefact or analytic result. Data lineage is both a technical record (who computed what from what) and a governance artefact (audit evidence for regulatory compliance). Represented formally by the W3C PROV data model.

• Time-travel query — a query against a versioned table (in Apache Iceberg, Delta Lake, or lakeFS) that returns results as of a specified past timestamp or snapshot identifier, without physically restoring any data — the historical state is reconstructed from the immutable snapshot log. Essential for audit, debugging, and retrospective analysis workflows.

• Schema evolution — the controlled addition, removal, or modification of columns, data types, or structural elements in a versioned dataset without breaking existing consumers. Apache Iceberg implements schema evolution as a metadata-only operation using column IDs rather than column names, making it safe and zero-cost.

• Data contract — a formal specification of the expected schema (column names, types, and nullability), statistical distribution properties (feature means, ranges, cardinality), quality guarantees (null rate thresholds, referential integrity checks), and SLA commitments promised by a data producer to downstream consumers. Data contracts make the implicit assumptions of data pipelines explicit and testable.

• Branching — creation of an independent copy of a dataset or data namespace for parallel development, mirroring the Git feature-branch paradigm. In lakeFS, branches are instantaneous (zero-copy at creation time, using a reference pointing to the current main branch state) and can be merged back to main after validation. Branching enables safe experimentation on production-scale datasets without risking production data integrity.

• FAIR principles — Findable, Accessible, Interoperable, Reusable: the four properties characterising well-managed scientific data assets as defined by Wilkinson et al. (2016). FAIR requires each dataset version to carry a persistent, globally unique identifier (F), to be retrievable by its identifier (A), to use standardised metadata schemas enabling cross-system interoperability (I), and to carry a licence and provenance record enabling reuse (R).

• Open table format — a storage specification (Iceberg, Delta Lake, Hudi) that embeds versioning, schema evolution, and time-travel semantics directly into the data file organisation layer of a data lake, independent of the compute engine that reads it. Open table formats decouple the data versioning capability from any specific Data Pipeline or query engine.

• Data mesh — an architectural pattern in which data is owned, managed, and published as versioned data products by domain teams, with a federated governance model rather than a centralised data lake. Data versioning in a data mesh context requires distributed version registries and cross-domain Data Lineage federation.

• Pointer file — in DVC, a small text file committed to Git that records the content hash and remote storage location of a large data artefact, serving as the versioned reference to that artefact without storing the artefact itself in Git. The pointer mechanism decouples version history (in Git) from binary storage (in object stores).

• Continuous training (CT) — the third loop in MLOps, complementing Continuous Integration (CI) and Continuous Delivery (CD): automated retraining of production models triggered by data drift detection, scheduled data freshness windows, or upstream data pipeline updates. Continuous Training depends critically on data versioning to ensure that each retraining run uses a validated, identified snapshot of training data rather than a mutable live dataset.

• Experiment reproducibility — the ability to exactly re-run a past ML experiment — same code, same data, same environment, same hyperparameters — and obtain the same results. Data versioning is a necessary (though not sufficient) condition for full experiment reproducibility, alongside code versioning, environment versioning (containers), and random seed management.

  Summary

  Data Versioning is the essential infrastructure layer that transforms ephemeral, mutable data assets into stable, identifiable, and reproducible resources. It applies the proven semantics of software Version Control — commit, branch, diff, merge, rollback — to the large binary artefacts of machine learning: raw datasets, processed feature snapshots, model weights, and pipeline configurations. As AI systems become subject to legal accountability requirements under the EU AI Act and national AI governance frameworks, and as the scientific community intensifies demands for Reproducibility under the FAIR principles, data versioning is transitioning from an optional best practice to a mandatory infrastructure component. The ecosystem has consolidated around a clear layered architecture: file-level versioning (DVC) for individual data scientists, table-level versioning (Apache Iceberg, Delta Lake) for analytical data products, and environment-level versioning (lakeFS) for enterprise data lake governance — with the lakeFS acquisition of DVC in November 2025 signalling the beginning of unified multi-layer versioning platforms. The UK research community, through the UK Reproducibility Network and institutions including Edinburgh, Manchester, and the Alan Turing Institute, has been a significant driver of FAIR data versioning adoption, while UK industrial sectors from financial services to aerospace face compliance-driven mandates for dataset traceability that are accelerating enterprise adoption of data versioning infrastructure.

Provenance