Distributed storage is a class of storage system architecture in which data is partitioned, replicated, and managed across multiple physically separate nodes or clusters to achieve scalability, fault tolerance, and high availability beyond the capacity of any single machine. Such systems employ replication protocols, erasure coding, consistent hashing, and distributed consensus algorithms to maintain data integrity and consistency under node failure, network partition, and concurrent access. Examples include object stores such as Amazon S3, distributed file systems such as HDFS and Ceph, and NewSQL databases with sharded storage layers.
Content
- Distributed storage systems emerged from the observation that single-node storage bottlenecks constrain the scalability of data-intensive applications. Google’s GFS (Google File System, 2003) and its open-source successor HDFS (Hadoop Distributed File System) established the foundational architecture: a master/namenode for metadata management and a fleet of datanodes holding chunked replicas, providing fault tolerance through triple replication. This design trades strong consistency for throughput and simplified recovery semantics.
- The CAP theorem (Brewer, 2000) formalises a fundamental constraint: in the presence of network partitions, a distributed storage system must choose between consistency and availability. Object stores such as Amazon S3 and Google Cloud Storage favour eventual consistency and high availability, tolerating brief periods where replicas diverge after writes. Distributed databases such as CockroachDB and Spanner employ distributed consensus (Paxos or Raft) to maintain linearisable consistency at the cost of write latency and geographic reach.
- Erasure coding is an alternative to full replication that dramatically improves storage efficiency. Rather than storing three full copies, erasure codes such as Reed-Solomon split data into k data shards and m parity shards such that any k of k+m shards suffice to reconstruct the original data. Ceph, the dominant open-source software-defined storage system, uses CRUSH (Controlled Replication Under Scalable Hashing) for placement and supports both replication and erasure-coded pools, enabling organisations to balance durability against storage overhead.
- Machine learning workloads impose distinctive I/O patterns on distributed storage. Training large models requires sustained high-throughput sequential reads of shuffled training shards across thousands of GPU workers simultaneously. Object stores with parallel multipart downloads and streaming iterators (e.g., WebDataset format over S3-compatible APIs) address this pattern. Checkpoint saving introduces bursty write loads that can saturate metadata servers; systems like Lustre and GPFS are deployed in HPC clusters specifically to handle these parallel write workloads at the scale required for frontier model training.
- Emerging challenges for distributed storage include multi-cloud and hybrid deployments requiring consistent namespace federation across cloud providers, latency-sensitive AI inference requiring co-location of model weights near compute, and the management of exabyte-scale data lakes at hyperscaler facilities. Data lakehouse architectures (Delta Lake, Apache Iceberg) add transactional metadata layers over distributed object storage, enabling ACID semantics and schema evolution without sacrificing the cost efficiency and scalability of commodity object stores.