Object storage is a data storage architecture that manages data as discrete objects — each comprising an opaque payload, a globally unique identifier, and extensible metadata — accessed via a flat namespace through RESTful HTTP APIs rather than a file hierarchy or block device. It is designed for massive horizontal scalability, high durability, and cost-effective storage of unstructured data such as media files, backups, and machine-learning datasets. Amazon S3 established the de facto API standard, now implemented by numerous compatible services.
Content
- Object storage emerged from web-scale requirements at Amazon, where the Dynamo paper (2007) and the launch of S3 (2006) demonstrated that traditional NAS and SAN architectures could not economically serve internet-scale read workloads. The innovation was to decouple storage management from the file system abstraction, replacing POSIX semantics with a simple PUT/GET/DELETE API that could be implemented across commodity hardware clusters with automatic replication and self-healing.
- The core architecture consists of a stateless gateway tier that handles API requests and translates them to operations on a distributed object pool. Data is striped and replicated — typically achieving 11-nines durability — across failure domains using erasure coding or replication. Metadata (bucket name, key, content-type, user metadata, access control lists) is stored separately in a distributed key-value index. Versioning and lifecycle policies allow automatic tiering to cheaper storage classes such as infrequent-access or archive tiers.
- The S3 API has become a universal storage interface, implemented by open-source projects (MinIO, Ceph RGW) and every major cloud provider (Azure Blob Storage, Google Cloud Storage). Content delivery networks are routinely placed in front of object stores to serve static assets globally. In AI/ML workflows, frameworks such as PyTorch and TensorFlow include native S3-compatible data loaders, enabling training jobs on distributed clusters to read petabyte-scale datasets without local staging.
- Between 2023 and 2025, object storage has become the default substrate for AI model weights, training datasets, and inference artefacts. Tiered intelligent storage — where access patterns automatically move objects between hot, warm, and cold tiers — has reduced infrastructure costs for organisations managing multi-petabyte corpuses. S3 Express One Zone and similar high-performance variants have narrowed the latency gap with block storage for read-heavy ML workloads, while immutable object locking has gained adoption as a ransomware defence mechanism.