Designing a Distributed Object Storage System

Object storage holds immutable blobs addressed by bucket and key: images, backups, logs and data lake files. This design mirrors the architecture of Amazon S3 and Ceph RGW.

Requirements

Functional

• PUT, GET, DELETE, LIST by prefix, with objects from 1 byte to 5 TB.
• Multipart and resumable uploads, ranged reads.
• Bucket-level versioning and lifecycle rules.

Non-functional

• Durability of eleven nines per year.
• 99.99 percent availability for reads.
• 100K requests per second per region, aggregate throughput of 100 GB/s.
• First-byte latency under 50 ms for small objects.
• Strong read-after-write consistency for new objects.

Capacity estimation

Assume 100 billion objects averaging 1 MB: 100 PB logical. Three-way replication would need 300 PB; erasure coding with 10 data plus 4 parity shards uses 1.4x, so 140 PB. At 20 TB drives and 60 drives per server, one rack of 10 servers gives about 12 PB raw, so roughly 12 racks of raw capacity, plus headroom, in the low tens of racks. Metadata: 100 billion entries at 300 bytes is 30 TB, which must be sharded in a database.

High-level design

 Client --> Load balancer --> API / front-end servers (stateless)
                                  |            |
                        Metadata service    Placement service
                        (sharded, Raft)     (maps object to volumes)
                                  |
                        Storage nodes (data plane)
                        [chunk store on JBOD disks]
                                  |
              Background: repair, scrubber, GC, tiering

Separating metadata from data lets each scale independently: metadata is small and random-access, data is large and sequential.

Data model

Metadata record: bucket, key, version id, size, ETag, content type, a list of chunk descriptors (chunk id, erasure group, shard locations), user metadata and a delete marker flag. The primary key is bucket plus key plus version, stored in ordered shards so prefix listing is a range scan. Data nodes store chunks (for example 64 MB) in append-only extents with a checksum per block.

Key deep-dives

Write path. The front end streams the body, splits it into chunks, and encodes each chunk into 14 shards placed across distinct racks and zones. It commits after a quorum of shards (say 12 of 14) is durable, then writes the metadata record. Metadata is committed last, so a failed upload leaves only orphaned chunks, which garbage collection reclaims.

Erasure coding versus replication. Reed-Solomon 10+4 survives any four shard losses at 1.4x storage but makes reads and repairs touch many nodes. Use replication (3x) for small or hot objects and erasure coding for large, cold ones. Local reconstruction codes reduce repair traffic.

Metadata sharding. Range-partition by bucket and key hash prefix with automatic splitting when a shard grows or turns hot. Each shard is a Raft group, which gives strong consistency. LIST uses ordered scans with continuation tokens. Avoid sequential key prefixes that create hot ranges.

Multipart upload. The client uploads parts independently (5 MB to 5 GB), each producing an ETag. A complete call stitches part references into one metadata record without copying data.

Durability machinery. Scrubbers read every block and verify checksums, catching bit rot. When a disk or node dies, the repair service rebuilds missing shards from survivors, prioritizing groups with the fewest remaining shards. Placement across failure domains keeps correlated failures from exceeding parity.

Deletion and versioning. Deletes write a tombstone marker. Space is reclaimed asynchronously after lifecycle expiry, using reference counting for deduplicated or copied chunks.

Failure modes and trade-offs

• Disk failure: routine. At 100 PB, expect several failures per day; repair bandwidth must exceed failure rate or durability falls.
• Zone loss: shards spread across at least three zones so 10+4 tolerates one zone only if no zone holds more than four shards; plan shard counts accordingly.
• Metadata hot shard: split automatically; throttle abusive clients.
• Listing consistency: strongly consistent listings cost more; older systems offered eventual listing.
• Small-object overhead: billions of tiny objects make metadata the bottleneck. Pack small objects into larger extents.
• Cost tiers: tiering to HDD and archive trades retrieval latency for price per GB.

Key takeaways

• Split the control plane (metadata, placement) from the data plane (chunk nodes) so each scales on its own.
• Erasure coding delivers high durability at roughly half the replication cost, at the price of repair complexity.
• Commit metadata last; orphan cleanup is easier than recovering partially visible objects.
• Continuous scrubbing and fast repair are what actually produce eleven nines.
• Spread shards across racks and zones, and size parity to the largest failure domain you must survive.