Storing and Distributing Multi-GB Container Images

Question

Design a container image distribution service. Users push images that are hundreds of MB to several GB, pull them by tag or digest, list available images, and search.

Docker interview candidates · Answer

Problem Design a container image distribution service. Users push images (hundreds of MB to several GB each), pull them by tag or by digest, list the images and tags that exist, and search for images. Cover the API surface, the data model, the push and pull flows, how concurrent pushes to the same tag behave, how unused data is reclaimed, and a capacity estimate. A reasonable planning load is about 5M pulls per day. Hints 1. Separate an image's name from its actual content. Users pull by tag, but tags are only pointers to immutable, content-addressed layers, which is what makes deduplication across millions of pulls possible. 2. You need a metadata store mapping tags to digests and a blob store keyed by content hash (sha256:...). A pull is two lookups: resolve the tag, then fetch only the layers the client lacks. 3. Define push and pull endpoints, a database holding manifests and layer references, and object storage where each layer lives once under its digest. Then decide how concurrent tag pushes and garbage collection work. Approach Start naive: store every image as one tarball. Push uploads the file, pull downloads it. It breaks at once because every pull re-downloads the whole image even when one layer changed, and the same base layer is stored thousands of times. 1. Split images into layers identified by content hash. Two images sharing a base store it once, and clients download only layers they do not have. 2. Separate metadata from blobs. Tags (latest, v1.2.3) are mutable pointers, layers are immutable. The database maps (image name, tag) - manifest digest; a manifest is JSON listing layer digests. Tag resolution is a database query, layer fetch is an object-store key lookup. 3. Push path: before uploading a layer the client issues a HEAD on its digest. Because storage is content-addressed, the check is just a hash lookup, and existing layers are skipped. 4. Upload by streaming (chunked transfer) to object storage or a temp file; never buffer a multi-GB layer in memory. 5. Concurrency: the tag flip is a single-row atomic UPDATE, last writer wins. Manifest and blobs are committed before the flip, so readers never see a half-written state. 6. Garbage collection: an overwritten tag orphans the old manifest, but its blobs cannot be deleted instantly because a client may be mid-pull. A periodic mark-and-sweep over the manifest-to-blob join table deletes unreferenced blobs. 7. Scale: pulls are read-heavy and blob traffic is bandwidth-bound, so put a CDN in front of immutable layers. The tags, manifests and blobs tables are small enough that one PostgreSQL primary with read replicas suffices; pushes are only 1-2% of pull traffic. Solution A content-addressed blob store plus a metadata layer for tags and manifests. Tables (all timestamps default to now): Push: start an upload, stream the blob, finalize with the digest (server re-hashes and moves it to permanent storage under sha256:<hash ), then PUT the manifest; the registry checks every referenced blob exists, stores the manifest and atomically repoints the tag. Capacity: 5,000,000 / 86,400 is about 58 TPS average, design for about 250 TPS peak (3-5x). Base layers are 25-80 MB compressed, app layers 1-50 MB, a typical image 200-800 MB over 5-10 layers. If 100K unique layers at 50 MB arrive daily, that is 5 TB/day raw and about 1.8 PB/year, but dedup leaves roughly 30-50%. Protocol and structure: an image is a manifest (small JSON: ordered layer list plus a config reference), gzipped tar layers, and a config blob (env, entrypoint, labels). The manifest digest is the immutable identity. Clients speak the open OCI distribution protocol, so any compliant registry interoperates. Large layers use a resumable chunked upload: POST opens a session UUID, repeated PATCH sends chunks, PUT with the expected digest finalizes; after an interruption the client resumes from the last confirmed chunk. Auth: short-lived (about 5 minutes) signed bearer tokens scoped per action, for example repository:myorg/myapp:pull. Supply chain: sign images/manifests (cosign, Notary) and scan on every push. Abuse and cost: popular public images are pulled far more often than private ones, and serving immutable layers from a CDN takes 90%+ of the bandwidth off the registry. Anonymous pulls are rate-limited by IP, authenticated users get higher limits, and a pull-through cache mirror saves bandwidth and limit pressure. Search: tag listing is a DB query. Full-text search over names and metadata goes to Elasticsearch or similar, updated asynchronously (eventually consistent is fine). Caching: blobs by digest are immutable, so cache forever at the CDN (offloads 90%+ of bandwidth). Manifests by digest are also immutable; manifests by tag get a short TTL (for example 60 seconds). Pitfalls - A layer still referenced by another image cannot be deleted; only manifests are deleted, and a layer goes only when no manifest references it (reference counting or reachability sweep). - Moving a tag does not

Storing and Distributing Multi-GB Container Images

Problem

Hints

Pitfalls

Talking Points

Follow-ups

Preparation

Comments

Problem

Hints

Hint 1

Hint 2

Hint 3

Approach

Solution

Pitfalls

Talking Points

Follow-ups

Preparation

Comments