Where do you put the cache, and how big does it need to be?

The layers, and the question that picks one

There are five places a response can be held, and they differ along exactly two axes: how many requests can reuse one copy, and how far from the origin the copy sits, which is the same as how hard it is to correct once wrong.

A client cache in the browser or the mobile app is free, has zero network cost on a hit, and serves exactly one user. It is unreachable — you cannot evict it, so anything you put there must be safe to serve for its full lifetime. A CDN or edge cache serves every user in a region from one copy, which is where the leverage is for anything public, and it can be purged, though purge propagation is measured in seconds to minutes and not instantly. A reverse proxy in front of your own fleet does the same job with less geographic reach and more control. An in-process cache inside the application is the fastest possible hit, tens of nanoseconds against the hundreds of microseconds a network round trip costs, but there is one copy per instance. A distributed cache such as Redis or Memcached is a shared tier: one copy for the whole fleet, invalidatable in one operation, at the cost of a network hop and its own availability.

The question that picks between them is who else wants these exact bytes. A rendered marketing page is wanted by everyone, so it belongs at the edge. A feature-flag configuration blob is wanted by every instance and changes rarely, so an in-process cache with a short refresh is right and putting it in Redis adds a network hop to something that could be a field read. A user's session or shopping basket is wanted by one user but by every instance that might serve them, which is exactly the shape a distributed cache exists for. Answering "Redis" to all three is the answer that gets probed.

Sizing is a working-set question, not a data-volume question

The dataset size is nearly irrelevant. What you must cover is the working set — the distinct objects actually requested inside your chosen retention window — and access to almost anything user-facing is heavily skewed, so a small fraction of the objects account for most of the requests.

Make the assumptions explicit and the arithmetic falls out. Suppose 40 million product records, an average serialised size of 2 KB, and a measured pattern where the 400,000 most-requested products serve the bulk of traffic. Caching the full dataset is 40M × 2 KB = 80 GB, which needs a sharded cluster and is expensive to keep warm. Caching the hot 400,000 is 400,000 × 2 KB = 800 MB, which fits comfortably in one node's memory with room for overhead, and captures most of the benefit. The right move in an interview is to state the skew as an assumption to be validated by measurement, not to invent a percentage and present it as fact — the discipline of naming it as an assumption is itself part of what is being scored.

Then add the overheads that people forget. Key strings, per-entry metadata and the allocator's fragmentation are real; budgeting only for serialised values understates the requirement. And you need headroom above the eviction threshold, because a cache running at its memory limit evicts on every write and the hit rate collapses precisely when traffic is highest.

Reason in miss rate, because that is what the origin sees

Two derivations are worth having ready.

Latency: if a cache hit costs 1 ms and a miss costs 1 ms plus a 40 ms origin fetch, then the average is 1 + (1 - h) × 40. At a 90 percent hit rate that is 5 ms; at 95 percent, 3 ms; at 99 percent, 1.4 ms. Diminishing returns, and a candidate who only quotes averages stops here.

Origin load is the interesting one, and it moves the other way. Origin queries per second are (1 - h) × total. At 100,000 requests per second, a 99 percent hit rate sends 1,000 to the origin and a 95 percent hit rate sends 5,000 — the same four-point change that barely moved the average latency has quintupled the load on the thing you were protecting. This is why hit rate must be watched as a first-class alarm rather than a dashboard curiosity, and why a small regression in cacheability, such as someone adding a per-request query parameter that fragments the key space, can take an origin down while every latency percentile still looks acceptable.

The layer that gets less effective as you scale

A per-instance in-process cache has a property that surprises people: its hit rate depends on the fleet size. With requests spread evenly across 50 instances, each instance sees a fiftieth of the traffic for any given key, so each must fetch and hold its own copy. You pay 50 origin fills for the first request to each key instead of one, hold 50 copies of the hot set in memory, and — because each instance expires independently — you have up to 50 differing views of the same record at once. Scale out to 200 instances and all three get four times worse.

That is not an argument against local caches; it is an argument for using them where the effect is acceptable. It is acceptable for a small, uniformly hot, slowly changing dataset where every instance wants everything anyway. It is not acceptable for a large key space with a long tail, which is what the shared tier is for. The strong version of the answer combines them deliberately: a short-lived local cache for the very hottest keys to absorb the request-coalescing problem, a shared cache behind it for breadth, and an explicit statement that the local layer's TTL is now the floor on how fast an invalidation can take effect fleet-wide.

Getting HTTP caching right buys the edge for free

For anything served over HTTP, the cache metadata is the API. Cache-Control: max-age tells clients how long to reuse without asking; s-maxage sets a different, usually longer, lifetime for shared caches only, which is exactly the split you want when the CDN can be purged and browsers cannot. stale-while-revalidate allows a cache to serve the expired copy immediately and refresh in the background, which converts an expiry from a latency spike into no user-visible event at all. An ETag with a conditional request lets a client revalidate cheaply: a 304 Not Modified costs a round trip but no body, so it is the right tool when objects are large and change unpredictably, whereas a longer max-age is right when they are small and staleness is tolerable.

The one to be careful with is Vary, and this is where a caching change becomes a security incident. A shared cache keys on the URL plus the headers you nominate in Vary, so a response personalised to a signed-in user and returned without Cache-Control: private can be stored at a shared layer and handed to the next person who requests that URL. The rule is that anything derived from a credential is private, and anything that legitimately varies by a request header must name that header in Vary — noting that varying on something high-cardinality, a raw User-Agent for instance, fragments the key space so badly that the entry is effectively uncacheable. Deciding what to cache is also deciding what is not tenant-specific, and that decision deserves to be made out loud.

Size the cache from the working set and alarm on the miss rate, because origin load is linear in misses while user-visible latency is not. A hit rate that drops four points looks harmless on a latency graph and is a five-times traffic multiplier on the tier you were shielding.