When do you scale out instead of up, and what does that force you to change?

Vertical first, and unapologetically

A single larger machine is the cheapest capacity you will ever buy, because it costs you no distributed-systems complexity at all. No partial failure, no coordination, no cache coherence between nodes, no debugging a bug that only appears when two replicas disagree. Modern instance sizes are large enough that a great many services which "need to scale horizontally" would run on one box with room to spare, and saying so in an interview is a credibility signal rather than a naive one.

Vertical scaling runs out for three reasons, and it is worth being precise about which one applies. The first is a hard ceiling: there is a largest instance, and past it there is nothing to buy. The second is cost curvature, since the top of a machine family is priced disproportionately and two mid-sized nodes often beat one large one on throughput per unit cost. The third, and usually the real one, is that one machine is one failure domain. A single node cannot be patched, redeployed or rebooted without an outage, so any availability target above "we accept scheduled downtime" pushes you to at least two nodes regardless of load. Candidates who frame horizontal scaling purely as a throughput decision miss that redundancy is frequently the motive, and that is precisely why the second node arrives long before the first one is saturated.

What statelessness has to exclude

"Stateless" does not mean the service holds no data; it means no request depends on state that lives only in the process that served an earlier request. That is a stronger and more inconvenient claim than it sounds, and it rules out four things people forget. In-process session objects, because the next request may land elsewhere. Local in-memory caches used as sources of truth rather than as caches, because two replicas will diverge and the divergence is invisible until a user sees inconsistent results. Writes to the local filesystem, including uploaded files staged before processing, because the node that holds them may be terminated by an autoscaler. And anything scheduled in-process, such as a timer that expires a record in thirty minutes, because scaling to three replicas silently triples the work and scaling to zero loses it.

The fix in each case is to move the state to something shared and durable: sessions into a token or a session store, caches to a distributed cache with an explicit invalidation story, staged files to object storage, timers to a queue with a delivery delay or a leader-elected scheduler. Once that is true, replicas become interchangeable, and interchangeability is what actually makes a fleet scalable — you can add nodes, remove nodes, and replace nodes mid-request without any of those events being a user-visible occurrence.

Choosing a balancing algorithm

Round robin is correct when requests cost roughly the same, which is more often false than candidates assume. As soon as one endpoint does a 5 ms cache read and another does a 900 ms report generation, round robin will happily send a node its Nth expensive request while a neighbour sits idle, because it is counting requests rather than measuring load. Least-connections, or better, least outstanding requests, is the right default under heterogeneous cost: it approximates load by what is currently in flight, so a node stuck behind slow work stops receiving new work automatically. Modern proxies commonly implement this by sampling two backends at random and picking the less loaded of the pair, which gets most of the benefit of a global view without maintaining one.

Hash-based balancing solves a different problem: affinity. If each backend maintains an expensive per-key cache, routing by a hash of the key means the same key lands on the same node and the cache actually hits. Consistent hashing is the version worth naming, because plain modulo hashing remaps nearly every key when the node count changes, which empties every cache simultaneously at exactly the moment you were adding capacity under load.

upstream api {
    # Counting in-flight requests, not requests dispatched: the right
    # default when endpoint cost varies by two orders of magnitude.
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 backup;
}

The layer matters too. A layer-4 balancer forwards connections and is cheap, protocol-agnostic and the only option for non-HTTP traffic, but it cannot see requests, so it cannot retry a failed one, route on a path or header, or terminate TLS with per-request policy. A layer-7 balancer sees requests and can do all of that, at the cost of parsing every byte and becoming a component with its own timeouts and buffering behaviour that will eventually appear in an incident.

Health checking deserves a sentence because it is where balancing goes wrong in production. Checks must be shallow enough not to fail for reasons unrelated to that node's ability to serve, or you get correlated failure: a slow shared dependency fails everyone's check, the balancer removes the whole fleet, and an outage that would have been degraded service becomes total. Removing capacity under overload is how a brownout becomes a blackout, which is why balancers need a floor below which they stop evicting backends.

Sticky sessions are a deferral, not a design

The move that costs candidates the round is reaching for sticky sessions to make a stateful tier appear horizontally scalable. It works in a demo and it quietly keeps every problem statelessness was meant to remove. Losing a node now logs its users out rather than costing them a retry, so node replacement is user-visible and deployments become disruptive by construction. Scaling in does not rebalance existing sessions, so the load stays where it was. Balancing quality degrades, because assignment is now decided by a cookie or a client address rather than by load, and a proxy or corporate NAT presenting thousands of users behind one address concentrates them on one node. Worst, it hides the coupling: the code still assumes it can keep things in memory between requests, so the day you need to move a session you find the state was never serialisable.

Stickiness is legitimate as a performance optimisation on top of a design that is already correct without it — warm local caches, long-lived connections such as WebSockets that have to land somewhere. The test is simple and worth stating aloud: if you turned stickiness off, would the system still be correct, only slower? If the answer is no, you have a stateful tier with a routing trick on it, and the interviewer will find that out by asking what happens when a node dies.

The bottleneck moves; it does not vanish

Adding replicas to a stateless tier is easy precisely because that tier was never the hard part. Every one of those replicas talks to the same database, the same cache, the same third-party API, and the same connection pool limits, so throughput plateaus at whatever is shared and the plateau arrives faster than the arithmetic suggested. Worse, more replicas means more connections to the database, and a database can be pushed into contention by connection count alone, so scaling out the app tier can reduce total throughput. This is why the honest version of the answer treats horizontal scaling of stateless services as the easy half, and the partitioning of whatever they all depend on as the design problem that actually needs your remaining time.