Backend engineering fundamentals

Resources identified by URLs, manipulated through a uniform set of methods whose semantics the whole internet already agrees on, with each request carrying everything needed to serve it. The practical value is not elegance: it is that proxies, caches, load balancers and client libraries can reason about your API without knowing anything about it, because GET is safe and PUT is idempotent everywhere. Most APIs called REST are HTTP-with-JSON, which is fine — the honest answer names which constraints you actually kept.

When the operation is a verb that does not map onto a resource, and the consumer is another service rather than a browser. "Recalculate this invoice" is not a noun, and forcing it into a resource produces contortions like POST /invoices/7/recalculations. RPC frameworks also give you a generated, type-checked client and a binary encoding, which matters on a hot internal path. The cost is that you lose HTTP's generic tooling — caching, standard status semantics, and the ability to debug with a browser.

It solves the client having to make several round trips or accept over-fetching, by letting the caller specify exactly the shape it needs — genuinely valuable when many different clients consume one backend. The costs land on the server: query complexity must be bounded or a single request can ask for something quadratic, resolvers make N+1 the default rather than the exception so batching becomes mandatory, and HTTP caching largely stops working because every request is a POST to one URL with a different body.

Versioning is not a style preference; each option decides who bears the cost of a breaking change.

1. In the path            GET /v2/orders/7
   Obvious, trivially routable, and cache-friendly.
   Cost: every URL changes, so links and bookmarks break, and you often end
   up maintaining two whole codepaths rather than two representations.

2. In a header            GET /orders/7
                          Accept: application/vnd.acme.v2+json
   URLs stay stable and the resource identity is honest — it is the same
   order either way.
   Cost: invisible in a browser, easy to forget, and caches must be told to
   vary on the header or they will serve v1 to a v2 client.

3. No version; never break
   Add fields, never remove or repurpose them. Make new behaviour opt-in.
   Cost: the schema accretes forever and you carry dead fields for years.

The third is the one that scales, and it is what most large public APIs actually do despite having a /v1 in the path that never changed. The discipline behind it is knowing which changes are breaking: removing a field, renaming one, making an optional request field required, narrowing an accepted range, or adding a new value to an enum a client switches on exhaustively.

The one people miss is that last one. Adding an order.status of PARTIALLY_SHIPPED looks additive, and it breaks every client whose switch statement has no default branch. If you expect an enum to grow, say so in the contract from day one and require clients to tolerate unknown values, because you cannot retrofit that.

That it is stable while the data changes underneath it, which rules out offsets for anything users scroll. An opaque cursor — a token the client passes back without interpreting — lets you change the underlying keyset later without a breaking change, and stops clients constructing their own. Return the cursor for the next page and nothing else about position; the moment you return a total page count you have committed to computing it, which on a large table is the expensive part of the query.

Because a sequential integer leaks your volume and growth rate to anyone willing to create two records a week apart, and it invites enumeration — walking /users/1, /users/2 and discovering what your authorisation actually checks rather than what you assumed it checked. It also couples the public contract to your storage, so sharding, merging two databases, or migrating to a different engine all change identifiers that customers have hard-coded. A separate opaque public identifier, indexed and generated independently, costs one column and removes all three problems at once. The related discipline is that an unguessable id is not an authorisation control: it raises the cost of enumeration and does nothing about a caller who has legitimately seen the id once.

Cross-cutting concerns that would otherwise be reimplemented in every service: authentication, rate limiting, request validation, routing by path to different backends, and response shaping for particular clients. A load balancer chooses an instance; a gateway makes policy decisions about the request. The risk is that a gateway accretes business logic, at which point it becomes a shared component every team must change to ship anything — the distributed monolith arriving through the back door. The line worth holding is that a gateway may reject a request, and must not transform it in ways that encode domain rules.

The error shape, the idempotency semantics of every mutating endpoint, the rate limits and how they are signalled, the pagination scheme, and what "optional" means — absent, or present and null. Those are the parts a client cannot infer and will otherwise discover in production. A schema that documents only the happy path leaves every consumer to invent its own guess about what a 409 means.

GET, HEAD and OPTIONS are safe, meaning they must not change state — a crawler or a prefetching browser will call them uninvited, so a GET that deletes something will eventually delete it by accident. PUT and DELETE are idempotent: repeating them lands in the same state, which is what makes retrying them safe. POST is neither, which is precisely why it is the method that needs an idempotency key.

Choosing between these is a real design decision, because clients and infrastructure behave differently for each.

200 vs 201 vs 202
  201 means created, and must carry a Location header pointing at the thing.
  202 means accepted and not yet done — the honest code for queued work, and
  it obliges you to give the caller somewhere to check.

400 vs 422
  400 is malformed: the body is not valid JSON, a required field is missing.
  422 is well-formed but semantically wrong: a valid date that is in the past
  when it must be future.

401 vs 403
  401 means unauthenticated — I do not know who you are; send credentials.
  403 means authenticated and not permitted — I know who you are, and no.
  Returning 401 for a permission failure sends clients into a login loop.

404 vs 403 for a resource you may not see
  Deliberate: 404 hides existence, 403 confirms it. Choose per resource, and
  choose 404 where the id itself is sensitive.

409 vs 412
  409 conflict: an optimistic-concurrency or duplicate-state failure.
  412 precondition failed: the caller sent If-Match and it did not match.

429 and 503
  Both should carry Retry-After. A 429 without one tells the client to back
  off by an amount it has to guess, and it will guess badly.

The pattern behind the whole table is that a status code is machine-readable routing information, not a mood. Load balancers retry some and not others, browsers cache some and not others, and client libraries throw different exception types. Returning 200 with {"error": ...} in the body defeats all of that — every layer between you and the caller now believes the request succeeded.

Locally there is one hop. In production there are six, and each has its own limits — which is why a request that works on your machine returns a status code your application never produced.

flowchart LR
    C[Client] --> D[DNS and TLS handshake]
    D --> L[Load balancer<br/>idle timeout, TLS termination]
    L --> G[Gateway<br/>auth, rate limit, body size cap]
    G --> S[Your service<br/>thread pool, request timeout]
    S --> P[Connection pool]
    P --> B[Database or downstream]

Reading it as a failure map is the useful exercise. A 413 came from the gateway or the balancer, not from you — your handler never ran. A 502 means something in front got a broken or absent response, so your process died or closed the connection. A 504 means it gave up waiting, so you are alive and slow. A 429 came from the gateway unless you implemented limiting yourself.

The timeouts must be ordered, and almost never are. If the balancer gives up at 30 seconds and your service's own timeout is 60, then for thirty seconds you are computing a response for a connection that no longer exists — holding a thread and a database connection to produce nothing. Each layer's timeout should be shorter than the layer in front of it, so failure surfaces at the innermost point that can report it usefully.

The other thing this diagram makes visible is that the client IP your service sees is the balancer's. Rate limiting or audit logging on the socket address will happily rate-limit your own infrastructure, so both must read X-Forwarded-For — and trust it only when it arrives from the proxy, since a direct caller can set it to anything.

DNS resolution, a TCP handshake, a TLS handshake, then the request travels through a load balancer, probably a reverse proxy or gateway, and finally your process. Each hop can have its own timeout, its own body-size limit and its own idea of how long a header may be, which is why a request that works locally can fail at 413 or 502 in production. Knowing the chain is what lets you locate a failure rather than guess at it.

It reuses an established TCP and TLS connection for subsequent requests, which removes a round trip or three per request — dominant on a high-latency link and significant even inside a data centre. The consequences are that connections are now stateful resources with pool limits on both sides, that an idle-timeout mismatch between client and server produces intermittent connection-reset errors, and that a load balancer distributing connections rather than requests can leave traffic unevenly spread after a scale-up.

Through two mechanisms that people conflate. Freshness — Cache-Control: max-age — lets a cache serve a response without asking you at all, which is the only version that saves a round trip. Validation — ETag with If-None-Match, or Last-Modified with If-Modified-Since — still costs a round trip but returns 304 with no body. no-cache means revalidate every time, not "do not store"; the one that means do not store is no-store, and mixing them up is how private data ends up in a shared cache.

Because everything between the termination point and your service is plaintext. Terminating at the load balancer is normal and means the internal network is a trust boundary you have to actually secure. It also means your service no longer sees the client's IP or protocol, so it must trust X-Forwarded-For and X-Forwarded-Proto — and trust them only from the proxy, because a client that can set those headers directly can spoof its own address and defeat rate limiting.

Authentication establishes who is making the request; authorisation decides whether that identity may perform this action on this resource. They fail differently and are checked at different layers — identity once at the edge, permission at every operation — and conflating them produces the common bug where a service verifies a valid token and never checks that the token's subject owns the record being modified. That is the single most frequent real-world API vulnerability.

A JWT is three base64url segments: a header, a payload of claims, and a signature over both.

eyJhbGciOiJSUzI1NiJ9 . eyJzdWIiOiJ1LTkxIiwiZXhwIjoxNzY... . MEUCIQD...

payload, decoded:
{
  "sub": "u-91",           who
  "iss": "auth.acme.com",  who issued it
  "aud": "api.acme.com",   who it is for
  "exp": 1767225600,       when it stops being valid
  "roles": ["editor"]
}

The signature means any service holding the public key can verify it without calling the auth server, and that is the whole point: verification is local, so authentication does not become a synchronous dependency on every request.

It is also the cost, and it is not fixable. A JWT cannot be revoked. Between the moment you disable an account and the token's expiry, that token is still valid everywhere, because nothing in the verification path consults any state. Logging out is a client-side deletion; the token still works if anyone kept it.

The three real mitigations, none of which is free. Keep the access token short-lived — five to fifteen minutes — and pair it with a long-lived refresh token that is checked against a store, which reintroduces the round trip once per window instead of once per request. Or maintain a denylist of revoked token ids, which is a state lookup on every request and therefore admits you did not want a JWT. Or version the subject, so a token_version claim mismatching the user's current version invalidates every token issued before a password change.

Two implementation traps worth naming. Verify the alg against an expected value rather than trusting the header, or you accept a token signed with none. And verify aud, or a token minted for one of your services is accepted by another that was never meant to trust it.

When you need immediate revocation, when the client is a browser you control, or when the session carries more state than you want on every request. A session id is opaque and meaningless if stolen from storage, revocation is a delete, and the cookie can be HttpOnly so JavaScript cannot read it — which a token in localStorage cannot be. The cost is a session store lookup per request and the need for that store to be shared and highly available.

OAuth2 is a delegation protocol: it gets your application an access token permitting it to act on a user's behalf against some API. It deliberately says nothing about who the user is. OpenID Connect is a thin layer on top that adds an identity token with standard claims, which is what you actually want for "log in with". Using a raw OAuth2 access token as proof of identity is a known antipattern — the token proves access was granted, not to whom.

The redirect dance exists for one reason: to keep the token out of the browser's address bar and history.

sequenceDiagram
    participant B as Browser
    participant A as Your app
    participant I as Identity provider
    B->>A: Requests a protected page
    A-->>B: Redirect to the provider with client id and a state value
    B->>I: Signs in and consents
    I-->>B: Redirect back carrying a short-lived code
    B->>A: Delivers the code
    A->>I: Exchanges code plus client secret, back channel
    I-->>A: Access token and id token

The exchange on the last two lines happens server to server, so the token never touches the browser. The code that did travel through the browser is single-use and expires in seconds, and it is worthless without the client secret.

The state value is not optional. It is a random value your app generates, stores against the session, and verifies on return; without it an attacker can feed a victim's browser a code of the attacker's own, logging the victim into the attacker's account. That is CSRF against the login flow itself.

For a public client that cannot hold a secret — a mobile app or a single-page app — the secret is replaced by PKCE: the client sends a hash of a random verifier up front and the actual verifier at exchange time, so a stolen code cannot be redeemed by anyone else. PKCE is now recommended for confidential clients too, since it costs nothing and closes the interception case.

Role-based access control grants permissions to roles and roles to users, which is simple, auditable and static — it answers "what can an editor do". Attribute-based control evaluates a rule over attributes of the user, the resource and the context, which is what you need for "an editor may modify a document in their own department before it is published". RBAC is the right default and tends to degenerate into hundreds of roles as soon as the answer depends on the resource.

With a deliberately slow, salted, memory-hard hash — Argon2id, scrypt, or bcrypt where those are unavailable — never a general-purpose hash like SHA-256, which is fast and therefore excellent for an attacker. The salt is per user and stored alongside, defeating precomputed tables. The cost parameter is tuned so a single verification takes a noticeable fraction of a second on your hardware, and revisited over time, because the attacker's hardware improves and yours does too.

Both models are fine until something downstream is slow. Then they fail in completely different ways, and the numbers explain both.

Dependency latency rises from 20ms to 2s.

Thread-per-request, pool of 200 threads
  Throughput ceiling = 200 threads / 2s = 100 requests/s
  Was 200 / 0.02 = 10,000/s. Requests beyond that queue, then time out.
  Every thread is asleep in a socket read, using no CPU. The server is idle
  and unable to serve anything — including endpoints that never touch the
  slow dependency, because they cannot get a thread.

Event loop, single thread, non-blocking I/O
  Throughput ceiling is set by memory for pending continuations, not threads.
  10,000 in-flight requests cost callbacks and buffers, not 10,000 stacks.
  Unrelated endpoints keep serving, because nothing is blocked.

The event loop wins here and has a sharper edge of its own: one piece of CPU-bound or accidentally blocking code stalls every request in the process, not just its own. A synchronous file read, a large JSON parse, or a tight loop over a hundred thousand items and the whole server stops responding — including its own health check, so the orchestrator kills it.

Virtual threads close much of the gap for the blocking model, giving you blocking-style code with continuation-style cost, which is why the choice is now less about the model and more about the ecosystem.

The point that survives either way: bulkhead your dependencies. A separate, bounded pool per downstream means a slow one exhausts its own allowance and nothing else. Without it, the resource being shared is what couples services that have no logical relationship at all.

That any instance can serve any request, because nothing needed to serve it lives only in that process. In-memory sessions, a local cache treated as authoritative rather than as an optimisation, an in-process scheduler, a counter, a file written to local disk — each one breaks it, and each is easy to introduce without noticing. It matters because statelessness is the precondition for load balancing, autoscaling and rolling deploys, all three of which assume that a request landing on any instance is equivalent. The useful test is to ask what happens if this request goes to a different pod than the last one, and what happens if this pod is killed mid-request. If either answer is "the user loses something", you have found the state.

From the workload's nature rather than from a round number. For CPU-bound work the pool should be roughly the core count, because more threads than cores adds context switching and no throughput. For I/O-bound work it can be far larger, since threads spend most of their time blocked, and the practical limit becomes memory for stacks plus whatever the downstream can absorb. The number that actually matters is usually the downstream's capacity, not yours: two hundred threads all calling a database that supports forty concurrent queries means one hundred and sixty of them are queueing inside the database where you cannot see it. Size to the constrained resource and queue in front of it.

Concurrency is structuring the program so several things are in progress at once; parallelism is actually executing them simultaneously on different cores. A single-threaded event loop is highly concurrent and not parallel at all, which is sufficient because a typical service spends most of its wall-clock time waiting on I/O rather than computing. The distinction matters when the workload is genuinely CPU-bound, where concurrency alone buys you nothing.

When the caller does not need the outcome to decide what to do next. Sending an email, generating a report, reindexing a document, calling a slow third party for something non-blocking — all of these make the response faster and the request path less fragile. It is wrong when the result determines the response, because queuing does not remove the wait, it hides it and adds a correlation problem. The queue also obliges you to make the consumer idempotent.

Because an unbounded queue converts an overload into a memory leak and a latency collapse. Work accumulates, every response gets slower, and eventually the process dies holding hours of undone work — and until it does, every metric looks fine because nothing is failing. A bounded queue makes the overload visible immediately, at the point where you can shed load or reject with a 503, which is worse for a few callers and survivable for the system.

Because the default in many client libraries is to wait forever, and a dependency that hangs rather than fails will hold your thread or connection until the pool is exhausted — at which point endpoints unrelated to that dependency start failing. A timeout converts an unbounded failure into a bounded one you can handle. It should also be shorter than your own caller's timeout, or you finish work nobody is waiting for.

A retry is additional load applied exactly when a dependency is least able to take it, so the naive version turns a slowdown into an outage.

Baseline 1,000 req/s. Dependency slows. Clients retry 3x, no backoff.
Effective load: up to 4,000 req/s at the worst possible moment.
And every client that failed at the same instant retries at the same
instant, so it arrives as a wave rather than a ramp.

RetryPolicy policy = RetryPolicy.builder()
    .maxAttempts(3)                                  // bounded, not "until success"
    .backoff(Duration.ofMillis(100), 2.0)            // waits of 100ms then 200ms
    .jitter(0.5)                                     // spread the wave
    .retryOn(IOException.class, TimeoutException.class)
    .abortOn(BadRequestException.class)              // 4xx will never succeed
    .build();

Every line answers a specific failure. Bounded attempts stop an infinite loop. Backoff gives the dependency room. Jitter desynchronises clients — the element people omit, and the one that turns a thundering wave into a smooth ramp. And abortOn matters because retrying a 400 is pure waste: the request was wrong and will be wrong again.

The piece missing from almost every implementation is a retry budget: a global cap so retries never exceed, say, ten per cent of normal traffic. Backoff limits one client and says nothing about ten thousand clients each backing off politely.

The precondition underneath all of it is idempotency. Retrying a non-idempotent write does not improve reliability — it creates duplicate charges, and it does so specifically during an incident when you can least afford to reconcile them.

The client supplies a key; the server guarantees the operation happens once no matter how many times the request arrives.

CREATE TABLE idempotency_keys (
  key            TEXT PRIMARY KEY,        -- the constraint IS the mechanism
  request_hash   TEXT        NOT NULL,
  response_body  JSONB,
  status         TEXT        NOT NULL,    -- in_progress | complete
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

On POST /payments with Idempotency-Key: 7f3c...

  INSERT the key with status in_progress, inside the same transaction as
  the payment itself.

  Insert succeeded  -> this is the first time. Do the work, store the
                       response, commit. One transaction, so a crash rolls
                       back both and the key is free again.
  Insert conflicted -> seen before.
       status complete    -> return the stored response, do not redo the work
       status in_progress -> a duplicate is in flight; return 409 and let
                             the client retry shortly

The uniqueness constraint doing the work is the point. Checking for the key and then inserting it is a read-modify-write with a race in the middle, and under retry storms that race is not theoretical — duplicate requests arrive milliseconds apart by construction.

Storing the response is what makes it correct rather than merely safe. A retrying client needs the same answer it would have got the first time, not a 409; returning the original 201 and its body is what lets the client proceed.

The request_hash guards against key reuse with different content. If the same key arrives with a different body, that is a client bug, and returning 422 surfaces it rather than silently replaying an unrelated response. And the table needs a retention policy — twenty-four hours is typical — because it is a log of every mutating request you have ever served.

It fails fast for the whole client rather than per request. Once the failure rate crosses a threshold it stops calling the dependency at all, returning immediately and freeing the connections and threads that would otherwise be waiting. The half-open state is the real design: after a cooldown it lets a few probe requests through, so a recovering dependency is not hit with full traffic the instant it comes back and knocked over again.

Deciding per dependency what a response looks like without it. A product endpoint that omits recommendations when that service is down still serves the product; one that returns 500 has converted somebody else's outage into your own. Making this explicit means each call site has a defined fallback — a default, a cached value, or an omitted field — and that the API contract says those fields are optional, so clients were built to tolerate their absence.

The difference between a usable error and a useless one is whether a program can branch on it without reading English.

{
  "type": "https://api.acme.com/errors/insufficient-funds",
  "title": "Insufficient funds",
  "status": 422,
  "code": "INSUFFICIENT_FUNDS",
  "detail": "Balance 4200 is below the requested 5000, in minor units.",
  "request_id": "req_01J9X2K",
  "errors": [
    { "field": "amount", "code": "EXCEEDS_BALANCE" }
  ]
}

The code is the contract. It is stable, it never gets reworded for clarity, and it is what a client switches on — whereas title and detail exist for a human reading a log and may change freely. Systems that ship only a message force every consumer to string-match, and then a copy edit becomes a breaking change nobody predicted.

The errors array is what makes a form usable, because it maps each problem to the field that caused it. Returning the first validation failure only means the user fixes one thing, resubmits, and discovers the next — an interaction that takes five round trips where one would do.

request_id is the cheapest thing on the list and the most useful during an incident. A user quoting it lets you find the exact trace, which turns "it failed this morning" into a specific request in a specific service.

What is deliberately absent: stack traces, SQL fragments, internal hostnames and identifiers. They are useless to the caller and are reconnaissance for anyone probing the API — and they end up in the client's own logs, which are outside your control entirely.

With a stable machine-readable code alongside the human message, because clients must not parse prose. The status code carries the category, a code field carries the specific condition, and a details array carries per-field problems for validation failures. Include a request id so a user can quote it and you can find the trace. And never leak stack traces or internal identifiers, which are free reconnaissance and tell the caller nothing actionable.

The same event, logged two ways. Only one of them can be queried.

String logging:
  2026-07-27T09:14:22Z INFO Payment 4f21 for user 91 failed after 3 retries

Structured logging:
  {"ts":"2026-07-27T09:14:22Z","level":"info","msg":"payment failed",
   "payment_id":"4f21","user_id":91,"attempts":3,
   "trace_id":"a3f1...","service":"payments","version":"1.14.2"}

The first requires a regular expression to answer "how many payments failed after three or more retries this hour", and that expression breaks the day someone rewords the message. The second is a filter on two fields.

The fields that matter most are the ones nobody adds until an incident. trace_id is what lets you pivot from this line to every other line for the same request across every service. version is what tells you the errors started with a specific deploy, which is the single most useful correlation available and is invisible without it.

Two rules that keep it useful. Log the event once, at the boundary where it is handled, rather than at every level of the stack as it propagates — duplicated logging of one failure makes error counts meaningless. And never log secrets, tokens or full request bodies, because logs are widely readable, retained for months and replicated to systems with weaker access control than your database.

Logs are discrete events with detail, good for "what exactly happened to this request" and expensive at volume. Metrics are aggregated numbers over time, cheap and good for "is the system healthy and is it getting worse", but they cannot tell you about an individual request. Traces show one request's path across services with timing per hop, which is the only one that answers "where did the 900 milliseconds go". You need all three and they answer different questions.

An identifier generated at the edge for each inbound request and propagated to every downstream call as a header, so all the work done on that request's behalf can be found together. The important detail is that it must be threaded through everything — including asynchronous work, where the natural call stack is broken and the id has to be carried in the message payload. A trace that stops at the queue boundary is exactly where the interesting failures live.

Because a time-series metric stores one series per unique combination of label values, so adding a label with high cardinality multiplies your storage and query cost. Labelling a request-count metric by user_id on a system with a million users creates a million series and will take down the metrics backend before it takes down anything else. Labels are for bounded dimensions — endpoint, method, status class, region — and unbounded identifiers belong in logs and traces.

They answer different questions, and conflating them produces a specific and recognisable outage.

livenessProbe:            # "is this process broken beyond recovery?"
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:           # "should this instance receive traffic right now?"
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5

/healthz  checks only the process itself. No dependencies. Returns 200 if
          the event loop is turning and the app has not deadlocked.
/readyz   checks that the database is reachable and warm-up finished.

Now the failure. Put the database check in the liveness probe, and a brief database blip fails liveness on every pod simultaneously. The orchestrator concludes every instance is broken and restarts all of them — at the same moment, into a cold cache, all reconnecting to the database that was already struggling. You have converted a thirty-second dependency wobble into a full outage with an empty cache, and the restart loop continues as long as the dependency is unwell.

Readiness is where dependency checks belong, because failing readiness removes the instance from the load balancer and leaves it running to recover. If every instance fails readiness you have an outage either way, and at least you have not also destroyed your warm state.

The subtler trap is a liveness probe that shares the request thread pool. Under saturation the probe times out because the pool is full, so the orchestrator kills instances precisely when you need the most capacity.

Because a trace per request at full volume produces more data than the traffic itself and costs more to store than the service costs to run. Head-based sampling decides at the edge — keep one per cent — which is cheap and means the interesting request is almost certainly the one you dropped. Tail-based sampling buffers the whole trace and decides after seeing the outcome, so you can keep every error and every request over a latency threshold while discarding the boring majority. It costs memory at the collector and is worth it, because the entire value of a trace is that it is available for the request that went wrong.

Latency, traffic, errors and saturation, measured as near the user as you can get them. They are chosen because they describe whether the service is doing its job without requiring knowledge of how it is built, so they stay valid as the implementation changes. The most common mistake is alerting on causes — high CPU, a full disk — rather than symptoms: high CPU while users are served fine is not an incident, and every instance being healthy is compatible with every request failing.

Because the same build artefact must be deployable to every environment; if configuration is compiled in, staging and production are different binaries and you have tested neither. Environment variables or a mounted config file keep the artefact identical and the differences external and auditable. The corollary is that nothing environment-specific belongs in the repository, and that a missing required variable should fail loudly at startup rather than producing a default that silently points at the wrong database.

They need to be injected at runtime from a secret manager rather than sitting in environment variables baked into a deployment manifest, because manifests get committed, logged and copied into support tickets. They also need rotation without a code change, which means the application should read them in a way that tolerates them changing. And they must be excluded from logs and error reports explicitly, since a config dump on startup is a very common way to publish a database password.

Because rebuilding for each environment means the thing you tested is not the thing you shipped. Dependency resolution is rarely fully reproducible, base images move, and a transitive version that was fine on Tuesday differs on Thursday — so a rebuild for production can contain code that never ran in staging. Building once and promoting the same digest through each environment makes the pipeline a series of gates on one artefact rather than three separate constructions that happen to share a commit. It also makes rollback concrete: you redeploy a specific digest that demonstrably worked, rather than rebuilding an old tag and hoping it produces the same bytes.

Blue/green runs two complete environments and switches all traffic at once, which makes rollback instant — flip back — and means the new version faces full production load with zero prior evidence. Canary routes a small percentage to the new version and increases it while watching error rates, so a bad release affects one per cent of users. Canary needs traffic splitting and automated metric comparison to be worth anything; without the automation it is just a slow deploy.

Between SIGTERM and the process exiting there are in-flight requests, and by default they are simply severed.

process.on('SIGTERM', async () => {
  // 1. Fail readiness first, so the load balancer stops sending new work.
  ready = false;

  // 2. Wait for the balancer to notice. This pause is the part people omit.
  await sleep(5000);

  // 3. Stop accepting, then let in-flight requests finish.
  server.close();
  await Promise.race([drained(), sleep(25000)]);

  // 4. Release resources in dependency order.
  await queueConsumer.stop();
  await db.end();
  process.exit(0);
});

Step two is the one that is always missing and the reason "we do graceful shutdown" often is not true. Closing the server the instant SIGTERM arrives still drops requests, because the load balancer's health check runs on an interval and it will keep routing to you for several seconds after you decided to stop. Failing readiness and then waiting is what actually drains the inbound traffic.

The termination grace period must exceed the total of your waits, or the orchestrator sends SIGKILL in the middle of step three and you have written a slower ungraceful shutdown.

For a queue consumer the same logic applies in reverse: stop pulling new messages, then finish the one in hand, then acknowledge. Exiting with an unacknowledged message is fine if the consumer is idempotent — which it must be anyway — and exiting after doing the work but before acknowledging is what makes that requirement concrete rather than theoretical.

Separating deploy from release, which is the actual value. Code ships dark, gets enabled for internal users, then a percentage, then everyone — so the deployment risk and the product risk are decoupled and a bad feature is turned off without a rollback. The cost is that every flag is a branch in the code and two paths to test, so flags need owners and expiry dates. A codebase with hundreds of stale flags has combinatorial behaviour nobody can reason about.

How to revert the code, and separately what to do about the data, because a migration that ran cannot be undone by redeploying the previous image. That is the argument for expand/contract migrations: if every schema change is backwards-compatible with the previous release, rollback is only a deploy. It should also say who decides, what signal triggers it, and how long it takes — a rollback nobody has rehearsed takes far longer than anyone estimates during an incident.

POST is defined as neither safe nor idempotent because it means "process this according to your own semantics", so the protocol cannot promise anything about repeating it. Your specific endpoint can be made idempotent, and should be if it moves money or creates records, via a client-supplied key. The distinction worth stating is that idempotency is a property you implement, not one the method grants — nothing about choosing PUT makes your handler safe to repeat.

Both come from something in front of your service. A 502 means the proxy got an invalid or no response — your process crashed, closed the connection, or returned something malformed. A 504 means the proxy gave up waiting, so your service is alive and too slow. They point at completely different investigations, and treating them interchangeably is why people restart a healthy service that was merely blocked on a downstream call.

Because it names a category rather than a design. The questions immediately behind it are what is cached, keyed by what, for how long, invalidated how, and what happens on a cold start when the hit rate drops to zero and the origin sees full traffic it was never sized for. A candidate who says "I would cache the product page for sixty seconds, keyed by id and locale, accepting that a price change takes a minute to appear" has designed something.

Anything that assumed exclusivity. A scheduled job now runs twice. An in-memory cache now has two divergent copies. A counter increments in two places. A generated sequence collides. A file written to local disk is visible to half the requests. None of these produce an error — they produce quiet inconsistency, which is why the second instance is where a whole class of latent bugs becomes visible for the first time.

Because it usually checks the wrong thing. A health endpoint that returns 200 if the process is up says nothing about whether requests are succeeding, and one that checks every dependency will fail for reasons outside your control. The useful signal is the service's own error rate and latency measured on real traffic, with the health check reserved for routing decisions. "All pods healthy, all requests failing" is a common and entirely coherent state.

Because it converts a failure into corrupt state plus a log line nobody reads. The method returns as though it succeeded, the caller proceeds on data that was never written, and the actual damage surfaces somewhere unrelated hours later with no connection to the log entry. Catching is only correct when you can genuinely handle the condition — retry it, fall back to a default the caller was told to expect, or translate it into a domain error that propagates. If none of those apply, letting it propagate to the boundary where a request can be failed cleanly is the correct behaviour, and it is what makes the error rate metric mean something.

"What happens the second time this request arrives?" It is answerable only if you have thought about retries, duplicates and partial failure, and it exposes immediately whether the mutating path has an idempotency story or merely assumes the network is reliable. The same instinct shows up as asking what the caller's timeout is before choosing your own, and what the client does with each error you return — treating the contract as something with a consumer rather than an output.