Backend Engineering

What this area actually covers

Everything a server does on behalf of a caller it cannot see. Concretely: the protocol you speak, the shape of the interface you expose, how you decide who the caller is and what they may do, how state is stored and kept consistent, how the service behaves when a dependency is slow or gone, and how you find out what happened afterwards.

It is easiest to define by its two neighbours, because it sits between them. Below it is the language and framework layer - writing correct code, choosing data structures, making a query fast. Above it is system design - capacity, partitioning, geography, the shape of an architecture on a whiteboard. Backend engineering is the middle: given that the code works and the architecture is chosen, make this one service a well-behaved participant in a network of unreliable peers.

In interview terms that middle is a real category. A coding round asks whether you can implement. A design round asks whether you can plan a system at scale. A backend round asks whether you know what a 409 means, why your retry made an incident worse, and where you put the authorisation check. Candidates who prepare only the two ends are visibly unprepared for the middle, which is where most day-to-day work lives.

What gets wrongly bundled in: database internals, which is its own area and deeper than most backend roles require; infrastructure and deployment, which is DevOps even though backend engineers must be conversant in it; and framework fluency, which is the most commonly mistaken-for-the-whole-thing part of the job. Knowing Spring or Express thoroughly is table stakes, not the differentiator.

There is one more boundary worth drawing, because it decides how a whole interview goes. Backend engineering is not the same as writing business logic. Business logic is the part that changes when the company changes its mind about how refunds work; backend engineering is everything that has to remain true regardless - that the refund happens once, that the caller is allowed to ask for it, that a timeout does not turn into two refunds, and that when it goes wrong you can say which request it was. Candidates who describe only the first half sound like they have implemented a lot of tickets. Candidates who describe both sound like they have owned something.

The seven areas underneath

This section is divided into seven subsections, and the division is not arbitrary - each one is a place where backend engineers reliably get asked something they cannot bluff. Read them in order if you are starting from scratch, or jump to the two that match the job description in front of you.

HTTP & REST covers methods and status semantics, caching headers, content negotiation, and the differences between HTTP/2 and HTTP/3. It exists separately because the protocol is the one part of your interface that infrastructure you do not control acts upon: a proxy caches on your headers, a client library retries on your status codes. Expect questions that look like trivia and are not, such as which methods are safe, what a 409 means as opposed to a 422, and why Cache-Control and ETag interact the way they do.

Authentication & Authorisation covers sessions versus tokens, the OAuth 2 and OpenID Connect flows, the specific ways JWT validation goes wrong, and role-based against attribute-based access control. It is separate from general security because it is the part every backend engineer implements personally rather than delegating to a specialist, and because the two words in the title name genuinely different decisions. Inside you will find the flow diagrams, the list of claims that must be checked, and the reasoning behind short-lived access tokens paired with refresh tokens.

Microservices covers service boundaries, data ownership, the saga and outbox patterns, and the honest case for a monolith. It exists as its own subsection because the questions here are architectural but examined at the backend level: not "design a system" but "these two services both write this table, what is wrong and how would you unpick it". Expect a lot of material on why a shared database defeats the point of splitting.

Resilience Patterns covers timeouts, retries, idempotency keys, dead-letter queues and designing for partial failure generally. This is the subsection that most directly carries the theme of the whole area, and it is where the difference between a mid-level and a senior answer is widest. What you will find is each mechanism explained by the failure it answers, rather than as a catalogue.

GraphQL & gRPC covers schema design, the N+1 resolution problem, streaming RPC, and how to pick a protocol for a given consumer. It is separated from HTTP & REST because the trade-offs are genuinely different: GraphQL moves query composition to the client and hands you a caching and cost-control problem, while gRPC gives you a strict contract and streaming at the price of browser ergonomics and human-readable traffic.

Distributed Systems Theory covers time and ordering, leader election, quorums, idempotence as a formal property, and the classic impossibility results. It sits here rather than in system design because backend engineers meet it concretely - clock skew breaking a "latest write wins" rule, a quorum read returning something a previous read did not. The material is the smallest amount of theory that changes what you build.

Backend Scenarios covers production situations to work through out loud: a latency spike, memory growing steadily across a week, a cascading failure, a deploy that broke something an hour later. It exists because a growing share of backend interviews are diagnostic rather than definitional, and practising the narration - what you would look at first, what would rule out what - is a separable skill from knowing the answer.

Where it sits in a real system

Follow one request. It arrives at an edge - a load balancer or gateway - which terminates TLS, may enforce rate limits, and picks an instance. The service authenticates the caller, usually by validating a token rather than by looking anything up. It then authorises the specific action, which is a different decision made with different information. It validates input, does its work, and in doing so calls a database and probably two or three other services. It writes a response, emits a log line and a metric, and possibly publishes an event that other services consume asynchronously.

flowchart TD
    A[Edge terminates TLS and rate limits] --> B[Authenticate the token]
    B --> C[Authorise this action on this resource]
    C --> D[Validate and normalise input]
    D --> E[Apply business rules]
    E --> F[Write to the owned datastore]
    F --> G[Emit event for downstream consumers]
    G --> H[Respond and record metrics]

The step worth staring at is the gap between the second and third boxes. They look like one security check and they are two entirely separate decisions, made with different information, and the second is the one that gets skipped.

The two places where design lives are visible in that walk. The first is the boundary: what this service owns exclusively, versus what it asks another service for. Get it wrong and you get a distributed system with a shared database, where two services write the same table and no one can change a schema or reason about a transaction. The second is the call graph: every synchronous call to another service is a dependency you have taken on that service's availability and tail latency, and those multiply. Three sequential calls each succeeding 99.9% of the time give you a worse number than any of them, and the arithmetic is unforgiving as call graphs deepen.

That arithmetic is worth doing once by hand, because it changes how you feel about adding a dependency. If your endpoint calls three services and each is independently available 99.9% of the time, the availability of the composed path is 0.999 cubed, which is about 99.7%. Over thirty days that is the difference between roughly forty-three minutes of failure and roughly two hours. Nothing in your code got worse. You simply took on two more ways to fail, and the only fixes are to remove a dependency from the critical path, make it optional with a degraded response, or make it asynchronous.

This is also where the synchronous/asynchronous decision earns its keep. A caller who needs an answer now gets a request/response call. Work that merely must happen gets an event or a queued message, which converts an availability dependency into a latency one - your service can accept the order even while the fulfilment service is down, at the price of the order being fulfilled later and of having to handle the message arriving twice.

The last row is the one people learn the hard way. A synchronous system fails loudly and a queued system fails silently, so an asynchronous design does not remove your obligation to monitor - it changes what you monitor from error rate to consumer lag.

The vocabulary you will hit on day one

Interviewers use these words as though everyone shares a definition, and most disagreements in a backend interview are actually definitional. Learn them precisely once and a whole class of confusion disappears.

Two of those deserve a sentence more. Idempotency is a statement about state, not about responses: a DELETE that returns 204 the first time and 404 the second is still idempotent, because the resource is absent either way. And exactly-once is not something a message broker can give you across a network partition; what you build is at-least-once delivery plus a consumer that recognises a repeat, and the combination behaves as though it were exactly once.

The HTTP contract, read properly

Most backend interviews contain a stretch where you are asked what status code you would return, and it is not trivia. A status code is the part of your interface that generic infrastructure understands: proxies cache on it, clients retry on it, and monitoring alerts on it. Getting it wrong means a load balancer retries a request that must not be repeated, or a client gives up on something it should have tried again.

The method semantics from RFC 9110 are the base layer, and the two properties that matter operationally are safety and idempotency.

The interesting rows are the last two. POST is not idempotent, which is why every create-a-charge, place-an-order or send-a-message endpoint needs a client-supplied idempotency key if it is going to be safe to retry. And PATCH is idempotent when it sets a field to a value and is not when it increments one, which is a distinction worth naming aloud in an interview because it shows you are reasoning from the semantics rather than from a memorised table.

On status codes, the useful discipline is to decide what the class means before arguing about the exact number. A 4xx says the caller must change something before retrying; a 5xx says the caller may retry unchanged and hope. That single rule resolves most disputes. A request that is well-formed but forbidden by current state is a 409, and it belongs in the 4xx class because retrying it byte-for-byte will fail again. A request rejected because you are shedding load is a 503, ideally with a Retry-After, because the caller genuinely should try again. And a request that failed validation is a 400 or 422, never a 500, because a 500 tells your own alerting that you have a bug when you have a rude client.

Two codes are worth memorising precisely because candidates confuse them. 401 means the request lacked valid authentication, and it should carry a WWW-Authenticate header saying how to authenticate. 403 means you know exactly who the caller is and they may not do this. Returning 403 where you meant 401 sends a client into a pointless retry loop; returning 401 where you meant 403 makes them re-authenticate against a problem that authentication will never solve.

Who does this work

The builders are backend engineers, API engineers where an organisation's product is its interface, and integration engineers whose whole job is other people's systems, contracts and quirks. A typical day is less writing than deciding: what this endpoint should return when the input is valid but the state forbids the action, whether this operation can be retried safely, why a p99 latency doubled after a deploy that changed nothing obvious.

The operators are site reliability and platform engineers, who own the runtime, the deployment path, and the alerting - and who will ask you, during an incident, questions your code should have made answerable. Technical leads and architects specify boundaries and contracts, and in most organisations senior backend engineers do this work without the title.

The distinction worth keeping is between people who own a service in production and people who contribute code to it. The first group carry a pager and consequently design differently. That difference shows up in interviews within about five minutes.

It shows up as a change in what a candidate volunteers. Ask someone who has only contributed code how they would add a new field, and you get a schema change and an API version. Ask someone who has been paged, and unprompted you get the deployment order - add the column nullable, deploy readers that tolerate its absence, backfill, then start writing it - because they have watched a migration and a deploy race each other. Nobody asked about deployment order. The habit of thinking about it is the signal.

Demand, adoption and how that is changing

Demand is very high and unusually stable, because backend work is where business rules live and business rules keep changing. Every organisation with software has more backend work than people. Three specific pressures keep it that way: the long migration of monolithic systems onto services and managed platforms, which is nowhere near finished; the growth of machine-to-machine integration, where an organisation's API is the product rather than a detail of it; and cost pressure, which turned efficiency from a nice property into a funded project.

The newest driver is that AI features are backend features. A language-model-backed capability still needs authentication, rate limiting, timeouts, retries with a budget, caching, cost attribution and a graceful answer when the provider is degraded. That is ordinary backend engineering applied to an unusually slow and expensive dependency, and it has created demand rather than displaced it. It has also made a formerly niche concern mainstream: when a single call can take tens of seconds and costs real money per invocation, streaming responses, request deduplication and per-tenant budgets stop being refinements and become the design.

What is genuinely commoditising is hand-rolled infrastructure: you do not write a load balancer, a queue, or a token verifier, and increasingly you do not manage the servers. Boilerplate generation has compressed the value of framework recall while leaving the value of judgement untouched. What has not commoditised is deciding where a boundary goes, what a contract promises, and how a system behaves when half of it is unavailable.

The interview format has shifted to match. A round that once asked you to recall how dependency injection works in your framework now more often hands you a scenario - this endpoint times out intermittently under load, here is what the dashboards show - and watches how you narrow it. That change rewards people who have debugged something real and penalises people who prepared by re-reading documentation.

What makes it hard

The defining difficulty is that failure is partial. Not "the system is up or down" but "the payment provider accepted the charge and then the connection dropped before you learned that". Every remote call has three outcomes, not two: success, failure, and unknown. The third is the one that generates production incidents, because handling it requires you to have decided in advance whether the operation can be repeated safely.

That is what idempotency is for, and it is the single clearest separator between two candidates with identical framework experience. One says "I retry on failure". The other says "the client sends a key it generates, the server records that key with the result of the first attempt, and a repeat of the same key returns the stored result instead of charging again". Same years of experience, different depth, and the second answer only comes from having watched a customer get charged twice.

sequenceDiagram
    participant C as Client
    participant A as Orders API
    participant P as Payment provider
    C->>A: POST order with key K
    A->>P: Charge card
    P-->>A: Timeout, outcome unknown
    C->>A: Retry POST order with key K
    A->>A: Key K found, first attempt recorded
    A-->>C: Same order id, no second charge

Look at the fourth and fifth lines rather than the timeout. The client behaves identically on both attempts and knows nothing about the failure; all the correctness sits in the server recognising a key it has seen.

Timeouts and retries compound the problem rather than solving it. A retry is extra load applied at precisely the moment a dependency is struggling, so naive retries turn a slow dependency into a dead one. The corrective habits are a timeout on every remote call including the ones the library defaults to infinite, a bounded number of attempts, exponential backoff with jitter so that clients do not synchronise into waves, a retry budget so retries cannot exceed some fraction of normal traffic, and a circuit breaker that stops calling a dependency that is clearly failing. Each is simple. Knowing that you need all of them together, and why, is not.

The dual-write row deserves emphasis because it defeats intuition. Writing to your database and publishing an event look like one logical action but are two operations with no shared transaction, so any crash between them leaves the system inconsistent with no error recorded anywhere. The standard fix is to write the event into a table in the same transaction as the data, and have a separate process publish from that table - the transactional outbox. Arriving at that unaided is unusual; recognising the problem when a system misbehaves is the more valuable skill.

Designing for partial failure, mechanism by mechanism

The resilience patterns are usually taught as a list, which hides the fact that they compose into one decision path per outgoing call. Walking that path once makes the list memorable, because each element is answering the previous element's failure.

flowchart TD
    A[Outgoing call needed] --> B{Breaker open}
    B -- Yes --> C[Fail fast or serve degraded]
    B -- No --> D[Call with a deadline]
    D --> E{Result}
    E -- Success --> F[Record success and return]
    E -- Retryable error --> G{Attempts and budget left}
    E -- Permanent error --> C
    G -- Yes --> H[Wait with jittered backoff]
    H --> D
    G -- No --> C

The branch to notice is that a permanent error and an exhausted budget arrive at the same place. Both mean stop, and treating a 400 as retryable is how a client burns its budget on a request that will never succeed.

A deadline is stronger than a timeout and is the concept most candidates are missing. A timeout is local - this call gets three seconds. A deadline is a point in time that travels with the request through every hop, so a service that received a request with 800 milliseconds remaining does not start a two-second call downstream. Without propagated deadlines, work continues on requests whose callers have already given up, which is exactly the load you least want during an incident.

A circuit breaker is a small state machine wrapped around a dependency, and its value is that it converts a slow failure into a fast one. Slow failures are worse than fast ones because they hold resources.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate over threshold
    Open --> HalfOpen: cooldown elapsed
    HalfOpen --> Closed: trial calls succeed
    HalfOpen --> Open: trial call fails

The half-open state is the whole design. Without it a breaker either stays open forever or slams the recovering dependency with full traffic the instant the cooldown ends, and the second is how a service that was about to recover gets knocked over again.

A bulkhead limits how much of your capacity any one dependency can consume. If every request handler shares one thread pool or one connection pool, a single slow dependency drains it and endpoints that never touch that dependency start failing. Separate pools per dependency mean the damage stays where it started. This is the mechanism behind the observation that an outage in a minor recommendation service can take down a checkout page: nothing was logically coupled, but the resources were.

Load shedding and backpressure are the pair that handles the case where the problem is you. When your queue of pending work is longer than you can serve within the deadline, accepting more work makes every response worse and helps nobody, so the correct behaviour is to reject early with a 503 and a Retry-After rather than to accept and time out. Candidates almost always describe scaling up here, and scaling up is right eventually; shedding is what you do in the ninety seconds before the new instances are ready.

Consistency across a service boundary

Consistency is the conceptual leap. A transaction within one database gives you guarantees you learn to rely on, and crossing a service boundary silently removes them. There is no distributed transaction available to you in practice, so multi-service operations become sequences of local commits with compensating actions when a later step fails - which means designing the business-level undo and accepting that intermediate states are visible.

The transactional outbox is the smallest useful example, and it is worth seeing as schema rather than as prose, because the entire trick is that two writes go into one transaction.

-- Both statements commit together or neither does. That is the whole mechanism.
BEGIN;

INSERT INTO orders (id, customer_id, total_pence, status)
VALUES ('ord_8812', 4210, 2599, 'placed');

INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
VALUES (gen_random_uuid(), 'ord_8812', 'OrderPlaced',
        '{"orderId":"ord_8812","totalPence":2599}', now());

COMMIT;

-- A separate poller reads unpublished rows, publishes them, and marks them sent.
-- It may publish the same row twice after a crash, which is why every consumer
-- must be idempotent. At-least-once is the guarantee you actually get.

Two consequences follow and both are interview-relevant. The publisher is at-least-once, so consumers must deduplicate on the event id or be naturally idempotent. And the outbox table grows forever unless something prunes it, which is the kind of operational detail that separates a design that has been run from a design that has been drawn.

Compensation is the other half. When a three-step business operation fails at step three, you cannot roll back steps one and two because they were committed in other systems, so you issue business-level reversals: cancel the reservation, refund the charge, mark the order abandoned. This is a saga, and the honest thing to say about it in an interview is that the difficulty is not the happy path but the compensations themselves failing, which is why each compensating action must be idempotent and retryable and why the orchestrating state must be persisted rather than held in memory.

Finally, note what you gain by not crossing the boundary. If two pieces of data must change atomically and always will, that is strong evidence they belong in the same service and the same database. A large share of distributed-consistency pain is self-inflicted by a boundary drawn in the wrong place, and the strongest answer to "how would you handle the distributed transaction here" is sometimes "I would move the boundary so there is not one".

Where security quietly fails

Security failures here are quiet. A missing authentication check is a bug you find. A missing authorisation check is a bug an attacker finds: the endpoint works, the token is valid, and nothing stops user A from passing user B's identifier. That class - checking who you are but not what you may touch - remains the most common serious web vulnerability, and it survives code review precisely because the happy path is indistinguishable from correct.

The structural fix is to make the authorisation decision impossible to omit rather than to remember it. That means scoping every data access by the caller's identity at the query level, so that SELECT ... WHERE id = ? AND tenant_id = ? is the only way the repository can be called, rather than fetching by id and checking ownership afterwards in a branch someone can forget to write. Ownership checks written as separate if statements are exactly as reliable as the discipline of the person adding the next endpoint.

Token handling is the other recurring gap. A bearer token is a bearer instrument: anyone holding it is the user, which is why it must be short-lived, transmitted only over TLS, never logged, and never placed in a URL where it will land in access logs and referrer headers. When a candidate says they validate a JWT, the useful follow-up is what they validate - signature, issuer, audience, expiry, and the algorithm being the one you expected rather than whatever the token's own header claims. Trusting the token's header to tell you how to verify the token is a well-known and entirely avoidable mistake.

The last quiet failure is the one in your logs. Backend services accumulate personal data in log lines, error messages and traces without anyone deciding that they should, and the discovery usually happens during an audit rather than a review. Deciding what a log line may contain is a design decision that belongs at the same moment as the schema, not afterwards.

Why study it

It is the highest-leverage general skill in software engineering and the least fashion-dependent. HTTP semantics, idempotency, transaction boundaries and failure handling are the same problems in every language, and time invested in them does not depreciate the way framework knowledge does. It is also the widest door: more roles want a competent backend engineer than any other kind. And backend depth is what makes a system design answer believable - candidates who study design directly produce architecture-shaped answers with no failure reasoning underneath, which an interviewer detects with one question about a dependency timing out.

There is a second, less obvious argument. Backend work gives you an unusually direct feedback loop with reality. A frontend mistake is often visible and forgiving; a backend mistake is invisible and permanent, because it wrote something. That asymmetry makes the discipline an excellent teacher: you learn quickly to think about what happens between two statements, because that is where the state you cannot recover gets created.

Do not bother if what you enjoy is what users see; frontend and product engineering are a different craft and skill in one does not transfer. If your goal is machine learning modelling, backend competence helps you ship but is not the shortest path. And if you want to avoid operational responsibility, this is the wrong choice, because the interesting problems are inseparable from being on the hook when they occur.

Your first hour

Build one endpoint that survives being called twice. Any language, any framework, in-memory storage is fine. Accept a POST that creates something and requires the caller to supply an idempotency key in a header. Store the key with the created resource. On a repeat of the same key, return the original result and the original identifier rather than creating a second resource. Then decide and write down what you do when the same key arrives with a different body, because that is a real case and the answer is a deliberate error rather than either of the two easy options.

The state you need is small enough to write out, and writing it out is most of the learning:

key            -> the client-supplied idempotency key, unique per caller
request_hash   -> hash of the request body, to detect key reuse with new content
status         -> in_progress | completed
response_code  -> what the first attempt returned
response_body  -> what the first attempt returned
created_at     -> so the record can expire, typically after 24 hours

The in_progress state is the one people leave out, and it is the one that matters. Two concurrent requests with the same key must not both execute; the second should either wait or receive a 409 saying a request with this key is already running. Without it your idempotency layer protects against sequential retries and does nothing against the concurrent ones that a retrying client actually produces.

Then break it deliberately. Add a fake downstream call that sleeps for ten seconds, give it a one-second timeout, and watch what your framework does to the request. Add three retries without backoff and observe the load you have just generated. Add jitter and observe the difference. Then remove the timeout entirely and hold the connection open until you run out of workers, so that you have seen resource exhaustion happen on your own machine rather than read about it.

The artefact is a single file plus four sentences: what your endpoint returns on a repeat, what it returns on a key conflict, what your timeout is and why, and how many attempts you allow. Those four sentences answer a large share of what a backend interview asks.

What this is not

It is not system design. Design asks how to build something at scale; backend engineering asks how to make one service correct and well-behaved. They are examined separately because they are different skills, and depth in the second is what makes the first believable.

It is not framework knowledge. Two candidates with the same years in the same framework routinely differ enormously, and the difference is almost always failure reasoning rather than API recall.

It is not the same as DevOps, though the boundary is genuinely blurred and shrinking. You must understand deployment, containers and observability; you are not usually the person who owns the cluster.

It is not REST, either, despite the two words being used interchangeably in job advertisements. REST is one interface style among several, and a service exposing gRPC, GraphQL, a message contract or a plain RPC endpoint over HTTP is doing the same engineering with different serialisation and different failure ergonomics. Treating the style as the discipline leads to arguments about URL design in place of arguments about what the interface promises when it is called twice.

It is not a settled argument about microservices, and treating it as one is a tell. Services buy independent deployment and clear ownership at the cost of network calls where you previously had function calls, distributed debugging, and consistency problems you did not have before. Which side of that trade is right depends entirely on organisational size and change rate, and the strong answer in an interview names the trade rather than the preference. The most credible thing a candidate can say about a monolith is that it was the correct choice for the team that had it.

No amount of reading substitutes for having been paged for a partial failure, because that is what converts idempotency, timeouts and outboxes from vocabulary you can define into instincts you reach for before the bug exists.