The discipline of running software in production: Linux and networking underneath, containers and Kubernetes in the middle, infrastructure as code and pipelines around it, and reliability treated as a budget you spend rather than a target you hit.
Very high demand24 min readDevOps engineer, Site reliability engineer, Platform engineer, Cloud infrastructure engineer, Build and release engineer, Backend engineer on-call for their own serviceUpdated 2026-07-27
Assumes you know: Comfort at a command line and with a text editor over SSH, One programming or scripting language well enough to write a small tool, A rough idea of how an HTTP request reaches a server
Overview
What this area actually covers
This is the work of getting software from a developer's branch onto machines that
strangers depend on, and keeping it there. Concretely: the operating system and network
beneath a service, the packaging and scheduling that decides where it runs, the pipeline
that builds and promotes it, the declarative description of the infrastructure it needs,
the observability that tells you what it is doing, and the on-call practice that responds
when it stops.
There is a gap between the title and the advertisements. "DevOps" was named as a cultural
argument, that the people who write software should share responsibility for operating it,
and then the industry turned it into a job description for the specialists who build the
machinery that makes sharing possible. So a DevOps advertisement is usually a platform or
operations role: you are hired to give other engineers a safe way to deploy, not to be the
person who deploys everything. Read the tooling list to work out which reality you are
entering.
Several things get wrongly bundled in. Tuning inside the process, garbage collection, query
plans, hot loops, is a backend and performance-engineering concern that merely surfaces
through your dashboards. Data pipelines and warehouses are data engineering, even when they
run on your cluster. Security is shared: you own patching, secrets handling, identity and
network boundaries, but threat modelling and application security review belong elsewhere.
One further clarification saves a lot of wasted study. This is not a tools discipline with a
theory attached; it is a systems discipline with tools attached. The tools change every few
years and the questions they answer do not. Where does this process's memory go, what
happens to an in-flight request when the container is asked to stop, who is allowed to call
this endpoint and how is that enforced, how do we get back to the last known good state -
those are the same questions on a bare-metal fleet in 2010 and a managed cluster today, and
learning them in terms of a specific tool is why so many candidates cannot transfer between
employers.
The section runs bottom-up, which is also the order in which the knowledge is load-bearing.
The first three build from the operating system to the orchestrator, the next three cover
how change reaches production and how infrastructure is described, the seventh is the
reliability discipline layered over all of it, and the last is where you practise applying
the whole stack to a situation under time pressure.
Subsection
What it is for
Linux & Networking
The layer every abstraction above is built from, and every incident bottoms out in
Containers & Docker
Packaging a process with its dependencies, and what isolation really means
Kubernetes
Declaring desired state and letting controllers reconcile it across a fleet
CI/CD
Getting a tested artefact from a commit to production safely and reversibly
Infrastructure as Code
Describing infrastructure so it can be reviewed, reproduced and diffed
Cloud Platforms
Compute, network, identity and managed services across the major providers
SRE & Reliability
Reliability as a measured objective with a budget, plus incident practice
DevOps Scenarios
Situations to work through out loud, which is how the interview is run
Linux & Networking covers processes and signals, file descriptors, DNS and TLS, and
debugging connectivity from first principles. It is first because it is the substrate: the
out-of-memory killer, an exhausted descriptor limit, a slow resolver and an expired
intermediate certificate are the actual causes behind incidents that present as "the
platform is broken". Expect material on reading dmesg, following a connection with
tcpdump or ss, and knowing precisely what SIGTERM does to your process.
Containers & Docker covers namespaces and cgroups, image layering, multi-stage builds
and container security basics. It exists separately from Kubernetes because most confusion
about orchestration is really confusion about containers: a container is an ordinary
process with a restricted view of the system, and once that lands, cgroup throttling and
image size and the reason a shell inside the container sees a different filesystem all stop
being mysterious.
Kubernetes covers scheduling, controllers and reconciliation, services and ingress,
probes, and resource management. It gets its own subsection because it is the current centre
of gravity for this field and because its behaviour is rule-governed rather than arbitrary.
You will find the mechanics behind a pod that will not schedule, a rollout stuck on a
readiness probe, and a container throttled by a CPU limit.
CI/CD covers pipeline design, artefact promotion, deployment strategies and safe
rollback mechanics. Separate because it is the change-management half of the discipline, and
because the interesting questions are about sequencing and reversibility rather than about
YAML: build once and promote the same artefact, keep the pipeline fast enough that people do
not route around it, and make rollback a rehearsed operation rather than an improvisation.
Infrastructure as Code covers Terraform state and drift, module design, and expressing
several environments without copying everything. It is here because writing the
configuration is easy and owning the state is not, and because the genuinely hard parts -
blast radius of a plan, how modules compose, what to do when reality and state disagree -
are conceptual rather than syntactic.
Cloud Platforms covers compute, networking, identity and managed-service selection
across Azure, AWS and GCP. It exists as a subsection because the providers differ in
vocabulary far more than in concept, and the useful skill is mapping a need onto whichever
catalogue is in front of you. Identity gets disproportionate weight here, because in a cloud
account identity policy is the real access control and a misconfigured role is the most
common serious mistake.
SRE & Reliability covers service level indicators and objectives, error budgets,
incident command, on-call design and blameless postmortems. It is separated because it is a
way of deciding rather than a way of building: it gives you a defensible answer to how
reliable something should be and what to do when it is not, and those answers are
organisational as much as technical.
DevOps Scenarios covers situations to work through: a failing rollout, a node that has
run out of resources, a leaked credential, a three-in-the-morning page. It is last because
it exercises everything above, and it exists as its own subsection because the interview
format here is overwhelmingly scenario-based and narrating a diagnosis is a separable skill
from being able to perform one.
Where it sits in a real system
Every layer here exists to hide the one below it, and the job is knowing when that hiding
leaks.
Layer
What it gives you
What leaks through
Cloud account and network
Machines, subnets, identity, managed services
Region and zone failure, quota limits, IAM being the real access control
Host operating system
Processes, file descriptors, a network stack
Memory pressure, disk filling, ephemeral port exhaustion, conntrack tables
Container runtime
An image, an isolated process tree
cgroup limits causing throttling, and a process that is still just a process
Orchestrator
Desired state, scheduling, service discovery
Eviction, DNS behaviour, the scheduler refusing to place a pod
Deployment pipeline
A tested artefact promoted between environments
Environment drift, and a rollback that only rolls back code
Observability and on-call
Knowing, and being told
Signals that describe the machine rather than the user's experience
The load-bearing idea is desired state and reconciliation. You do not tell the platform to
start three copies of something; you declare that three should exist, and a controller works
continuously to close the gap between that declaration and reality.
Loading diagram…
The loop back from the corrective action is the part worth internalising: the controller
does not execute a plan to completion, it re-observes after every step, which is why it
recovers from interference and why it will fight you if you change something by hand.
Kubernetes is built entirely from this pattern, Terraform applies a batch version of it to
cloud resources, and once you see it, half the surprising behaviour in both stops being
surprising: the system is not doing what you asked, it is repeatedly trying to make the
world match what you wrote. It also explains the characteristic frustration of the field. If
you scale a deployment by hand and something scales it back, you have not found a bug; you
have found a second controller with a different opinion about the desired state, and the fix
is to change the declaration rather than the world.
A deploy, followed end to end
Almost every DevOps interview contains some version of "walk me through what happens when a
developer merges a change". It is a good question because it touches every subsection at
once, and because the interesting content is in the parts people skip.
Loading diagram…
Look at the second box and the fourth. Building once and referring to the result by digest
rather than by a mutable tag is what makes the rest of the diagram meaningful, because
otherwise the thing you tested and the thing you deployed are only probably the same.
Three details in that flow carry most of the difficulty. The first is configuration:
promoting the same artefact between environments only works if the environment-specific
parts are injected rather than baked in, and the moment someone builds a
staging-specific image the promotion guarantee is gone. The second is the health check that
gates the rollout, which must reflect user-visible success rather than process liveness - a
readiness probe returning 200 from a handler that does not touch the database will happily
certify a completely broken release. The third is that rollback is only cheap for the parts
that are stateless. Code rolls back in seconds; a schema migration and a message consumer
that has already processed events in the new format do not, which is why the expand and
contract pattern exists and why "we can always roll back" is a claim that needs qualifying
in front of an interviewer.
Deployment strategies are enumerable and each buys something specific.
Strategy
How it works
Buys you
Costs you
Recreate
Stop the old, start the new
Simplicity, no version overlap
Downtime
Rolling
Replace instances a few at a time
No downtime, no extra capacity
Two versions live at once
Blue-green
Run a full second environment, switch traffic
Instant switch and instant rollback
Double the infrastructure during the switch
Canary
Send a small share of traffic to the new version
Real evidence before full exposure
Needs good metrics and traffic splitting
Feature flag
Ship the code dark, enable per cohort
Decouples deploy from release entirely
Flag debt and combinatorial test surface
The row that catches people out is rolling, which is the default in most clusters and which
quietly requires that two versions of your code can run simultaneously against the same
database. Every backward-incompatible change you make is a bet that no request lands on the
wrong version during the overlap.
Who does this work
A DevOps engineer in a mid-sized company spends a day on pipelines, Terraform modules,
cluster upgrades and the unglamorous queue of other engineers' requests: a certificate to
rotate, a new environment, a job that ran out of memory at 2am. Much of the skill is deciding
which of those requests should become self-service instead of a ticket.
A platform engineer treats internal developers as customers and builds a product for them,
whether that is a paved-road deployment template, a developer portal or an opinionated set of
base images. Success is measured by adoption and by how rarely anyone needs to understand the
layer underneath.
A site reliability engineer, where the role exists as described, works to explicit reliability
objectives and is expected to spend part of their time eliminating operational work rather than
performing it. The distinctive artefacts are service level objectives, error budgets, incident
reviews and capacity models.
Then there are people who specify rather than build: cloud architects choosing managed services
and landing-zone structure, FinOps analysts who care that your autoscaling floor is set to nine,
and the compliance function that needs to see who approved a production change. That negotiation,
about how much friction a control is worth, is a real part of the job.
The distinction that matters most when you read a job advertisement is whether the role is
measured by tickets closed or by problems eliminated. A team measured on tickets ends up as an
internal help desk with a Kubernetes cluster; a team measured on eliminated toil ends up
building a platform. Both exist under the same title, and asking during an interview what
fraction of the team's week goes to unplanned requests tells you which one you are joining.
Demand, adoption and how that is changing
Demand is very high because it is structural rather than fashionable. Almost every organisation
now ships software continuously rather than quarterly, and continuous shipping needs machinery
and someone to own it. Cloud migration created a decade of work; the tail of that work, the
applications too awkward to move first, is still going. Cost pressure has made a second wave,
because the same organisations that moved fast now want their bills explained and their
instances right-sized.
The composition is shifting in two directions at once. Below, raw operations work is genuinely
commoditising: managed control planes, managed databases and serverless runtimes have removed a
great deal of what used to be a career, and hand-building a Kubernetes cluster is now a learning
exercise rather than a job. Above, the work is moving towards platform engineering, which is a
product role with an infrastructure substrate. The bar has risen accordingly, because being the
person who knows the console stopped being valuable once the console was documented.
Two smaller currents are worth naming honestly. Vendors sell a "shift left" story in which
developers own everything, and the tooling burden that creates is precisely why platform teams
exist. And AI tooling generates manifests and pipeline configuration competently, without
touching the part that matters, which is deciding whether a change is safe to apply to a system
with live traffic on it.
A third current is regulatory and it is growing. Supply-chain requirements have pushed artefact
provenance, dependency inventories and signed builds from a specialist concern into ordinary
pipeline work, and organisations in regulated sectors increasingly need to show who approved a
production change and reproduce what was running on a given date. That is unglamorous work, it
is not going away, and being able to discuss it credibly distinguishes candidates in enterprise
interviews.
What makes it hard
The fundamentals are unavoidable and quietly do most of the work. Almost every serious
incident bottoms out in Linux and networking: a process killed by the out-of-memory killer, a
file descriptor limit reached, a disk filled by logs, DNS resolution that succeeds slowly, a TLS
handshake failing on an expired intermediate, a connection blocked by a rule three hops away. No
abstraction removes this, because the abstractions are built out of it. An engineer who can read
dmesg, follow a packet with tcpdump, and reason about what a SIGTERM does to their process
is diagnosing while others are restarting things.
The signal-handling example is worth expanding because it is asked constantly and answered
poorly. When an orchestrator removes a pod it sends SIGTERM and waits for a grace period before
sending SIGKILL. A process that ignores SIGTERM, or a shell wrapper that receives it and never
forwards it to the child, will be killed abruptly with in-flight requests still open. The correct
behaviour is to stop accepting new work, finish or fail what is in progress, close connections
cleanly and exit, all inside the grace period. Every dropped request during a routine deploy
traces back to some part of that chain being absent.
Kubernetes is the current centre of gravity and its complexity is a real cost. It is
genuinely good at what it does, keeping declared workloads running across a fleet that is always
partly broken, and if you work in this field you will meet it. But it is a distributed system you
now operate in addition to yours, with its own failure modes, its own upgrade cadence and an
ecosystem of components you must keep current. Behaviour that reads as arbitrary usually is not:
a pod that will not schedule, a rollout stuck because a readiness probe never passes, a container
throttled because its CPU limit is enforced as a quota per period, a node evicting workloads under
memory pressure, all follow from rules you can learn.
resources:requests:# used by the scheduler to decide where this fitscpu:"250m"memory:"256Mi"limits:# enforced at runtime; CPU throttles, memory killscpu:"1"memory:"512Mi"
Requests are a scheduling promise and limits are a runtime ceiling, and the asymmetry matters:
exceeding a CPU limit slows you down, exceeding a memory limit ends the process. Getting these
wrong is the most common cause of a service that is mysteriously slow under load while every
dashboard looks healthy.
The two probe types are the other configuration people conflate, and the consequences differ
sharply. A readiness probe controls whether traffic is sent to this instance; failing it removes
the instance from the load balancer and leaves it running. A liveness probe controls whether the
instance is killed and restarted. Pointing a liveness probe at a health endpoint that checks a
shared database turns a database blip into a cluster-wide restart loop, because every instance
independently concludes it is unhealthy at the same moment. Liveness should test the process, and
readiness should test whether the process can currently serve.
Infrastructure as code makes the hard part explicit rather than removing it. Writing Terraform
is easy; owning state is not. State is a record of what the tool believes exists, so it can be
wrong in both directions, and drift, someone changing a resource in the console, is the normal
condition rather than an anomaly. The real difficulty lives in module boundaries, in expressing
three environments that differ in four ways without copying everything, in the blast radius of a
plan that proposes to replace a database, and in the discipline of reading a plan properly before
applying it.
Reading a plan properly has a specific meaning that is worth stating. Additions are usually safe
and modifications usually are too; the line to search for is the one that says a resource will be
destroyed and recreated. That happens when a property the provider cannot change in place is
altered, and on a database, a disk or a static address it means data loss or an outage rather than
an update. The habit of scanning every plan for replacements before applying is the single most
valuable thing to bring to a Terraform interview.
Reliability is a budget, not an absolute. This is the conceptual leap the SRE framing offers,
and it is worth taking even if you never hold the title. You choose a service level objective, say
99.9% of requests served successfully over 30 days, which permits roughly 43 minutes of failure in
that window. That permitted failure is an error budget, and it is meant to be spent: on releases,
on migrations, on experiments. If the budget is intact you are being too cautious and should ship
faster; if it is exhausted, the argument about whether to freeze changes has been settled by data
instead of seniority.
The arithmetic is worth having at hand, because interviewers ask for it and the numbers are easy
to derive rather than memorise. A 30-day window contains 43,200 minutes, so the permitted downtime
is that figure multiplied by one minus the objective.
Objective over 30 days
Permitted failure
What that realistically allows
99%
About 7 hours 12 minutes
Generous; suits internal or batch systems
99.5%
About 3 hours 36 minutes
Routine maintenance windows still fit
99.9%
About 43 minutes
One bad deploy consumes most of it
99.95%
About 22 minutes
Requires automated rollback, not human response
99.99%
About 4 minutes 20 seconds
No single-region dependency survives this
The last row is the one to raise when somebody asks for four nines. Four minutes over a month is
shorter than the time it takes a person to be paged, wake up and open a laptop, so committing to
it means committing to automated detection and automated remediation, and to removing every single
point of failure including the ones in the deployment path. That is a large programme of work
rather than a target, and being able to say so calmly is a mark of seniority.
The hard part is not the arithmetic but choosing an indicator that reflects what a user experiences
rather than what is easy to measure. The proportion of successful requests measured at the edge is
usually a better indicator than instance uptime, because the instances can all be up while every
request fails. And once you have chosen an indicator you must hold the line on it, which is a
political skill rather than a technical one.
Production is where experience is not substitutable. The judgement of what to look at first,
when to stop investigating and restore service, how to change a system that is currently on fire
without making it worse, is built from incidents, and it is what interviewers probe hardest with
scenario questions.
Identity is the real perimeter
Network boundaries used to be the control that mattered, and in a cloud account they are no
longer the primary one. What decides whether a compromised workload can read your customer
data is the identity it runs as and the policy attached to that identity, which is why a
leaked credential is the scenario interviewers reach for and why the answer is more
structured than "rotate it".
Loading diagram…
The transition to notice is the loop through rotation. A credential that rotates
automatically on a short cycle turns an exposure into a bounded window, whereas a static key
checked into a repository three years ago has no expiry to rely on.
Three habits follow from that. Prefer workload identity over long-lived keys, so that a
process proves what it is to the platform and receives a short-lived token rather than
carrying a secret at all. Scope each identity to the smallest set of actions and resources it
genuinely needs, because the blast radius of a compromise is exactly the policy you attached.
And keep secrets out of images and environment files that end up in logs, using a secret
store the platform injects at runtime. When the scenario question comes, the strong answer
sequences it: revoke first, then work out what the credential could reach and whether it was
used, then rotate everything in that blast radius, and only then discuss how it leaked.
The first fifteen minutes of an incident
Because the scenario question is the dominant interview format here, it is worth having an
explicit order of operations. The one thing being assessed is whether you separate restoring
service from understanding the cause, and whether you can say which you are doing at any moment.
Start by establishing impact in user terms rather than system terms: which users, doing what, are
affected, and is it total or partial. That framing determines urgency and it is also what anyone
who joins the call needs first. Next, ask what changed, because the overwhelming majority of
incidents follow a change - a deploy, a configuration edit, a certificate expiry, a scheduled job,
a traffic shift, or someone else's release upstream. Checking the deployment timeline against the
start of the symptoms takes thirty seconds and resolves a large share of incidents on its own.
Then mitigate before diagnosing. If a recent change correlates, roll it back and confirm recovery;
you can understand it afterwards with the site up. If nothing correlates, look for the cheapest
containment available - shedding load, disabling a feature flag, failing over, adding capacity -
and be explicit that you are buying time rather than fixing anything.
Throughout, keep two roles separate even if you hold both. Someone must own the decisions and the
communication, and someone must be hands-on the system, and the failure mode of a small team is
that the person typing is also the person everyone is asking for updates. Declaring "I am running
this, you are investigating" out loud in an interview scenario is a strong signal, because it is
what actually happens on a well-run call.
Afterwards comes the postmortem, and the word blameless has a precise meaning that is often
missed. It does not mean nobody made a mistake; it means the review treats a human error as
evidence that the system permitted it, and asks what made the wrong action easy and the right
action hard. A review that ends with "be more careful" has produced no change, whereas one that
ends with a confirmation prompt, a guard rail or a removed foot-gun has.
Why study it
The strongest reason is leverage over the rest of your career. Every engineer's work eventually
runs somewhere, and being the person who can explain why it broke changes what you are trusted
with. The skills transfer unusually well, because Linux, networking, TCP, DNS and TLS have been
stable for decades while the tools above them turn over every few years. It suits people who enjoy
diagnosis, who are comfortable being called when nobody knows what is wrong, and who find
satisfaction in removing a class of problem rather than shipping a feature.
There is also an unusual breadth argument. Few roles give you a view across every service an
organisation runs, and that vantage point makes you the person who notices the pattern - that
three teams have independently built the same fragile thing, or that the same class of incident
keeps recurring under different names. Turning that observation into a platform capability is
some of the highest-value work available to an engineer at any level.
Be honest about who should skip it. If you want to build products users see, this is a detour. If
you want depth in algorithms, machine learning or compilers, the daily work here is not that. And
if on-call is unacceptable to you, say so early, because a great many roles here carry a pager and
the ones that do not sit further from the interesting problems.
Your first hour
Do not start with Kubernetes. Start one layer down, on a single machine, where you can see
everything.
Take the smallest web service you can write in a language you know, and containerise it properly.
Write a Dockerfile with a multi-stage build so the final image holds the artefact and not the
toolchain. Run it with a memory limit low enough to be a problem, docker run --memory=64m, and
watch what happens when the process exceeds it. Then docker exec into the container and look
around with ps, ls /proc/1/fd and cat /proc/1/cgroup, until you are convinced the thing
inside is an ordinary process on your host with a restricted view.
Then do the signal experiment, because it takes five minutes and teaches something most candidates
do not know. Send the container a stop request and time how long it takes to exit. If your process
is started through a shell it will very likely not receive the signal at all and will sit there
until the grace period expires and it is killed. Fix that, add a handler that logs on SIGTERM
and sleeps for two seconds before exiting, and watch the difference in the shutdown timing. You
have just reproduced, on your laptop, the mechanism behind dropped requests during deploys.
Your artefact is a single page holding three things: the image size before and after the
multi-stage build, the exact exit status and message when the container hit its memory limit, and a
five-line description of what happens between the run command and the first served request. If you
cannot yet write the third, you have found what to learn next, which is more useful than a
certificate.
What this is not
It is not the tools. Listing Docker, Kubernetes, Terraform, Jenkins, Prometheus and three clouds on
a CV is the most common shape of a weak candidate here, because the next question is why a pod is in
CrashLoopBackOff and there is no memorised answer. Tool familiarity without systems understanding
collapses within two follow-ups, and the strongest candidates often name fewer tools and explain
them better.
It is not a cloud certification either, though one can be a reasonable structure for learning a
provider's vocabulary. Certifications test recall of managed-service features; interviews test
judgement about trade-offs, which is why people pass the first and fail the second.
It is not automation for its own sake. Automating a process nobody has understood produces a faster
version of a bad process and a new component to maintain, and the discipline's own literature is
clear that the first step is to remove the work rather than to script it.
And it is not the abolition of operations. The work did not disappear when it was renamed: it moved
up a layer, became more automated, and stayed just as consequential at three in the morning.
An abstraction you cannot see through is fine until it fails, and everything in this field
eventually fails, which is why the layer you understand rather than merely use determines how far
you can go.