Python across backend, data and automation

What this area actually covers

Python is a dynamically typed, garbage-collected, interpreted language with an unusually large standard library and an even larger third-party ecosystem. The language itself is small enough to learn in a fortnight. What takes years is the ecosystem, and the ecosystem is not one thing — it splits into three markets that happen to share a syntax.

Backend web means Django, FastAPI or Flask serving HTTP, backed by Postgres, with Celery or a similar worker for background jobs. Data and machine learning means NumPy, pandas or Polars, scikit-learn, PyTorch, and increasingly the orchestration and serving layer around models. Automation, scripting and platform work means glueing systems together: CI pipelines, cloud SDKs, infrastructure tooling, log processing, internal command-line utilities.

These three overlap far less than outsiders assume. A Django engineer of ten years' standing may have never opened a Jupyter notebook. A machine-learning engineer may have no view on how to structure a REST API, and no reason to. The skill that generalises across all three is understanding how the runtime behaves; the libraries do not transfer.

There is one more distinction worth drawing at the outset, because it silently governs a great deal. Python is a language specification with several implementations, and in practice almost everyone means CPython, the reference implementation written in C. Statements about the global interpreter lock, reference counting and C extension modules are statements about CPython rather than about the language. PyPy, a just-in-time compiling implementation, and the various embedded and alternative runtimes behave differently, and the reason they have never taken over is precisely the C extension ecosystem that the reference implementation defines. Knowing that the implementation and the language are separable is the difference between reciting a constraint and understanding one.

What Python is not, for hiring purposes, is the notebook-and-plot activity most people first encounter. That is a genuine use, but it is the shallowest end of a market where the paid work is production systems: data pipelines that must run every night, services that must not fall over, models that must be retrained and redeployed.

The seven areas underneath

The seven subsections split along a deliberate line: the first five are about the language and its runtime and are common to every Python job, and the last two are about the frameworks and the practice that most Python roles are actually measured on. Someone preparing for an interview in a hurry should work through core Python, then concurrency, then the framework their target role uses.

Core Python

Core Python covers the data model: names as bindings rather than boxes, mutability and what it implies for defaults and aliasing, scoping and closures, truthiness, equality versus identity, shallow versus deep copying, and comprehensions. It is a separate area because these are the semantics that every other subsection assumes, and because they are the ones the syntax actively disguises — a = b looks like a copy and is not, and a startling proportion of production Python bugs are that one sentence played out at scale. Expect questions that seem trivially easy and turn out to be probing whether you know what an object is.

OOP in Python

Dunder methods and the protocols they implement, the method resolution order and how multiple inheritance is linearised, descriptors, properties, class versus instance attributes, dataclasses, and metaclasses at the far end. This exists separately because Python's object system is protocol-based rather than inheritance-based: an object is iterable because it implements a method, not because it inherits from an interface, and that reverses much of the intuition carried over from Java or C#. What you will find here is less about designing class hierarchies — Python codebases tend to be flat — and more about how the machinery you use every day is wired.

Decorators, Generators & Iterators

Closures, higher-order functions, decorators with and without arguments, generators and the yield protocol, generator expressions, context managers, and the composable utilities you build from them. It is its own area because laziness is a distinct design tool: a generator lets you process a file larger than memory or an unbounded stream without changing the shape of your code, and understanding that is the difference between a pipeline that scales and one that loads everything first. Decorators earn their place here because they are the mechanism behind an enormous amount of framework behaviour, and reading one you did not write is a genuine skill.

Concurrency & the GIL

Threads, processes, asyncio, and the reasoning that picks between them. The global interpreter lock is the reason this cannot be treated as an afterthought: what it does and does not prevent determines which of the three models will help you, and the answer depends entirely on whether your workload is waiting or computing. This is a separate subsection because it is the most commonly misunderstood topic in Python interviews and the one where a memorised slogan is most easily distinguished from understanding. You will find questions about async/await mechanics, about when multiprocessing is worth its serialisation cost, and about mixing blocking code into an event loop.

Python Memory & Performance

Reference counting, the cycle-detecting garbage collector layered on top of it, object overhead, profiling with the standard tooling, and identifying where interpreter overhead dominates versus where you are simply waiting on something else. This area exists because "Python is slow" is a claim nobody should make without a profile, and because the fixes are so different depending on the cause: vectorise, move to a compiled extension, add concurrency, or fix the query. It is also where you learn that the correct optimisation is very often to do less work rather than to do the same work faster.

Django & FastAPI

The two dominant server frameworks, treated together because most backend Python roles use one of them and the comparison is instructive. Django brings an ORM, migrations, an admin interface, authentication and a strong set of conventions; FastAPI brings type-driven validation, generated API documentation and asynchronous handling built on ASGI. The subsection covers ORM behaviour and the N+1 query problem, middleware, dependency injection, async endpoints and the hazards of blocking inside them, and request validation. It exists separately because framework behaviour is where backend Python interviews spend most of their time, and because both frameworks have specific, well-known failure modes an interviewer can reliably probe.

Testing in Python

pytest fixtures and their scopes, parametrisation, mocking and patching and knowing where to point them, test doubles for external services, and reading a coverage report honestly. Testing gets its own area in a dynamically typed language for a reason that does not apply as strongly elsewhere: the compiler catches almost nothing for you, so the test suite is carrying load that a type checker carries in other ecosystems. The questions here are about boundaries — what to patch, what to let run, when a test that mocks everything has stopped proving anything.

Where it sits in a real system

In backend work Python occupies exactly the position Java or Node does: it receives a request, applies logic, reads and writes a database, and emits events. Django brings an ORM, an admin interface, authentication and migrations in one opinionated package, which is why it dominates product teams that need a lot of functioning software quickly. FastAPI takes the opposite approach — small, typed, built on ASGI for asynchronous handling — and has become the default for APIs whose main job is to sit in front of something else, very often a model.

In data work Python sits above the compute rather than doing it. This is the single most important structural fact about the data stack and the one beginners miss: pandas and NumPy are Python interfaces to array operations implemented in C and Fortran. When you write a vectorised expression you are not running a Python loop; you are asking a compiled library to run one for you over contiguous memory. When you write an explicit for loop over rows, you are, and it is often a hundred times slower. The same pattern repeats one level up — PySpark code describes a job that a JVM executes, and PyTorch code describes a computation graph that runs on a GPU. Python is the control plane.

flowchart TD
    A[Your Python code] --> B{Where does the work run}
    B -->|Pure Python loop| C[Interpreter, one bytecode at a time]
    B -->|NumPy or pandas| D[Compiled C over contiguous memory]
    B -->|PySpark| E[JVM executors on a cluster]
    B -->|PyTorch| F[GPU kernels]
    C --> G[Slow, and the usual surprise]

The branch worth staring at is the first one: the syntax gives you almost no clue which path you are on, and the performance difference between them is the whole subject of the data-performance conversation.

In platform work Python is the language people reach for when a shell script has become too complicated and a compiled tool is too much ceremony. Cloud provider SDKs, Ansible, and a large share of internal deployment tooling live here.

How a Python service is actually served

One structural detail is missing from most tutorials and comes up constantly in backend interviews: your framework does not talk to the network. Python defines two interface conventions between a web server and an application. WSGI is the older synchronous one, where the server calls your application, waits for it to return, and hands back a response. ASGI is the asynchronous successor, which additionally supports long-lived connections such as WebSockets and lets a single worker interleave many in-flight requests. Django was built for WSGI and has grown ASGI support; FastAPI is ASGI from the start, through Starlette.

In front of that interface sits a process manager. The conventional deployment runs several worker processes — because of the interpreter lock, one process cannot use several cores for Python work — with a supervisor restarting them and distributing connections. That is why a Python service's capacity is usually described in workers rather than threads, why a memory leak shows up as workers being recycled rather than as a crash, and why the sizing question "how many workers per container" is a real one with a real answer derived from cores and from how much of each request is spent waiting.

flowchart TD
    A[Client] --> B[Reverse proxy]
    B --> C[Process manager]
    C --> D[Worker 1]
    C --> E[Worker 2]
    C --> F[Worker 3]
    D --> G[Framework via WSGI or ASGI]
    G --> H[Database and cache]

The thing to notice is that the workers share nothing. Any in-process cache, counter or rate limiter you write lives in exactly one worker, which is why state that feels local has to move to Redis or the database the moment you scale past a single process.

Who does this work

Backend engineers on Python teams live in Django or FastAPI, and their day looks like any backend day: an endpoint, a migration, a slow query, an integration. Data engineers build and operate pipelines, and their day is largely orchestration and failure: a scheduled job that did not produce output, a schema that changed upstream without warning, a backfill that must not double-count. Data scientists frame questions and build models, working mostly in notebooks and answering to a business stakeholder rather than an on-call rota. Machine-learning engineers sit between those two and take the data scientist's model into production, which means packaging, serving, latency, monitoring and retraining. SREs and platform engineers write Python as the glue around everything else.

The distinction that matters for interviews is who builds versus who explores. Exploratory work is judged on the insight and the method; production work is judged on whether it runs unattended at three in the morning. Candidates who prepared for one and interviewed for the other are the most common mismatch in this market.

Demand, adoption and how that is changing

Demand is very high and comes from two independent directions, which is unusual and makes it robust. The first is that Python has become the default teaching language and the default scripting language, so it accumulates users continuously and every organisation ends up with Python somewhere. The second is that the entire machine-learning ecosystem standardised on it. Model training, inference libraries, evaluation tooling and the frameworks around large language models are Python-first, and no credible competitor has emerged. That has pulled a large volume of well-funded work into the language over the past decade.

Two shifts are worth knowing about. Performance has become a first-class concern for CPython itself, with sustained interpreter optimisation work through the 3.11, 3.12 and 3.13 releases; and separately, a free-threaded build of CPython that can run without the global interpreter lock became officially supported as an experimental option in 3.13. Neither means the ecosystem's assumptions have changed yet — most libraries still assume the lock exists — but the direction is real. Meanwhile the tooling layer is being rewritten in Rust: Ruff for linting and formatting, uv for packaging and environments, both markedly faster than what they replace, and both adopted quickly enough that a candidate who has never heard of them looks out of date.

A third shift is quieter and affects daily work more than either. Type annotations have moved from a curiosity to an expectation in serious codebases. They are not enforced by the interpreter, which is the point people miss, but a checker in the pipeline turns them into a real constraint, and frameworks have started deriving behaviour from them — FastAPI reads the annotations on a handler to validate the request and generate the schema. The practical consequence is that a Python interview in a backend role now frequently includes questions that would have been unimaginable a decade ago: how you would model an either-this-or-that value in the type system, what a generic type parameter buys you, why a checker complains about something that plainly works.

The data half of the market has its own direction of travel, and it is towards Python being the interface to something that is not Python. Query engines and dataframe libraries written in Rust or C++ are increasingly exposed through Python bindings, columnar in-memory formats let those libraries hand data to one another without copying, and orchestration tools describe workflows in Python that execute somewhere else entirely. None of this reduces the demand for Python skill; it changes what the skill consists of, away from writing the loop and towards knowing which engine to hand the work to and what the handover costs.

One honest caution about the demand. Because Python is easy to start with, the entry-level end of the market is extremely crowded, and "I know Python" is close to worthless as a differentiator. The demand is concentrated where the syntax is the least of it: production data engineering, ML systems work, and backend services at scale.

What makes it hard

The difficulty is not syntax, and this trips up people who conclude from an easy start that the language is simple. The difficulty is that Python's runtime semantics differ from the mental model the syntax suggests, and the divergence only becomes visible when something breaks.

Names are bindings, not boxes. Assignment does not copy; it points another name at the same object. Every mutable default, every aliased list, every surprising mutation through a second reference follows from that, and no amount of careful syntax avoids it:

def add_item(item, basket=[]):     # the list is created ONCE, when the function is defined
    basket.append(item)
    return basket

add_item("apple")    # ['apple']
add_item("pear")     # ['apple', 'pear']  -- same list, still there

The second hard thing is the global interpreter lock, and it is worth being precise because it is widely misdescribed. The GIL is a lock inside CPython that allows only one thread to execute Python bytecode at a time. So it does prevent multiple threads speeding up CPU-bound pure-Python work: four threads doing arithmetic will not beat one. It does not prevent concurrency in general. Threads still help enormously for I/O-bound work, because the lock is released while a thread waits on a socket or a disk. It does not apply to separate processes, which is why multiprocessing sidesteps it entirely at the cost of having to serialise data between processes. And it does not constrain NumPy or similar libraries, which release the lock while running compiled code — one of the reasons the scientific stack is fast despite the interpreter. A candidate who says "Python can't do parallelism because of the GIL" has learned a slogan; the correct answer names which of these four cases they are in.

The decision that follows is the one interviews actually ask about, and it has a short answer if you frame it correctly.

The row that catches people is asyncio's cost. An event loop only works if nothing on it blocks, so a single synchronous database driver call inside a coroutine stalls every other task in the process — the same failure the Node ecosystem has, arriving in a language where most libraries are still synchronous by default.

flowchart TD
    A[Is the work waiting or computing] --> B[Waiting]
    A --> C[Computing]
    B --> D{Are the libraries async}
    D -->|Yes| E[asyncio]
    D -->|No| F[Thread pool]
    C --> G{Is it array or matrix work}
    G -->|Yes| H[NumPy or a GPU library]
    G -->|No| I[Processes]

The third hard thing is the one nobody teaches, and it does more damage than the other two combined: packaging and environments. Python's default behaviour is to install into a shared location, so two projects wanting different versions of the same library conflict, and system tools break when you upgrade something for an unrelated reason. Virtual environments are the fix, and the fact that they are a bolted-on convention rather than a language feature is why the landscape is littered with competing solutions. Add native extensions — libraries with compiled C, Fortran or CUDA components whose wheels must match your platform and Python version — and you get the failures that consume real days: it works on my machine, it works locally but not in the container, the same requirements file resolved differently last Tuesday. Any senior Python engineer has strong opinions here, formed painfully.

The fourth is memory, and it is subtler than the packaging problem because it rarely announces itself. CPython frees an object when its reference count reaches zero, which is immediate and predictable, and runs a separate cycle-detecting collector for groups of objects that reference each other and would otherwise never reach zero. Two consequences follow. Objects are expensive relative to their contents — a list of a million small integers is not a million machine words — which is why the data ecosystem uses typed arrays rather than lists. And anything holding a reference keeps an object alive, so the classic Python leak is not a leak at all but a cache, a module-level dictionary, or a closure that captured more than you intended.

Why study it

Study Python if you want to work in data or machine learning, where there is effectively no alternative — the entire ecosystem is here, and any other choice means reimplementing what already exists. Study it if you want the fastest route to being genuinely useful in a scripting or platform role, because the standard library plus a cloud SDK covers an enormous amount of real automation. And study it if you want optionality: Python is the most common second language in the industry, so it is rarely wasted.

There is also an argument about the shape of the career it opens. Because Python sits at the boundary between engineering and analysis, it is unusually easy to move sideways from it — a backend engineer who picks up pandas can become useful to a data team, and a data scientist who learns to package and serve becomes an ML engineer. Few other languages sit on that many borders at once. If you do not yet know which kind of work you want, that optionality is worth something real.

Do not study it expecting the ease of entry to convert into an easy job market at the bottom. It will not; the junior end is saturated precisely because the language is welcoming. Do not choose it if you specifically want to learn how machines work — memory layout, ownership, cache behaviour — because Python is designed to hide exactly that, and C, Rust or Go will teach it faster. And if your target is high-throughput low-latency backend services, Python is a defensible choice but not the obvious one, and you should be able to say why you picked it.

Your first hour

Skip the tutorial and go straight to the two things that separate people who know the syntax from people who know the language.

python3 -m venv .venv                 # a project-local environment
source .venv/bin/activate             # on Windows: .venv\Scripts\activate
python -c "import sys; print(sys.executable)"   # prove which interpreter you got

That third command is the point of the exercise: understand what changed when you activated the environment, because environment confusion causes more wasted Python hours than any language feature.

Then write one file that demonstrates the runtime to yourself. Define a function with a mutable default argument and call it three times, printing the result each time. Bind a list to two names, mutate through one, print the other. Write a loop that sums a million integers in pure Python, time it with time.perf_counter, then do the same sum with sum() over a range and compare. Finally, run one CPU-bound function in four threads and then in four processes, time both, and see which one actually got faster.

For the last ten minutes, do something that will save you more time than the rest combined: profile something instead of guessing about it.

python -m cProfile -s cumtime your_script.py | head -20

Read the top of that output and find the line where your program is genuinely spending its time. It is very unlikely to be where you expected. Doing this once, early, establishes the habit that separates the people who optimise Python effectively from the people who rewrite loops that were never the problem.

The artefact is a single annotated script whose output you can explain line by line, plus one sentence in your own words saying what the GIL did to your thread timing. If you can explain that script, you are ahead of most candidates who list Python as a core skill.

What this is not

Python is not slow in the way the claim is usually meant. The interpreter is slow relative to compiled languages, and a numerical workload written as Python loops will be dreadful — but a numerical workload written properly spends its time in compiled libraries, and most production Python is waiting on a database or a network rather than on the interpreter. Naming the specific bottleneck is what distinguishes a real answer from a repeated opinion.

Python is not untyped. It is dynamically typed with an optional annotation system that has become standard practice in serious codebases; those annotations are not enforced at runtime, which is exactly why a separate checker such as mypy or Pyright exists in the pipeline. Notebooks are not a development environment for production code, and shipping one is a recognisable signal. Data science is not data engineering, and machine-learning engineering is a third thing again — the titles are used loosely in adverts and precisely in interviews, so read the responsibilities rather than the heading.

The GIL is also not a property of the Python language. It is an implementation detail of CPython, which happens to be the implementation nearly everyone uses, and the free-threaded work is an attempt to remove it without breaking the C extensions that depend on the guarantees it quietly provided. Stating it as "the GIL" without ever saying "CPython" is a small imprecision that tends to travel with a shallow understanding of the rest.

And "I know Python" is not a claim that survives contact with a specific role. The question that follows it is always which Python, and having a real answer — this stack, this kind of system, these failure modes — is most of what separates a strong candidate from a crowded field.

The three Python markets share a syntax and diverge on everything that gets graded, so the most valuable preparation is deciding which one you are interviewing for before you decide what to revise.