Python fundamentals

It binds a name to an object; it never copies. b = a makes a second name pointing at the same object, so if that object is mutable, changing it through either name is visible through both. This is the single sentence that explains aliasing bugs, mutable defaults, and why passing a list into a function lets the function change your list. Rebinding is different from mutating: b = [1, 2] points b somewhere new and leaves a alone, while b.append(3) changes the object both names share.

The default is evaluated once, when the function is defined, not each time it is called. So a mutable default is shared by every call that does not supply one.

def add_item(item, basket=[]):        # the list is created ONCE, at def time
    basket.append(item)
    return basket

add_item("apple")     # ['apple']
add_item("pear")      # ['apple', 'pear']  - the same list, still there
add_item("fig", [])   # ['fig']            - supplying one avoids it

You can see the shared object directly, which is what turns this from a rule into an explanation:

print(add_item.__defaults__)   # (['apple', 'pear'],) - it lives on the function

The fix is to default to None and build the real default inside:

def add_item(item, basket=None):
    if basket is None:
        basket = []
    basket.append(item)
    return basket

This is not a wart to memorise so much as a consequence of the first answer above: def is a statement that executes, the default expression is evaluated at that moment, and the result is stored on the function object. The same mechanism catches people with datetime.now() as a default, which freezes the timestamp at import time and produces a value that is quietly hours or days old.

is compares identity — whether two names refer to the same object — while == compares value, by calling __eq__. Use is only for singletons, which in practice means None, True and False. It appears to work for small integers and short strings because CPython interns them, so x is 256 may be true and x is 257 false, which is a caching detail rather than a language rule. That inconsistency makes it worse than if it never worked, because the bug survives testing with small values.

Any object can be used in a boolean context, and the rules come from __bool__ or, failing that, __len__. So empty containers, empty strings, zero, and None are all false. The surprise is that this collapses distinctions you may care about: if not count is true for both zero and None, and if not items cannot distinguish an empty list from a missing one. When absence and emptiness mean different things, test is None explicitly rather than relying on truthiness.

A shallow copy creates a new outer object whose elements are the same objects as the original's, so mutating a nested item is visible through both. A deep copy recursively copies everything, so the two are fully independent. list(x), x[:] and copy.copy(x) are all shallow. The practical rule is that a shallow copy is enough when the contents are immutable, and dangerous when they are not — a list of lists copied shallowly shares every inner list.

Because dictionary lookup uses the key's hash, and a mutable object's hash would have to change when it changes, which would make the entry unfindable. Python enforces this by making lists, dicts and sets unhashable, so the failure is a clear TypeError at insertion rather than a silent lookup miss later. Tuples are hashable only if everything inside them is, which is why a tuple containing a list cannot be a key either.

It replaces an instance's per-object __dict__ with a fixed set of descriptors, so attributes are stored in a compact array rather than a dictionary. That saves a substantial amount of memory per instance and speeds up attribute access slightly, which matters when you have millions of small objects. The cost is that you can no longer add attributes not declared in the slots, and multiple inheritance with slots is awkward. It is an optimisation to apply after measurement, not a default.

Passing a mutable object gives the callee the ability to change your data, and nothing in the signature tells you whether it will.

def normalise(tags):
    tags.sort()                    # mutates the caller's list
    return [t.strip().lower() for t in tags]

original = ["  Beta", "Alpha"]
cleaned = normalise(original)
print(original)   # ['  Beta', 'Alpha'] reordered - the caller did not ask for this

The caller wanted a cleaned copy and also got their own list reordered. Nothing raised, and the effect only shows up wherever original is used next — often in a different module.

Two conventions avoid it. Either do not mutate an argument at all and return a new object, which is the default expectation for anything named like a transformation:

def normalise(tags):
    return sorted(t.strip().lower() for t in tags)   # caller's list untouched

Or mutate deliberately, return None, and name the function so it is obvious — which is exactly the convention the standard library follows. sorted returns a new list; list.sort mutates and returns None. That returning of None is not an oversight, it is a signal: a function that returns nothing is telling you its effect was on the argument.

Name resolution searches Local, then Enclosing, then Global, then Built-in, in that order, and stops at the first match. It explains why a local assignment anywhere in a function makes the name local for the whole function, and why shadowing a built-in such as list or id breaks code far away that expected the original. The subtle part is that the decision is made at compile time from whether the name is ever assigned in the body, not from where the assignment sits.

Because the compiler decides a name is local if it is assigned anywhere in the function body, including after the read. So a read that would have found the global instead raises UnboundLocalError, because the local exists but has no value yet. The fixes are global for a module-level name and nonlocal for one in an enclosing function, though needing either is usually a sign the state should be passed in or held on an object.

Closures capture the variable, not its value at the moment of creation. So every function built in a loop sees the loop variable's final value.

handlers = []
for i in range(3):
    handlers.append(lambda: i)       # captures the name `i`, not 0, 1, 2

print([h() for h in handlers])       # [2, 2, 2]

All three closures refer to the same cell, and by the time any of them runs the loop has finished and i is 2. The same shape appears with functools.partial misused, with callbacks registered in a loop, and with default arguments in class bodies.

Two fixes, and the difference between them is worth knowing:

# 1. Bind at definition time using a default argument, which IS evaluated
#    per iteration - the same mechanism that causes the mutable-default trap
#    working in your favour here.
handlers = [lambda i=i: i for i in range(3)]

# 2. Give each closure its own scope by creating one.
def make(i):
    return lambda: i
handlers = [make(i) for i in range(3)]

print([h() for h in handlers])       # [0, 1, 2] either way

Note that a comprehension has its own scope in Python 3, so [lambda: i for i in range(3)] still exhibits the bug — the comprehension's i is shared by the closures it creates, even though it does not leak to the enclosing scope.

A function that takes a function and returns a replacement, applied with the @ syntax as sugar for f = decorator(f). That is the whole mechanism, which is why decorators are ordinary code rather than a special language feature. The practical consequence is that the returned wrapper is what callers see, so without care the original function's name, docstring and signature are lost — which breaks documentation tools, some frameworks, and any code that inspects the function.

The naive version replaces the function with a wrapper whose identity is the wrapper's, and the damage is invisible until something introspects it.

import functools, time

def timed(func):
    @functools.wraps(func)                 # copies __name__, __doc__, __wrapped__
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:                           # runs even if func raises
            print(f"{func.__name__} took {time.perf_counter() - start:.3f}s")
    return wrapper

@timed
def fetch(url: str) -> bytes:
    """Fetch a URL."""
    ...

print(fetch.__name__)   # 'fetch'  - without functools.wraps this is 'wrapper'
print(fetch.__doc__)    # 'Fetch a URL.'

Two details carry the answer. functools.wraps is what preserves the metadata, and omitting it is the most common decorator bug — it breaks Sphinx, confuses debuggers, and makes framework registration by function name silently register everything under wrapper.

The try/finally is the second. A decorator that measures or logs after the call, without a finally, records nothing when the wrapped function raises — which is precisely the case you most wanted timing for.

A decorator that takes arguments needs one more layer, because @timed(2) calls timed(2) first and decorates with whatever that returns:

def retry(times):                    # takes the argument
    def decorate(func):              # takes the function
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            ...
        return wrapper
    return decorate

*args collects extra positional arguments into a tuple and **kwargs collects extra keyword arguments into a dict, which is what lets a wrapper accept any signature and pass it through. In a call rather than a definition, the same symbols unpack: f(*items) spreads a sequence into positional arguments. They are also how you write a decorator that works on any function, which is their most common use in real code.

A / in a signature marks everything before it as positional-only, and a * marks everything after it as keyword-only. Positional-only lets you rename a parameter later without breaking callers, which is why many standard library functions use it. Keyword-only forces callers to be explicit, which is worth doing for boolean flags and for functions with several similar parameters, since send(msg, True, False) is unreadable at the call site.

Because a lambda is an expression that produces a function, and Python's grammar separates expressions from statements. So a lambda cannot contain an assignment, a loop, or a try. That is a deliberate constraint rather than an omission: it keeps lambdas small enough to read inline. If you find yourself wanting more, the answer is a named def, which also gives the function a name in tracebacks — a real practical benefit when something raises.

When the collection has a fixed shape and its elements mean different things by position, such as a coordinate or a database row. Tuples are immutable, so they are hashable and can be dictionary keys or set members, and they signal to a reader that the contents will not change. Lists are for homogeneous sequences that grow and shrink. The performance difference is real but small; the semantic difference is the reason to choose.

Since Python 3.7 it is a language guarantee rather than an implementation detail. The implementation keeps a compact array of entries in insertion order plus a sparse index table of positions into it, which is also why modern dicts use less memory than the older design. The practical consequence is that you can rely on iteration order, which makes dicts usable for ordered configuration and makes OrderedDict mostly unnecessary — though it still has a use for order-sensitive equality and move_to_end.

Four idioms, each for a different situation, and choosing wrongly is a common source of subtle bugs.

counts = {}

# 1. get: read with a fallback, no insertion. Use when you only need the value.
n = counts.get("apple", 0)

# 2. setdefault: read and insert in one step. Note the default is ALWAYS
#    constructed, even when the key exists - expensive if it is not a literal.
group = index.setdefault(key, [])
group.append(value)

# 3. defaultdict: the factory runs only on a miss, which is the usual choice
#    for accumulating.
from collections import defaultdict
index = defaultdict(list)
index[key].append(value)

# 4. Counter: the right answer when you are counting rather than grouping.
from collections import Counter
counts = Counter(words)
print(counts.most_common(3))

The trap with defaultdict is that merely reading a missing key inserts it:

d = defaultdict(list)
if d["missing"]:      # this read creates the key
    ...
print(dict(d))        # {'missing': []} - the check changed the data

That matters when the dict is later serialised or its length checked, and it is why get still has a place: it reads without side effects. The general rule is defaultdict when you are building up a structure, get when you are inspecting one.

Both store unique hashable elements with constant-average membership testing; frozenset is immutable and therefore itself hashable, so it can be a dictionary key or an element of another set. The main reason to reach for a set is membership testing: x in some_list is linear and x in some_set is constant, and converting a list to a set before a loop of membership checks is one of the most common real performance fixes in Python code.

They look almost identical and behave completely differently in memory.

squares_list = [n * n for n in range(10_000_000)]    # builds all 10M now
squares_gen  = (n * n for n in range(10_000_000))    # builds nothing yet

import sys
sys.getsizeof(squares_list)   # ~89,000,000 bytes
sys.getsizeof(squares_gen)    # ~200 bytes - just the generator object

The comprehension allocates every element before the next line runs. The generator produces them one at a time as they are consumed, so memory is constant regardless of the size of the input.

That makes the generator the right default for a pipeline you are going to consume once:

# Reads a file of any size in constant memory: nothing is materialised.
with open("access.log") as f:
    errors = (line for line in f if " 500 " in line)
    first_ten = list(itertools.islice(errors, 10))

The trade-off is that a generator is single-use and has no length. Iterating it twice yields nothing the second time, and len() raises. So use a list when you need to iterate more than once, index into it, or know its size — and a generator when you are streaming, particularly when the source is larger than memory or you may stop early.

Because a list is a contiguous array, so inserting at the front shifts every existing element one position along, making it linear in the length of the list. Doing it in a loop is quadratic, which is why building a list by repeatedly prepending is a classic accidental performance bug. collections.deque gives constant-time appends and pops at both ends and is the correct structure for a queue; appending to a list and reversing once at the end is another fix.

A slice of a list produces a new list containing the same element objects, so it is a shallow copy: the outer container is new, the contents are shared. Slicing a string or a tuple gives a new immutable object, which is why s[:] on a string may return the original. The other thing worth knowing is that slicing never raises for out-of-range indices — items[5:10] on a three-element list returns an empty list rather than an error, which hides some off-by-one bugs.

list.sort sorts in place and returns None; sorted returns a new list and accepts any iterable. The None return is deliberate signalling rather than an oversight — it tells you the effect was on the argument, and it stops x = y.sort() silently producing None. Both are stable, meaning equal elements keep their relative order, which is what lets you sort by one key and then another to get a compound ordering.

self is the instance, passed as the first argument to every method. It is explicit because Python resolves attributes at run time and wants the lookup to be visible: self.x is unambiguously an instance attribute, where an implicit this would make it unclear whether a bare name was local or a field. The practical consequence is that a method is just a function whose first parameter happens to be an instance, which is why you can call MyClass.method(obj) directly.

The order in which Python searches base classes for an attribute, computed by the C3 linearisation algorithm and available as Cls.__mro__. It guarantees that a class appears before its own bases and that the order of bases is preserved, which makes multiple inheritance predictable. It is also why super() does not simply mean "my parent" — it means the next class in the MRO of the actual instance, which in a diamond hierarchy may be a sibling rather than a parent.

In a diamond, super() follows the MRO of the instance being constructed, so a class can be handed control from a sibling it knows nothing about.

class Base:
    def __init__(self): print("Base")

class Left(Base):
    def __init__(self): print("Left");  super().__init__()

class Right(Base):
    def __init__(self): print("Right"); super().__init__()

class Both(Left, Right):
    def __init__(self): print("Both");  super().__init__()

Both()
# Both
# Left
# Right     <- Left's super() reached Right, not Base
# Base

Left.__init__ calls super().__init__() and reaches Right, even though Left does not inherit from Right and its author may never have heard of it. That is the MRO doing its job — it visits each class exactly once.

print([c.__name__ for c in Both.__mro__])
# ['Both', 'Left', 'Right', 'Base', 'object']

Two consequences follow. Calling the base explicitly, as Base.__init__(self), breaks the chain and skips Right entirely, which is why cooperative inheritance requires everyone to use super(). And every class in such a hierarchy must accept and forward the arguments it does not understand, usually via **kwargs, or an initialiser somewhere down the chain will be called incorrectly.

The convention that an object's suitability is determined by the methods it implements rather than by its declared type. An object is iterable because it implements __iter__, not because it inherits from an interface. This is why Python code tends to be written against behaviour and tested with simple fakes, and why isinstance checks against concrete classes are often a design smell — they refuse objects that would have worked perfectly.

A method with a name surrounded by double underscores, which the interpreter calls in response to syntax rather than you calling it directly. len(x) calls x.__len__(), a + b calls a.__add__(b), with x: calls __enter__ and __exit__, and x in y calls __contains__. They are the mechanism behind duck typing: implementing the right dunders makes your object work with built-in syntax and standard library functions without inheriting from anything, which is why a class defining __iter__ can be used in a for loop and passed to sorted or list with no declaration of intent anywhere.

A context manager guarantees cleanup on every exit path, including exceptions, which is what with is for.

class Timer:
    def __enter__(self):
        self.start = time.perf_counter()
        return self                       # this is what `as t` binds

    def __exit__(self, exc_type, exc, tb):
        self.elapsed = time.perf_counter() - self.start
        # Returning True would SWALLOW the exception. Return None or False
        # unless suppressing is genuinely what you mean.
        return False

with Timer() as t:
    do_work()
print(t.elapsed)

The generator form is shorter and covers most cases:

from contextlib import contextmanager

@contextmanager
def timer():
    start = time.perf_counter()
    try:
        yield                             # everything before is __enter__
    finally:                              # everything after is __exit__
        print(f"{time.perf_counter() - start:.3f}s")

The try/finally is not optional in the generator form. Without it, an exception inside the with block propagates out of the yield and the cleanup code never runs — which is exactly the failure the context manager existed to prevent, now hidden behind an abstraction that looks correct.

The __exit__ return value is the other trap. Returning a truthy value suppresses the exception, so a stray return True turns a context manager into a silent except: pass for everything inside it.

A property makes attribute access run a method, so obj.total can compute rather than store while callers still write it as an attribute. It is right when a value is genuinely derived, or when you need to add validation to an existing attribute without changing every caller. It is wrong when the work is expensive or has side effects, because callers reasonably assume attribute access is cheap and repeatable — a property that issues a database query is a performance bug waiting for a loop.

A class attribute lives on the class and is shared by every instance; an instance attribute lives on the object. Reading falls back from instance to class, so a class attribute looks like a default. The trap is that assigning through an instance creates an instance attribute and shadows the class one, while mutating a shared mutable class attribute changes it for everyone — a class-level items = [] is the Python equivalent of the mutable default argument.

An iterable can produce an iterator, via __iter__; an iterator produces values one at a time, via __next__, and raises StopIteration when exhausted. A list is iterable and not an iterator, which is why you can loop over it repeatedly. A generator is both, and is exhausted after one pass — which is the source of the common bug where a second loop over the same generator silently does nothing.

It suspends the function, returning a value to the caller and preserving all local state, so the next call resumes exactly where it left off. That makes a generator function a way to write an iterator without managing state by hand. The practical consequence is laziness: nothing in the body runs until the first value is requested, so a generator that opens a file does not open it at the point you call the function, which surprises people relying on that side effect.

Two functions that look equivalent behave completely differently at scale.

def load_all(path):
    rows = []
    with open(path) as f:
        for line in f:
            rows.append(parse(line))       # entire file in memory
    return rows

def stream(path):
    with open(path) as f:
        for line in f:
            yield parse(line)              # one row at a time

For a two-gigabyte file the first needs two gigabytes and produces its first usable row only after parsing all of it; the second uses constant memory and yields a row immediately.

Laziness also makes early exit free:

# Stops reading the file as soon as it finds a match. The eager version
# parses every line first, then throws almost all of it away.
first_error = next((r for r in stream(path) if r.status >= 500), None)

There is a real cost to know. The with block inside the generator stays open until the generator is exhausted or closed, so an abandoned generator holds the file handle until garbage collection runs. And the generator can only be consumed once — a second loop produces nothing, with no error, which is the single most common laziness bug. Where both matter, materialise deliberately with list() and accept the memory.

Composable, lazy building blocks that avoid materialising intermediate results: islice for taking a slice of a stream, chain for concatenating without copying, groupby for runs of adjacent equal keys, tee for consuming one iterator twice. The value is that they compose into a pipeline that stays constant-memory. The one to be careful with is groupby, which only groups adjacent items, so the input almost always has to be sorted by the same key first.

:= assigns within an expression, so a value can be computed, tested and reused without a separate statement. Its best use is a loop or comprehension where you would otherwise call something twice: if (n := len(data)) > 10 uses n afterwards without recomputing. It is easy to overuse — a comprehension with two walruses is harder to read than the loop it replaced — so the test is whether it removes a duplicated call rather than whether it saves a line.

It stays suspended, holding its local state and anything it has open, until it is garbage collected or explicitly closed. On collection, Python throws GeneratorExit into it, which runs any finally block. That is what closes a file opened inside a generator, and it is why abandoning generators in a loop can hold many file handles open for an unpredictable time under a non-reference counting implementation.

The global interpreter lock in CPython allows only one thread to execute Python bytecode at a time, so several threads cannot speed up CPU-bound pure-Python work. It does not prevent concurrency in general: the lock is released while a thread waits on a socket, a file or a subprocess, so threads help substantially for I/O. It also does not apply across processes, and libraries such as NumPy release it while running compiled code. A candidate who says "Python cannot do parallelism" has learned a slogan.

The decision is almost entirely about whether the work waits or computes.

flowchart TD
    A[Is the work waiting or computing] --> B[Waiting on IO]
    A --> C[Computing]
    B --> D{Are the libraries async}
    D -- Yes --> E[asyncio, thousands of concurrent waits]
    D -- No --> F[Thread pool, blocking calls are fine]
    C --> G{Is it array or matrix work}
    G -- Yes --> H[NumPy or a GPU library, releases the lock]
    G -- No --> I[Processes, one interpreter each]

The branch people get wrong is the one under computing. Reaching for threads to speed up a pure-Python calculation gives you no speed-up at all and adds locking bugs, because the interpreter lock serialises the bytecode regardless of how many threads you start.

# Four threads, four cores, and no faster than one thread.
with ThreadPoolExecutor(4) as ex:
    list(ex.map(count_primes, ranges))       # GIL-bound

# Four processes: genuinely parallel, at the cost of pickling the arguments
# and results across a pipe.
with ProcessPoolExecutor(4) as ex:
    list(ex.map(count_primes, ranges))       # real parallelism

The cost of processes is the part to state: each has its own interpreter and memory, arguments and return values are pickled, and start-up is measured in tens of milliseconds. That makes them wrong for many small tasks and right for a few large ones. On platforms where the default start method is spawn, the child re-imports your module, so anything at module scope runs again — which is why process-based code needs an if __name__ == "__main__" guard.

Because the loop runs on a single thread and cooperatively switches only at an await. A synchronous call that takes 200 milliseconds — a blocking database driver, requests, a CPU-heavy computation — stalls every other task in the process for that whole time, including the health check. The fix is either an async-native library or pushing the blocking call to a thread with asyncio.to_thread, which keeps the loop free while it waits.

await suspends until that one operation completes, so awaiting in a loop is sequential no matter how asynchronous the code looks.

# Sequential: 3 x 200ms = 600ms. Each await finishes before the next starts.
async def fetch_all_slow(urls):
    results = []
    for url in urls:
        results.append(await fetch(url))
    return results

# Concurrent: ~200ms. All three start, then we wait for all of them.
async def fetch_all(urls):
    return await asyncio.gather(*(fetch(url) for url in urls))

The distinction is that gather schedules every coroutine before awaiting any, so the waits overlap. This is one of the most common review findings in async Python, and the tell is an await inside a for.

sequenceDiagram
    participant L as Event loop
    participant A as fetch one
    participant B as fetch two
    participant C as fetch three
    L->>A: start all three, then await
    L->>B: start
    L->>C: start
    A-->>L: done after 200ms
    B-->>L: done after 200ms
    C-->>L: done after 200ms

Look at the first three lines running before any reply arrives. In the sequential version those would be interleaved with their responses, so the diagram would be three times as tall.

Two refinements matter in production. gather cancels nothing by default when one task fails, so with return_exceptions=False you get the first exception while the others keep running unattended. And unbounded concurrency is its own problem — a thousand simultaneous requests to one host will be refused — so a semaphore usually belongs in the middle:

sem = asyncio.Semaphore(10)

async def limited(url):
    async with sem:                      # at most ten in flight
        return await fetch(url)

Modern code often prefers asyncio.TaskGroup, which cancels remaining tasks when one fails and waits for them properly, making the failure behaviour explicit rather than something you have to remember.

Calling an async def function returns a coroutine object and runs nothing; awaiting it runs it in the current task. Wrapping it in a task with asyncio.create_task schedules it on the loop immediately, so it runs concurrently with whatever you do next. Forgetting the difference produces the bug where creating coroutines in a loop and awaiting them afterwards is still sequential, and the other where a created task is never awaited and its exception disappears.

Because the event loop keeps only a weak reference, so a task with no strong reference anywhere can be garbage collected mid-execution and silently disappear. The symptom is background work that usually completes and occasionally does not, with no error. The convention is to store tasks in a set and discard them in a done callback, or to use a TaskGroup, which owns its tasks for you and is the reason it is now the recommended shape.

A build of CPython that runs without the global interpreter lock, so pure-Python threads achieve genuine parallelism. It arrived as an experimental option in 3.13 and became officially supported in 3.14 under PEP 779. It is still not the default build, it still carries a single-threaded performance cost, and the ecosystem is still catching up, since C extensions written against the lock's implicit guarantees need auditing. Worth knowing about as direction of travel, and worth being clear that it is a build of CPython rather than a change to the language.

Primarily by reference counting: every object tracks how many references point at it, and it is freed the moment that reaches zero. That makes reclamation immediate and predictable, which is why a file closed by falling out of scope usually closes right away. A separate cycle-detecting collector handles groups of objects that reference each other and would otherwise never reach zero. This is a CPython implementation detail rather than a language rule.

Guessing at Python performance is unusually unreliable, because the expensive line is rarely the one that looks expensive.

# Cumulative time, so a slow callee shows up under its caller.
python -m cProfile -s cumtime app.py | head -20

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.001    0.001   12.480   12.480 app.py:41(build_report)
     8400    0.052    0.000   11.900    0.001 db.py:88(fetch_one)     <- 8400 calls
        1    0.310    0.310    0.480    0.480 report.py:12(render)

The ncalls column is usually the finding rather than the timing. Eight thousand four hundred calls to fetch_one from a function that should query once is an N+1, and no amount of optimising the function itself will help — the fix is to call it once with a batch.

For memory the equivalent is tracemalloc, which attributes allocations to lines:

import tracemalloc
tracemalloc.start()
run_workload()
for stat in tracemalloc.take_snapshot().statistics("lineno")[:5]:
    print(stat)

The habit that matters more than either tool is measuring before changing anything. The most common wasted optimisation in Python is rewriting a loop that accounted for two per cent of the runtime, while the program spends ninety per cent waiting on a database — which the profile would have said in thirty seconds.

Because every operation goes through the interpreter: each iteration dispatches bytecode, checks types at run time, and allocates objects for intermediate values. A comparable C loop does none of that. The practical consequence is not to micro-optimise the loop but to move the work somewhere compiled — a vectorised NumPy operation, a built-in such as sum or any, or a library function — because the win is in leaving the interpreter, not in tightening the Python.

The syntax gives almost no clue which side of the interpreter boundary you are on, and the performance difference is the whole subject.

import numpy as np

values = np.random.rand(10_000_000)

# Interpreted: 10 million bytecode dispatches, 10 million float objects.
total = 0.0
for v in values:
    total += v * 2

# Compiled: one call into C, operating over contiguous memory.
total = (values * 2).sum()

The second is typically two orders of magnitude faster, and it is not because NumPy is a faster Python — it is because no Python runs per element. The loop you wrote executes in C over a densely packed array of machine doubles, while the first version boxes every value into a Python float object.

The same structure repeats one level up, which is the more useful generalisation:

your code            what actually executes
-----------------    -------------------------------------------
pure Python loop     the interpreter, one bytecode at a time
NumPy / pandas       compiled C over contiguous memory
PySpark              JVM executors on a cluster
PyTorch              GPU kernels

In every row Python is the control plane rather than the engine. That is why df.apply(some_function) is usually a disappointment — it runs a Python function per row, so it re-enters the interpreter and gives up the very thing you moved to pandas for.

They solve different problems and compose, so the question is a little unfair until you separate them. A dataclass generates the boilerplate — __init__, __repr__, __eq__ and optionally ordering — from the annotated fields, and by default still stores attributes in a per-instance dictionary like any other object. __slots__ changes the storage rather than the behaviour, replacing that dictionary with a fixed array, which cuts memory substantially and forbids adding undeclared attributes. Since 3.10 you can have both with @dataclass(slots=True), which is the right default for a value-like object created in large numbers — and the wrong choice where code relies on setting attributes dynamically.

Because CPython manages small objects in its own pools and arenas, and returns memory to the operating system only when a whole arena becomes free — which fragmentation often prevents. So a peak allocation can leave the process resident size permanently higher even though the objects are gone. The practical implication is that a worker that once processed a huge batch may stay large, which is why long-running workers are often recycled after a number of requests.

Something reachable that you forgot about, since the collector frees only what is unreachable. In practice: a module-level dict or list used as a cache with no eviction, an accumulating log or metrics buffer nobody drains, a closure holding a large object it only needed one field from, and objects kept alive by an exception's traceback stored for later. None of these is a leak in the allocator; all of them are references you still hold.

stateDiagram-v2
    [*] --> Referenced
    Referenced --> Unreferenced: last name goes out of scope
    Unreferenced --> Freed: refcount hits zero
    Referenced --> InCycle: objects reference each other
    InCycle --> Freed: cycle collector runs
    Referenced --> Retained: a cache or module global still holds it
    Freed --> [*]

The transition that costs money is the last one into Retained. Nothing is broken and the collector is behaving correctly — the object is genuinely reachable, so it is genuinely alive, and the bug is in your reference graph rather than in the runtime.

Because Python's default is to install into a shared location, so two projects needing different versions of the same library conflict, and upgrading a package for one can break system tooling. A virtual environment gives a project its own interpreter and site-packages directory, so its dependencies are isolated. The fact that this is a convention layered on top rather than a language feature is why the tooling landscape has so many competing solutions.

More Python hours are lost to environment confusion than to any language feature, and three commands resolve nearly all of it.

python -c "import sys; print(sys.executable)"    # which interpreter is running
python -c "import sys; print(sys.path)"          # where it looks for imports
python -m pip list                               # what is installed for THAT one

The critical detail is python -m pip rather than a bare pip. A bare pip is whatever executable is first on your PATH, which may belong to a different interpreter entirely — so you install a package with one Python and import it with another, and get ModuleNotFoundError for something you can plainly see was installed.

# The symptom, and why it happens.
pip install requests            # installs into /usr/bin/python3
python app.py                   # runs .venv/bin/python -> ModuleNotFoundError

Running the module form binds the installation to the interpreter you named, which removes the whole class of problem. The same applies to python -m pytest over bare pytest, which additionally puts the current directory on sys.path and fixes a large share of "my tests cannot import my package" reports.

A requirements file usually states constraints, so two installs weeks apart can resolve to different versions of a transitive dependency and produce different behaviour. A lockfile records the exact resolved version and hash of every package in the tree, so an install is reproducible and verifiable. That distinction matters most in CI and in a container image, where "it worked yesterday" and "the same code is running" need to be the same statement.

No. The interpreter records them and does not check them, so a function annotated to take an int will happily receive a string and fail somewhere downstream. Enforcement comes from a separate checker such as mypy or Pyright, run in the pipeline. Some libraries do read annotations and act on them — FastAPI derives request validation from them, and Pydantic coerces to them — but that is those libraries choosing to, not the language.

Annotating a parameter is table stakes. Making an invalid state impossible to represent is the part that earns the checker's cost.

from typing import Literal
from dataclasses import dataclass

# Two independent booleans allow four states, and two of them are nonsense.
@dataclass
class BadResult:
    is_loading: bool
    is_error: bool          # loading AND error is representable

# A tagged union allows exactly three, and the checker enforces the fields
# that belong to each.
@dataclass
class Loading:
    status: Literal["loading"] = "loading"

@dataclass
class Failed:
    status: Literal["failed"]
    message: str

@dataclass
class Loaded:
    status: Literal["loaded"]
    rows: list[str]

Result = Loading | Failed | Loaded

The payoff is exhaustiveness at the point of use:

def render(result: Result) -> str:
    match result:
        case Loading():                return "..."
        # Keyword patterns, not positional: __match_args__ follows field order,
        # so `Failed(message)` would bind the *status* field to `message`.
        case Failed(message=message):  return f"error: {message}"
        case Loaded(rows=rows):        return f"{len(rows)} rows"
    # mypy reports any missing case here, so adding a fourth variant
    # breaks the check rather than falling through at run time.

The same discipline applies at the process boundary. Annotations describe what you believe about data you have not seen, so anything arriving over a network or from a file needs validating at run time — with Pydantic or similar — and the static type should be derived from that validator rather than declared separately, so the two cannot drift apart.

A fixture is requested by name in a test's signature, so dependencies are explicit and only the tests that need something pay for it. It also has a scope — function, class, module or session — which controls how often it is constructed, and it can yield, so teardown lives next to setup. A shared setup method runs for every test in the class whether or not they use it, and hides what each test actually depends on.

Mock at the boundary you do not own — a network call, a clock, a payment provider — and let your own code run. Mocking your own internals produces tests that assert how the code is written rather than what it does, so they fail on every refactor and pass while the behaviour is broken. The specific pitfall in Python is patching the wrong name: you must patch where the object is used, not where it is defined, because the importing module holds its own reference.

Because the module form adds the current directory to sys.path, while the bare executable does not. That single difference resolves most "my tests cannot import my package" problems on a project without an installed package. The more durable fix is to install the project itself in editable mode, so the package is importable the same way in tests as in production, rather than depending on which directory the runner was invoked from.