A test passes on your machine and fails in CI roughly once a week. How do you track it down?

Reproduce first, and resist the rerun

The instinct is to guess at the cause and change something. Do not, because a once-a-week failure gives you no feedback loop: any change will appear to work for six days. The first job is to get a command that fails reliably.

Start from the CI artefacts. Note the full ordering of the run, because if the failure depends on which tests preceded it, that is already the answer. If you run with pytest-randomly the seed is printed in the header and can be replayed with --randomly-seed=, which turns an ordering-dependent failure into a one-line reproduction. Without it, --lf reruns the last failures and -p no:randomly removes shuffling from the variables. Then run the failing test entirely alone. A test that passes alone and fails in company is leaking state, which is a different investigation from a test that fails alone one time in fifty.

The other early decision is what to do about the red build in the meantime. Marking the test skipped or wrapping it in a rerun plugin is legitimate containment, and it is only legitimate with a ticket attached and a date. A rerun plugin applied as standing policy converts a signal into noise: the suite stops telling you about race conditions, and the ones it stops telling you about include the real ones.

Time is the commonest cause and the easiest to remove

Wall-clock dependence produces exactly this failure profile, because most of the time the clock is unremarkable. A test that constructs a date from today() and asserts on a month boundary fails on the last day of the month. A test asserting a 30-day expiry crosses a daylight-saving transition twice a year. A test asserting that an elapsed duration is under a threshold fails when the CI runner is busy.

The fix is not a wider tolerance, it is to stop the code under test reading the clock for itself. Pass in a callable, or a clock object, and let the test supply a fixed value. Libraries such as freezegun and time-machine exist because patching datetime.datetime.now directly is awkward — datetime is a C type whose attributes cannot simply be reassigned — and they are the right answer for legacy code you cannot restructure. For code you own, injection is cheaper and it documents the dependency.

import random
from datetime import datetime, timedelta, timezone

import pytest


class Trial:
    # The clock and the RNG are dependencies, declared like any other.
    def __init__(self, *, now=lambda: datetime.now(timezone.utc), rng=random):
        self._now = now
        self._rng = rng

    def expires_at(self):
        return self._now() + timedelta(days=30)

    def in_treatment_group(self):
        return self._rng.random() < 0.5


@pytest.fixture
def frozen_clock():
    fixed = datetime(2026, 3, 29, 0, 30, tzinfo=timezone.utc)
    return lambda: fixed


def test_expiry_is_exact(frozen_clock):
    assert Trial(now=frozen_clock).expires_at() == datetime(
        2026, 4, 28, 0, 30, tzinfo=timezone.utc
    )


def test_group_assignment_is_reproducible():
    # A private Random instance: the module-level RNG is left untouched.
    assert (
        Trial(rng=random.Random(1234)).in_treatment_group()
        == Trial(rng=random.Random(1234)).in_treatment_group()
    )

Two related details are worth volunteering. Keep datetimes timezone-aware, because comparing a naive one to an aware one raises TypeError in some code paths and silently compares as local time in others; datetime.utcnow() was deprecated in Python 3.12 precisely because it returns a naive value that everyone treats as aware, and datetime.now(timezone.utc) replaces it. And measure durations with time.monotonic, not time.time, so an NTP correction mid-test cannot produce a negative elapsed time.

Global state that nobody declared

The random module is a single shared generator. A test that calls random.random() advances it for every test that runs afterwards, so an assertion that depends on the sequence is coupled to the ordering of the whole suite. Seeding it in a fixture treats the symptom; giving each consumer its own random.Random instance removes the coupling, as above.

The subtler one, and a good answer to have ready, is hash-dependent iteration. String and bytes hashing is randomised per process by default, so set iteration order differs between runs — and a test that builds a list from a set, or asserts on the first element of one, fails at whatever rate the ordering happens to be wrong. PYTHONHASHSEED set to a fixed value in CI hides this rather than fixing it; sort the output or assert on a set instead. Note that dictionaries are not a source here: they have preserved insertion order since Python 3.7, so dict iteration is deterministic.

Anything shared between workers, and anything real

A test that opens a real socket has adopted every failure mode of the network and of somebody else's service. The useful move is to make that impossible rather than unlikely, so that an accidental network call is a hard failure on every run instead of a rare one.

import socket

import pytest


@pytest.fixture(autouse=True)
def no_network(monkeypatch):
    def deny(*args, **kwargs):
        raise RuntimeError("a test tried to open a socket")

    monkeypatch.setattr(socket.socket, "connect", deny)

The same reasoning applies to every resource the suite does not exclusively own. A shared development database, a fixed port, a fixed temporary filename, or a fixed row id will collide the moment the suite runs under pytest-xdist or two branches build at once. The remedy is to key the resource on the worker: tmp_path for files, a database name derived from the PYTEST_XDIST_WORKER environment variable, and port zero so the operating system allocates a free one.

Concurrency inside the test is the last family. If a test starts a thread or a task and then sleeps before asserting, it is racing, and the sleep is only the length of somebody's guess. Wait on the actual event — a threading.Event, a future, an await on the task — so the test either completes or times out for a reason you can read.

When the flakiness is the finding

The judgement an interviewer is listening for is that a flaky test is evidence, and sometimes the evidence points past the test. If the failure comes from two requests racing on a row, from an operation that is not idempotent on retry, or from code that reads the system clock in the middle of a calculation, the test is doing its job and production has the bug. Deleting or quarantining it in that case is not stabilising the suite, it is removing the one thing that noticed. Decide which kind you have before you decide what to do, and say so in the ticket.