A worker picks up the job you queued and cannot find the row it was told about. What went wrong?

Two connections, one uncommitted row

Inside transaction.atomic() the writes you make are on your connection and nowhere else. Under the read-committed isolation that PostgreSQL and MySQL use by default, no other session can see them until you commit. A Celery task, a webhook delivery, or a cache warm-up triggered from inside that block is a different process on a different connection, and it may well execute before your commit reaches the database. It then queries for a primary key that exists in your session's imagination and raises DoesNotExist.

The symptom is diagnostic in itself: it fails intermittently, more often under load, and never on a developer machine where the worker is slower to pick up than the web process is to commit. Retries appear to fix it, which is how the bug survives for a year.

There is a worse case that candidates who have only met the first one miss. If the transaction rolls back after the job was enqueued — a later IntegrityError, a validation failure, a 500 — the job still exists. It now describes an order that was never created. You have fired an email, charged a card, or published an event for a state the database never reached, and no retry policy can undo that.

on_commit moves the side effect outside the boundary

transaction.on_commit registers a zero-argument callable to run after the outermost atomic block commits successfully. If the transaction rolls back, the callbacks are discarded. If there is no transaction in progress, the callback runs immediately, which is what makes the same code correct with and without ATOMIC_REQUESTS.

from django.db import transaction


def create_order_bad(payload):
    with transaction.atomic():
        order = Order.objects.create(**payload)
        send_confirmation.delay(order.id)     # racing our own commit


def create_order(payload):
    with transaction.atomic():
        order = Order.objects.create(**payload)
        # Closes over order; runs only if this block commits, and after it does.
        transaction.on_commit(lambda: send_confirmation.delay(order.id))
    return order

Two properties of the callbacks are worth stating because they are the follow-up. They run synchronously, in the same thread, after the commit and outside any transaction — so a slow callback is latency on the request, and a callback that needs the database opens a fresh transaction. And an exception raised in a callback cannot roll anything back, because the commit already happened; it propagates to the caller, leaving earlier callbacks executed and later ones not. So on_commit gives you "at most once after commit", which is why the durable version of this pattern is a transactional outbox: write the intent as a row in the same transaction and let a separate process publish it.

Nested atomic is a savepoint, and only the outer one commits

atomic blocks nest, but the inner ones do not begin transactions. They issue SAVEPOINT, and on exception they roll back to that savepoint while the outer transaction survives. This is the mechanism behind two behaviours that otherwise look arbitrary.

The first is that on_commit registered inside an inner block fires when the outermost block commits, not when the inner block exits. A helper that thinks it owns a transaction does not. atomic(durable=True), added in Django 3.2, is the assertion for when that matters: it raises if the block turns out to be nested, so a function that must be the commit boundary can say so instead of hoping.

The second is what happens when you catch a database error. Once the database has rejected a statement, the transaction is in an aborted state and every further query on it fails. Django tracks this and raises TransactionManagementError on the next query rather than letting you continue into confusion. So except IntegrityError placed inside an atomic block that has no savepoint below it leaves you holding a transaction you cannot use. The fix is to put the risky statement in its own inner atomic, which gives the rollback a savepoint to return to.

from django.db import IntegrityError, transaction


def claim(code):
    try:
        with transaction.atomic():        # inner block: a savepoint to roll back to
            return Coupon.objects.create(code=code)
    except IntegrityError:
        return Coupon.objects.get(code=code)   # the outer transaction is still usable

That shape is also the correct answer to a concurrent get_or_create: let the unique constraint arbitrate and handle the loser, rather than checking for existence first and hoping nobody interleaves. Where you genuinely need to serialise, select_for_update takes row locks — and it raises TransactionManagementError outside a transaction rather than silently locking nothing, because a lock released immediately would be worse than an error. Lock rows in a consistent order across your codebase, or two transactions taking the same two locks in opposite orders will deadlock.

Where the transaction begins is a design decision

ATOMIC_REQUESTS wraps every request in a transaction, which makes the common case safe by default and has a real cost: the transaction stays open through view logic and template rendering, so any HTTP call, image resize or third-party SDK invocation in that path holds a database connection and its locks for the duration. On a connection pool sized for fast queries, that is how a slow external dependency becomes a database outage. Explicit atomic blocks scoped tightly around the writes keep the transaction short, at the cost of having to remember them.

The FastAPI and SQLAlchemy version of this question has identical shape under different names. A Session that has flushed but not committed has written nothing other connections can see, so publishing from inside the unit of work has the same race, and the same two answers apply: commit first and then publish, or attach the publish to SQLAlchemy's after_commit session event so the ordering is structural rather than remembered.

Your test suite is hiding it

Django's TestCase wraps each test in a transaction and rolls it back for speed, which means it never commits, which means on_commit callbacks never run. A test written before the fix will pass, and a test written after it will pass without exercising the callback at all. Use TestCase.captureOnCommitCallbacks(execute=True), available since Django 3.2, to run them and assert on the effect, or fall back to TransactionTestCase when you need genuine commits and can afford the table truncation between tests.

The visibility boundary is the commit, not the write, so anything outside your connection must be triggered after it. on_commit is the mechanism, a savepoint is what an inner atomic block actually gives you, and a suite that rolls back every test cannot tell you whether you got either right.