Why do low-latency systems preallocate arenas instead of calling the general-purpose allocator on the hot path?

Short answer

A general-purpose allocator is adaptive, so its cost varies - cache hits, contention on a shared structure, page faults on new memory, occasional housekeeping - and the rare expensive path is your tail.

Keep memory arena explicit in the answer because that is the concept the interviewer is actually trying to test. A good memory arena explanation names the trade-off, the failure mode, and the evidence you would use before choosing. Use memory arena once more at the decision point so the answer reads as judgement rather than a detached example.

Keep memory arena explicit in the answer because that is the concept the interviewer is actually trying to test. A good memory arena explanation names the trade-off, the failure mode, and the evidence you would use before choosing.

What a general-purpose allocator is optimised for

malloc and operator new are engineered to be good across workloads nobody described in advance: many sizes, unpredictable lifetimes, many threads, long-running processes that must not fragment to death. That is a hard problem, and the implementations that solve it well do so by being adaptive. They keep per-size free lists, per-thread caches, and a slow path that acquires memory from the operating system and occasionally reorganises what it already holds.

Adaptive means data-dependent, and data-dependent means variable. A request satisfied from a thread-local cache of the right size class is cheap. A request that misses that cache and has to consult a shared structure is more expensive and may contend with another thread. A request that exhausts the allocator's current supply causes new memory to be mapped, and the first write to each newly mapped page traps into the kernel as a page fault. Occasionally the allocator will do housekeeping - returning memory, coalescing free blocks - and that work lands on whichever unlucky caller triggered it.

Every one of those is fine. None of them is slow in a sense a throughput benchmark would object to. The problem is that you cannot tell, at the call site, which one you are about to get.

Why that variance is the whole objection

This is where a mediocre answer and a strong one diverge. The mediocre answer says the allocator is slow, and the interviewer will ask how slow, and the candidate will either invent a number or stall. The strong answer says the allocator's distribution is wrong for the use, which is a claim you can defend without a stopwatch.

Order the terms rather than quoting them. A pointer bump in already-touched memory is cheaper than a free-list lookup, which is cheaper than an uncontended lock acquisition, which is cheaper than a contended one, which is far cheaper than a page fault that enters the kernel. The fast paths of a good allocator sit at the cheap end. The rare paths sit at the expensive end. If the hot path allocates once per market-data message, then some small fraction of messages take the expensive path, and that fraction is the ninety-ninth percentile you are being judged on.

A trading system's economics make this asymmetry sharp. The cost of being late is not proportional to how late you were; past some threshold the opportunity is simply gone, and being twice as late costs no more than being barely late. That shape is why the mean is close to useless here and the tail is the number that matters, and it is the reason a technique that makes the average slightly worse but the maximum far tighter is a good trade.

The arena, and what it actually does

An arena is a large block of memory obtained once, before the latency-sensitive phase begins, from which allocation is a pointer increment and deallocation does not exist per object. You reset the whole thing at a natural boundary.

class Arena {
public:
  explicit Arena(std::byte* base, std::size_t bytes)
      : base_(base), cur_(base), end_(base + bytes) {}

  void* allocate(std::size_t n, std::size_t align) {
    // Round the cursor up to the requested alignment, then bump it.
    auto p = reinterpret_cast<std::uintptr_t>(cur_);
    auto aligned = (p + align - 1) & ~(align - 1);
    auto next = reinterpret_cast<std::byte*>(aligned) + n;
    if (next > end_) return nullptr;   // exhaustion is a design decision, not an exception
    cur_ = next;
    return reinterpret_cast<void*>(aligned);
  }

  void reset() { cur_ = base_; }       // frees everything at once, at a phase boundary

private:
  std::byte* base_{};
  std::byte* cur_{};
  std::byte* end_{};
};

The return of nullptr on exhaustion is the line worth discussing. An arena has a fixed size chosen in advance, so running out is not an exceptional condition to be papered over with a fallback to the system allocator - that fallback would reintroduce exactly the variance you built the arena to remove, and it would do so only under load, which is when you least want a surprise. The honest designs either size the arena so exhaustion means a bug, and treat hitting it as a fatal condition detected in testing, or degrade in a defined way such as refusing new work.

Note also that the constructor takes memory it did not obtain. That separation matters, because obtaining and pre-touching the memory is a startup activity with its own rules, discussed below.

Allocation succeeding is not the same as memory being resident

The subtlety candidates miss most often: mapping memory does not populate it. A large allocation at startup typically returns a range of addresses the kernel has promised but not backed with physical pages, and the first write to each page faults. If your arena is only touched for the first time once trading has begun, you have moved the page faults, not removed them - and you have moved them into the hot phase, which is worse than leaving them in the allocator.

The discipline is therefore to write to every page of the arena during startup, before the phase you care about, so that by the time the first message arrives the mapping is fully populated. Where the platform supports locking pages into memory and using larger page sizes, both help - locking prevents the kernel reclaiming pages you will need again, and larger pages reduce the number of address-translation entries the hot working set needs, which reduces translation misses. Both require privilege and configuration, so they are an operational commitment, not a code change.

What the arena costs you

Flexibility, and it is not a small loss. There is no per-object free, so any object whose lifetime does not end at the reset boundary cannot live in the arena. That constraint has to be enforced socially and structurally: a type allocated from a per-message arena must not be stored in a container that outlives the message, and nothing prevents you writing that bug. The usual defences are to make arena-allocated types non-copyable out of the region, to keep the arena's lifetime obviously narrower than anything that could hold a reference, and to review for escapes as a specific category rather than trusting general ownership rules.

Capacity planning becomes your problem. The allocator used to absorb variation in demand; now you must decide the maximum in advance and either be right or fail predictably. That is a genuine engineering cost, and it is why arenas fit phase-shaped work - handle a message, reset - better than long-lived heterogeneous object graphs.

For lifetimes that are not phase-shaped, the right structure is usually a pool of fixed-size objects with a free list, which restores per-object release while keeping allocation to a handful of predictable operations. A system typically has both: pools for orders and positions that live for varying periods, arenas for the transient working memory of processing one event.

The regression nobody notices

An allocation-free hot path stays allocation-free only as long as someone is checking. A refactor introduces a std::string in a log call, a container grows past its reserved capacity, an exception path builds a message, and the path allocates again on a fraction of calls. Nothing breaks; the tail widens.

The countermeasure is mechanical: in test and staging builds, install an allocation hook that aborts or records when called between two markers around the hot path, and run the normal test suite through it. That converts a silent latency regression into a failing test, which is the only form of this discipline that survives a team of more than a few people.

An arena is not faster than the allocator on average - it is a decision to move every source of variability out of the phase where variability costs you, and to pay for that with a fixed capacity you must size and lifetimes you must police yourself.

---

© 2026 Preptima. Originally published at preptima.com.