Low-latency and quantitative development fundamentals

The word is useless without two endpoints, and naming them is the first thing a competent interviewer listens for. Tick-to-trade means from the first bit of a market-data packet arriving at your network interface to the first bit of the resulting order leaving it. Wire-to-wire usually means the same thing measured outside the box by a tap, which is the only measurement nobody can argue with. Internal latency means from the point your code sees the message to the point it hands an order to the transport, which excludes everything you do not control. Candidates who quote a number without saying which of these they measured have told you nothing, because the three differ by more than any optimisation you are likely to make.

Because the cost of being late is not proportional to how late you were. Past some threshold the price has moved or another participant has taken the opportunity, and being twice as late costs you nothing extra because you had already lost. That shape makes an average meaningless: a path that is quick on almost every message and occasionally very slow loses money on exactly the messages that mattered, and its mean looks excellent. So the working statistics are high percentiles and the maximum, and the engineering goal is usually to narrow the distribution rather than to shift its centre. A change that makes the median slightly worse and the ninety-ninth percentile much better is a change you take.

Jitter is the variation in latency rather than its magnitude, and it earns a separate word because its causes are different. Magnitude comes from work you chose to do: parsing, lookups, arithmetic. Jitter comes from things that happen to you: an interrupt landing on your core, the scheduler putting another thread on it, a page fault, a frequency change, a cache line arriving from another socket rather than from L1. Almost every technique in this discipline - core isolation, preallocation, kernel bypass, avoiding the general-purpose allocator - is aimed at jitter rather than at the mean, because jitter is what sets the tail and the tail is what costs money.

Treating the path as a ledger of contributors, rather than as one number, is what makes an optimisation argument possible.

stage                          who controls it     what makes it vary
-----------------------------  ------------------  ---------------------------------
wire to NIC                    nobody              switch queuing upstream
NIC to your code               you, partly         interrupt vs polled receive
decode the message             you                 branchy parser, cache misses
book update                    you                 data structure, allocation
strategy decision              you                 lookups, floating-point work
order encode                   you                 string building, allocation
your code to wire              you, partly         syscall vs bypass, TCP vs UDP
NIC to venue                   nobody              distance, cross-connect, queuing

Read the third column first. The rows where the variance is yours are the rows worth work, and they are not always the rows with the largest mean contribution. A parser that is quick but allocates on one message in a thousand can own your tail while contributing almost nothing to your average.

The rows marked "nobody" set a floor you cannot optimise past in software, and knowing that floor stops a team spending a quarter on a path whose remaining budget is smaller than the switch's contribution. This is also the honest answer to "how much faster can you make this" - you subtract the fixed terms first.

The other use of the ledger is attribution during an incident. If you have a timestamp at each boundary, a widened tail localises to one row in minutes rather than becoming a week of argument. Systems built without those boundaries end up debugging by bisection over commits, which is far slower and often inconclusive.

Finally, the ledger tells you when to stop. Once no row you control is larger than the noise in your measurement, further work on that path is unmeasurable, and the next win has to come from removing a row entirely rather than shrinking one.

Batching is the clearest example. Handling messages in groups amortises fixed costs and improves throughput, and it necessarily delays the first message in each group, which harms latency. The same tension runs through interrupt coalescing on a network interface, through buffering in a transport, and through compression. A throughput-oriented design happily adds a queue anywhere, because a queue absorbs bursts; a latency-oriented design treats every queue as suspicious, because a queue is where waiting happens. Neither is more correct in general, and the mistake to avoid is importing a default from one world into the other - most infrastructure defaults are throughput defaults.

That the sequence of work done per message is the same every time, so the time it takes has little room to vary. Concretely: no allocation, no locks that could be contended, no branches whose outcome depends on data in a way that changes the work done, no I/O, no logging that formats a string, no lazily initialised state. Determinism in this sense is not about producing the same output - that is a separate and also useful property for replay testing - it is about doing the same amount of work. It matters because a path whose work varies has a tail set by its worst case, and you rarely get to choose which message hits the worst case.

It is the measurement error where a load generator that stalls stops issuing requests during exactly the period it should be recording as slow. If your harness sends a request, waits for the reply, and then sends the next, a long delay means fewer samples are taken while the system is unwell, so the slow period is under-represented and the percentiles look far better than reality. The fix is to drive load at a fixed rate independent of responses, and to record the delay from the moment a request was due to be sent rather than from when it actually went. Almost every homemade benchmark has this bug, and it always flatters the system.

The ordering is the reliable knowledge here, and the ordering is enough to reason with.

access                                relative cost      why it matters here
------------------------------------  -----------------  ---------------------------
register                              free               keep hot state in a few vars
L1 data cache hit                     cheapest real cost the target for hot working set
L2 hit                                several times L1   still fine, noticeable
L3 or another core's cache            much worse         coherence traffic lives here
main memory                           worse again        a miss you did not plan for
remote socket memory                  worse still        why you pin to one socket
page fault into the kernel            far worse          never on the hot path
syscall with a scheduler wakeup       far worse          why bypass exists

Treat the second column as an ordering, not as measurements - the actual figures depend on the part, the clock, and what else is running, and a candidate who recites specific nanosecond counts as fact is usually reciting a decade-old table.

What the ordering buys you is triage. If your hot working set does not fit in L1 you can predict the effect of shrinking a structure, and that predicts which of two layouts wins before you write either. If a profile shows time in a memory access, the question becomes which level it is being served from, and hardware counters answer that.

The two largest steps are the ones people forget, because they are not memory accesses at all. Crossing into the kernel dwarfs a cache miss, so a single stray syscall on the hot path outweighs a great deal of careful layout work. That is why the first pass on a latency problem is to find the syscalls and the allocations, and only then to argue about cache lines.

Because caches and the coherence protocol both operate on fixed-size lines - conventionally 64 bytes on common x86-64 parts - not on your variables. A read pulls in the whole line, which is why fields you access together should sit together: you get the neighbours for free. A write requires the line in an exclusive state, which invalidates every other core's copy of the whole line, including the parts holding variables you never touched. Both the benefit and the hazard follow from the same fact, and a layout decision has to consider both: pack what one thread reads together, separate what different threads write.

Two threads writing to different variables that happen to occupy the same cache line. Each write forces the line into that core's exclusive state and invalidates the other's copy, so the line ping-pongs between caches on every operation and two independent threads serialise through hardware. Nothing in the source looks wrong - there is no lock, no shared variable, no data race - which is why it is diagnosed late. The symptom is that adding a thread makes the program slower, and the confirmation is a hardware counter for cache-line transfers or stores hitting a line held modified elsewhere. The fix is to give each hot field its own line.

Padding without alignment is the mistake, so the fix has two halves.

// Wrong: same line, so the two threads' writes invalidate each other.
struct Counters {
  std::atomic<uint64_t> in;
  std::atomic<uint64_t> out;
};

// Right: alignas fixes where an instance starts, padding fixes its size.
struct alignas(64) Counter {
  std::atomic<uint64_t> v{0};
  char pad[64 - sizeof(std::atomic<uint64_t>)];
};

// And assert it, so a later field addition fails the build rather than the tail.
static_assert(sizeof(Counter) == 64);
static_assert(alignof(Counter) == 64);

Size and alignment have to agree. A struct padded to 64 bytes but allowed to start anywhere will straddle a line boundary for some instances in an array, so half your elements still share.

The static assertions are the part that reads as production experience. The common story is not the original bug but its return: somebody adds a field above the padded region, or changes how instances are allocated, and the layout regresses silently. Nothing fails, the tail just widens, and the change gets blamed on load.

The trade is footprint. Expanding an 8-byte counter to 64 multiplies an array of them eightfold, and a per-thread table that fitted in L1 may now spill. So this belongs where you measured the coherence traffic, not applied everywhere by policy.

Because you only pay for lines you touch. If a pass over ten thousand orders reads one 4-byte field of a 200-byte record, an array of structs brings in a whole line per record to use a few bytes of it, so the effective bandwidth is dreadful and the prefetcher cannot help much. Splitting that field into its own contiguous array packs sixteen values into every line and turns the scan into a stream the hardware predicts perfectly. The cost is that operating on one whole record now touches several arrays, so the layout is chosen per access pattern - and mixed workloads sometimes want both, with one view maintained as a projection.

It watches your access pattern and fetches lines ahead of the demand, which is why a linear scan over a contiguous array is dramatically faster than the same number of accesses in random order even though the instruction count is identical. It handles forward and backward sequential streams and simple constant strides. It cannot follow a pointer chase, because the next address is not known until the current line arrives, which is the real reason a linked list or a node-based tree performs badly on a hot path regardless of its asymptotic complexity. When your data structure is a graph of pointers you are choosing to pay a dependent miss per hop.

Every memory access needs a virtual-to-physical translation, and the translation lookaside buffer caches a limited number of page mappings. With standard small pages, a working set of a few hundred megabytes needs far more mappings than the buffer holds, so accesses spread over it miss the buffer and pay an extra walk through the page tables before the data access even begins. Huge pages cover far more memory per entry, so the same working set fits in the buffer and the walks disappear. The costs are that huge pages need configuration and often privilege, they can waste memory through internal fragmentation, and transparent variants can introduce their own stalls.

Because mapping is lazy. A large allocation typically returns addresses the kernel has promised but not backed with physical pages, and the first write to each page traps into the kernel to have one assigned. So a process that allocates its arenas at startup and first touches them once trading has begun has not removed those faults, it has relocated them into the phase where they hurt most. The discipline is to write to every page during startup, before the latency-sensitive phase, and where the platform allows it to lock the pages so the kernel cannot reclaim them and hand you the fault back later.

The whole idea is that allocation becomes a pointer bump and release becomes a single reset.

class Arena {
public:
  Arena(std::byte* base, std::size_t bytes)
      : base_(base), cur_(base), end_(base + bytes) {}

  void* allocate(std::size_t n, std::size_t align) {
    auto p = reinterpret_cast<std::uintptr_t>(cur_);
    auto a = (p + align - 1) & ~(align - 1);
    auto next = reinterpret_cast<std::byte*>(a) + n;
    if (next > end_) return nullptr;   // exhaustion is a design decision
    cur_ = next;
    return reinterpret_cast<void*>(a);
  }

  void reset() { cur_ = base_; }       // frees the whole phase at once

private:
  std::byte* base_; std::byte* cur_; std::byte* end_;
};

What it removes is variance rather than cycles. A general-purpose allocator is fast on its fast path, and its problem is that you cannot tell at the call site whether you are about to get the thread-local cache, a shared structure under contention, newly mapped memory with faults attached, or a housekeeping pass. Those outcomes differ by orders of magnitude and the expensive ones are rare, which is another way of saying they are your tail.

The nullptr on exhaustion is the interesting line. Falling back to the system allocator when the arena fills would reintroduce exactly the variance you built it to remove, and it would do so only under load. The honest designs size the arena so exhaustion means a bug, or degrade in a defined way such as refusing new work.

What it costs is lifetime discipline. There is no per-object free, so anything living past the reset cannot live here, and nothing in the language stops you storing such a pointer. For lifetimes that are not phase-shaped, a fixed-size object pool with a free list is the right structure instead, and most systems run both.

Progress, not speed. Lock-free means that at least one thread makes progress in a bounded number of steps regardless of what other threads do, which is a statement about the absence of a stall when a thread is descheduled mid-operation - the thing a mutex cannot promise. It does not mean fast, and a compare-and-swap loop under heavy contention can be considerably slower than an uncontended mutex because every failed attempt is wasted coherence traffic. Wait-free is the stronger property, where every thread completes in a bounded number of steps. Candidates who use lock-free as a synonym for fast get caught by the follow-up about contention.

Because an uncontended lock acquisition is cheap, so throughput measurements barely notice it. The damage happens on the rare occasion when the lock is held. If the holder is descheduled while holding it, everyone waiting is stuck until the scheduler runs it again, which is orders of magnitude longer than the work being protected. If the wait escalates from spinning to a kernel wait, you have added a syscall and a wakeup to the path. Neither event happens often, and both land squarely in the percentile you are judged on, which is why the fix is usually to remove the sharing rather than to make the lock faster.

One producer, one consumer, no locks, and the correctness rests on two indices and two orderings.

template <class T, size_t N>          // N a power of two
class Spsc {
  alignas(64) std::atomic<size_t> head_{0};   // written by producer only
  alignas(64) std::atomic<size_t> tail_{0};   // written by consumer only
  alignas(64) std::array<T, N> buf_;

public:
  bool push(const T& v) {
    auto h = head_.load(std::memory_order_relaxed);
    if (h - tail_.load(std::memory_order_acquire) == N) return false;  // full
    buf_[h & (N - 1)] = v;
    head_.store(h + 1, std::memory_order_release);   // publishes the slot
    return true;
  }
};

The alignment on each index is not decoration. Producer and consumer write different indices, so if the two share a line every push invalidates the consumer's copy and the queue false-shares its way to serialisation - the textbook case, and the reason a hand-rolled queue often underperforms the mutex it replaced.

The release store on head_ is what makes the slot visible. It orders the write to buf_ before the index update, so a consumer that sees the new index with an acquire load is guaranteed to see the payload. Reverse those and you have a queue that works on strongly ordered hardware and fails on weaker models, which is a bug that survives every test you run on your laptop.

The power-of-two capacity lets the modulo be a mask, and using free-running counters rather than wrapped indices makes fullness a subtraction rather than a special case. Both are small, and both remove a branch from the hottest loop in the system.

A compare-and-swap succeeds because the value it compared matches, but the world changed and changed back in between, so the success is a lie. The classic case is a lock-free stack: you read the head node, another thread pops it, frees it, pushes a different node that the allocator places at the same address, and your compare-and-swap sees the expected pointer and installs a stale successor. The usual defences are a version counter packed alongside the pointer so the comparison covers both, or a reclamation scheme - hazard pointers or epoch-based reclamation - that guarantees a node is not reused while any thread might still be looking at it.

They order your other memory operations, which is the whole point. A relaxed atomic guarantees the operation itself is indivisible and says nothing about the surrounding writes, so a consumer can see a published index before it sees the data the index refers to. A release store guarantees everything written before it is visible to anyone who performs an acquire load that reads it, which is exactly the publish-then-read handshake a queue needs. Sequential consistency adds a single global order across all such operations and costs more, often a full fence. The rule that survives interviews: relaxed for counters nobody synchronises on, acquire and release for handing over ownership of data.

A convoy is the state where threads repeatedly hand a contended lock to each other and each acquisition involves a wakeup, so the group moves in lockstep at the speed of the scheduler rather than of the work. Throughput collapses even though no thread is doing anything expensive. Priority inversion is the neighbouring failure: a low-priority thread holds a lock a high-priority thread needs, and a middle-priority thread keeps the holder off the CPU, so the high-priority thread waits on work that is not running. Both are arguments for not sharing mutable state across threads on a latency path at all.

It is right when the expected wait is shorter than the cost of sleeping and waking, which on a latency path is nearly always, and when you have a core to spare so the spinning thread is not stealing time from the thread it waits for. It is a mistake when those conditions fail: on a shared or oversubscribed machine a spinner can prevent the holder running, which is the worst possible outcome, and it costs power and heat permanently. A spin loop should include the architecture's pause hint so the core does not fight itself on speculative reads, and any spin that could last should eventually fall back to a blocking wait.

Four, and naming them separately is the answer. The interrupt and its deferred processing, which interrupt whatever the core was doing and can evict your working set. The scheduler wakeup, since a blocked thread becoming runnable is not the same event as it being on-CPU. The copy from kernel memory into your buffer. And the syscall boundary itself, a mode switch that is more expensive than it once was on hardware with speculative-execution mitigations enabled. A bypass path maps the interface's receive rings into the process and has a thread read descriptors out of them, so none of the four happens on the data path.

A core, permanently, because a spinning receive loop consumes one whether traffic arrives or not, and you will isolate it so nothing else lands there. Your tooling, because traffic that never enters the kernel is invisible to ordinary packet capture and socket statistics, so you use whatever capture the bypass stack offers or you tap in hardware upstream. Portability, because the path is tied to particular interface families and driver versions, making a card refresh a software project. And safety, because a malformed packet the kernel would have dropped now reaches code you wrote, where a mistake corrupts memory instead of returning an error.

The branch is worth drawing because the interesting property is not speed, it is how many places control leaves your thread.

flowchart TD
  A[NIC receives packet] --> B{Which path}
  B -- Kernel stack --> C[Interrupt then protocol stack]
  C --> D[Scheduler wakes the thread]
  D --> E[Copy to user buffer via syscall]
  B -- Bypass --> F[Descriptor in mapped ring]
  F --> G[Spinning thread reads in place]
  E --> H[Strategy sees the price]
  G --> H

Every box on the left leg is a point where something else on the machine can insert delay you did not ask for. The right leg has none, which is why the win is mostly in the tail rather than the median.

That also tells you when bypass is the wrong tool. A bulk transfer amortises per-packet cost across large segments and benefits from offloads, so the kernel stack does very well; bypass earns its keep only where a single small message must be handled with a predictable delay.

The middle position is often the right first step: pin the threads, isolate the cores, disable frequency scaling, and enable the kernel's own busy-polling on the socket. That removes part of the wakeup cost while keeping every tool you own, and the improvement tells you how much of your problem was ever the kernel.

Because it exists to improve efficiency at the expense of exactly what you care about. It withholds a small outbound segment until either an acknowledgement for the previous one arrives or enough data accumulates to fill a segment, which reduces the number of tiny packets on the network and adds a wait of up to one round trip to a message you wanted sent immediately. Interacting with delayed acknowledgement on the far side it can produce worse pauses still. Any latency-sensitive TCP socket disables it, and the deeper lesson is that most transport defaults were chosen for throughput and have to be revisited one by one.

It lets a venue send one copy of an update that the network replicates to every subscriber, so the publisher's cost does not grow with the number of consumers and every consumer receives the same bytes at nearly the same time - fairness that a fan-out of unicast connections cannot provide. The cost is that it runs over UDP, so there is no retransmission and no ordering guarantee. You get sequence numbers and you own gap detection, gap fill from a recovery service or a secondary feed, and the decision about what to do while you have a hole. That recovery logic is where most real feed-handler complexity lives.

It occupies a full core continuously, spinning on a ring or a socket looking for new data, which costs power, heat and one fewer core for anything else on the box. You accept it because the alternative is a blocking wait, and a blocking wait means the scheduler decides when your thread runs again. That decision is outside your control and varies with everything else happening on the machine, which is precisely the jitter you are trying to remove. Spinning converts an unbounded, externally controlled delay into a bounded one you pay for in hardware, and hardware is the cheaper of the two.

Because a fixed pipeline in hardware can parse a feed, apply a simple check and emit an order without a general-purpose core, a cache hierarchy or an operating system anywhere in the path, which gives both a lower and a far tighter latency than software can. It suits work that is simple, stable and on the critical path: feed decoding, risk limit checks, a straightforward quoting response. It stops being worth it as soon as the logic changes often or gets complicated, because development, testing and deployment cycles are much slower than software and the engineers are scarce. The common architecture is a hardware fast path with software owning everything else.

Two sides, each an ordered collection of price levels, and each level a queue of resting orders in arrival order. Bids descend from the best price, offers ascend, and the best bid and best offer define the spread. The structure exists to answer two very different questions quickly: what is the best price on each side, which is a lookup at one end, and where does this incoming order belong, which is a positioned insertion. Most of the design work is in choosing a representation that makes the first question free and the second cheap, because the first is asked far more often than the second.

The access pattern, not the asymptotics, decides the representation.

operation              frequency    what it needs            typical choice
---------------------  -----------  -----------------------  --------------------------
read best bid or ask   constant     O(1) at both ends        cached top of book
add order at a level   very high    O(1) append to a queue   intrusive list per level
cancel a known order   very high    O(1) removal by handle   hash from id to node
execute at best        high         O(1) pop front           same list, front removal
find a price level     very high    O(1) by price            array indexed by ticks
walk levels outward    lower        ordered traversal        contiguous array of levels

Cancels dominate. Real feeds carry far more cancellations and modifications than executions, so a representation that makes cancel cheap and market-order matching slightly dearer is usually the right trade - which is why an order identifier maps directly to its node rather than being searched for.

The price-to-level lookup is the design decision people miss. Because tradeable prices form a discrete grid of ticks, a dense array indexed by tick offset turns level lookup into arithmetic, and its neighbours are contiguous so walking outward streams through cache. A tree gives you the same ordering with a pointer chase per hop and no prefetching, which is why the asymptotically respectable structure loses.

The cost of the array is memory and range. You size it for a plausible price window and need a plan for instruments that move outside it, usually a sparse overflow structure plus a rebase. That is a real complication, and it is the follow-up question.

That among resting orders the better price is filled first, and within one price level the earlier arrival is filled first. It makes each level a strict queue, so joining a level means going to the back and cancelling and replacing means losing your position - which is why queue position is something strategies actively value. The alternative rules exist and change the engineering: pro-rata allocation divides an incoming order across the level in proportion to size, which removes the incentive to be early and means a level is not a queue at all. You should know which rule a venue uses before designing anything that depends on fill probability.

Market-by-order publishes individual order events - add, cancel, modify, execute - each with an identifier, so a subscriber can rebuild the full book including the queue at every level and can track its own position in it. Market-by-price publishes aggregated levels: price, total size, sometimes order count. The first is far more data and far more processing but is the only way to reason about queue position or to infer anything about individual participants; the second is cheaper and sufficient for a strategy that only needs the shape of the book. The choice drives your feed handler's cost more than almost any other decision.

Sequence numbers detect the gap, and then you have three unattractive options.

sequenceDiagram
  participant F as Feed handler
  participant P as Primary multicast
  participant B as Secondary feed
  participant R as Recovery service
  P->>F: seq 100 then seq 102
  F->>F: gap detected at 101
  F->>B: check arbitrated copy
  B-->>F: seq 101 delivered
  F->>R: request snapshot if still missing
  R-->>F: snapshot plus resume point

The first line is the arbitrated secondary. Venues commonly publish two identical streams over separate paths, and a handler that takes whichever copy of each sequence number arrives first fills most gaps with no extra latency at all, because independent paths rarely lose the same packet.

Failing that, a targeted retransmission request costs a round trip, during which more updates arrive and must be buffered rather than applied. The state to be careful about is the one where you are buffering: your book is stale, and continuing to trade on it is worse than not trading.

The last resort is a full snapshot and replay from the resume point, which is slow enough that a strategy is usually stood down for the instrument. The engineering question an interviewer wants answered is not how you recover but what you do while recovering - and the correct answer is to stop quoting, because a book you know is wrong is a liability.

It is a fill where the same participant is on both sides. It is a problem because it creates trade reports and market data implying activity that never transferred any economic interest, it can be used to manipulate an apparent price, and it creates real settlement and reporting mess. Venues therefore implement self-match prevention, typically cancelling one or both of the resting and incoming orders, or reducing them. For a system builder the point is that the engine must carry participant identity into the matching decision, and that a firm running multiple strategies on one identity has to coordinate them so they do not trade with each other.

The loop is short, and every subtlety is in what happens when it exits.

incoming: BUY 500 @ 101 limit

opposing offers, best first:
  100.99  x 200   order A
  101.00  x 150   order B
  101.00  x 100   order C   (arrived after B)
  101.01  x 900   order D

step 1: 100.99 <= 101  -> fill 200 against A     remaining 300
step 2: 101.00 <= 101  -> fill 150 against B     remaining 150
step 3: 101.00 <= 101  -> fill 100 against C     remaining  50
step 4: 101.01 >  101  -> stop crossing
step 5: rest 50 @ 101 on the bid side, at the back of that level

Fills happen at the resting order's price, not the incoming order's, so the buyer pays 100.99 for the first 200 - the price improvement belongs to whoever was resting. Candidates who fill everything at the limit price have misunderstood who sets the price.

Step 3 shows time priority doing its job: B and C are at the same price and B arrived first, so B is filled completely before C is touched. Reversing that is the most common bug in a hand-written matcher.

Step 4 is the branch that decides everything after it. Because this is a limit order it rests; a market order with quantity remaining would be cancelled or rejected depending on venue rules, and an immediate-or-cancel order would discard the remaining 50. The order type is not a label, it is the exit condition of this loop.

Step 5 is where the new resting order enters at the back of level 101, which is why cancelling and replacing an order to change its size costs queue position and why strategies avoid doing so casually.

Because matching is inherently sequential - the outcome depends on the exact order of arrivals - and because a single thread with no locks and no shared mutable state is both faster and vastly easier to reason about than a parallel version. Fairness and auditability demand a definite sequence of events, and a definite sequence is what one thread gives you for free. Scaling comes from partitioning by instrument across threads or processes, since orders in one symbol do not interact with another, and from making the single thread's work small: preallocated memory, arrays rather than trees, no logging in line, and all persistence handled by publishing an event stream someone else consumes.

With a monotonic clock that does not enter the kernel on each call, which on modern systems a coarse-grained call already avoids, or with the CPU's cycle counter where you need finer resolution. The cycle counter's traps are worth knowing: it is not a wall clock, it needs calibration against a real clock, comparisons across cores are only safe where the counter is invariant and synchronised, and reading it can be reordered by the processor unless you constrain that. For anything crossing the network, the credible timestamps come from hardware - a capture device or the interface's own clock, disciplined to a time source - because software timestamps include the very costs you are measuring.

Shape, and shape is what tells you the cause.

latency bucket    count      cumulative
--------------    --------   ----------
  0 -  1 unit     812,004      81.2%
  1 -  2 units    170,110      98.3%
  2 -  4 units     15,900      99.9%
  4 -  8 units        400     99.94%
  8 - 16 units          6    99.9994%
 16 - 32 units        580     99.999%   <- second mode, not a long tail

The last row is the point. A summary reporting a p99 and a p999 would show numbers that look like a smoothly decaying tail, and the natural response is to shave work off the hot path. The histogram shows something else: a cluster far out with a higher count than the buckets before it, which is not a tail at all but a separate population.

A second mode almost always has a discrete cause - a periodic task, a timer, a log flush, a page fault, a frequency transition, a garbage collection if a managed language is anywhere in the path. You find it by correlating the timestamps of the slow samples against other events rather than by optimising code, and shaving cycles off the fast path would have moved the first two rows and left the problem untouched.

Recording full histograms also lets you merge across processes and time windows without the arithmetic being wrong, which averaging percentiles does not - the average of two p99s is not a p99 of anything.

Sampling interrupts the program periodically and records where it is, so its overhead is bounded and independent of what the code does, and it tells you where time is spent in aggregate. That makes it right for finding the expensive function and wrong for finding a rare event, because a slow path taken once in ten thousand messages will barely appear. Instrumenting records explicit timestamps at chosen boundaries, so it captures every occurrence including the rare one and can attribute it to a specific message, at the cost of adding work to the path you are measuring. Latency work needs the second; throughput work is usually served by the first.

By making the in-line part as close to nothing as possible and moving everything else out. On the path, read a counter and store a few bytes into a preallocated ring - no formatting, no allocation, no locks, no branches on configuration. Off the path, a separate thread or a separate process drains the ring, formats, aggregates and writes. The mistake is to format or aggregate in line: building a string or updating a shared histogram introduces both cost and variance, and the variance appears in the numbers you collected, so the measurement reports the measurement. Sizing the ring so a slow drainer drops samples rather than blocking the producer is part of the design.

By stamping the message rather than the stage, so that each hop records its own timestamp into a slot the message carries and the final consumer can subtract adjacent pairs.

flowchart TD
  A[Capture device stamps arrival] --> B[Feed handler stamps in and out]
  B --> C[Book builder stamps in and out]
  C --> D[Strategy stamps decision]
  D --> E[Gateway stamps send]
  E --> F[Capture device stamps egress]
  F --> G[Offline join by message id]

The two capture-device stamps at the ends are what make the whole thing trustworthy: they bracket the software with measurements taken outside it, so the sum of your internal stages can be checked against a number nobody in the process could have distorted.

The gap between the arrival stamp and the feed handler's first stamp is usually the most informative single figure on the page, because that is where receive-path cost and scheduling delay live, and it is the part teams most often forget to measure at all.

Instructions and cycles together, because their ratio tells you whether you are limited by work or by stalls - a low instructions-per-cycle figure on simple code means you are waiting for memory, not computing. Then cache misses by level, to find out which level is serving your data. Then branch mispredictions, for a hot loop with data-dependent control flow. Then the counters for cache lines transferred between cores or stores hitting a line held modified elsewhere, which are how you confirm false sharing rather than guessing at it. Then page faults and context switches, either of which on a hot path is a bug rather than a tuning opportunity.

By changing one thing, running both under identical conditions, and comparing distributions rather than single figures. Identical conditions means the same machine, the same cores, the same input recording replayed at the same rate, the same thermal state, and interleaved runs rather than all of A then all of B, because the machine drifts. Comparing distributions means high percentiles and the shape, since a change that improves the median and doubles the p999 is a regression. Replaying a recorded feed rather than live traffic is what makes the comparison repeatable at all, and building that replay harness is usually the first real work on a latency project.

It moves your thread to another core, which abandons a warm cache, and it puts other work on the core your thread wants, which delays it. The defences are to pin each latency-critical thread to a specific core, to remove those cores from the pool the kernel schedules general work onto, and to steer device interrupts away from them. On a machine with more than one socket, also make sure the thread, its memory and the network interface it uses are all on the same socket, because reaching across to another socket's memory or device is measurably worse and the placement is easy to get wrong by accident.

Because the processor lowers its clock when it thinks it is idle, and a latency-sensitive thread is idle most of the time by design - it is waiting for a message that arrives rarely. When one does arrive, the first work happens at a reduced clock and the transition to a higher one takes time, so the messages you care most about are handled in the worst state. Deep idle states make it worse, adding a wakeup delay before any work happens at all. The remedy is to hold the frequency and forbid deep idle states on the isolated cores, and to accept the power and heat cost that follows.

Almost nothing in cycles on the non-throwing path, which is the point of the zero-cost model, but it is not free in the ways that matter here. The metadata increases binary size, which competes for instruction cache. The possibility of a throw constrains what the optimiser will do and forces the compiler to keep cleanup paths reachable. And the cost when one is thrown is large and hard to bound, involving table lookups and stack unwinding. So exceptions are fine for startup and configuration failures and wrong as control flow on a hot path, where a returned status code keeps the cost visible.

Because the indirect call itself is usually predicted well when the target is stable, so the branch cost is modest. The real damage is elsewhere: the call is opaque to the optimiser, so nothing inlines across it and no cross-function optimisation happens, which for a small function is a large relative loss. If the target does vary between calls, you additionally pay a mispredicted indirect jump and pull in a different function's instructions, so the instruction cache thrashes. The usual replacement is to resolve the type once outside the loop, or to use a template or a tagged union so the call is direct.

Naive logging formats a string, takes a lock, and writes to a file, which means allocation, contention and a syscall on the path you are protecting - and the write occasionally blocks on the filesystem, which is where the worst outliers come from. Done properly the in-line part writes a small fixed record into a preallocated per-thread ring: a format identifier and the raw arguments, no text. A separate thread drains the ring and produces human-readable output, or the records are written binary and formatted entirely offline. The trade is that logs are no longer readable without a tool, which is a price worth paying.

Because its cost is a pointer chase per level, and a pointer chase is a dependent memory access the prefetcher cannot anticipate: the next address is unknown until the current line arrives. Nodes are also individually allocated, so they are scattered, each visit likely touches a cold line, and construction goes through the allocator. A contiguous structure with worse asymptotics frequently wins at the sizes that occur in practice because it streams and the hardware helps. The general rule is to prefer contiguous storage and arithmetic indexing, and to treat any per-element allocation on a hot path as a defect.

Keeping the candidate causes as a checklist, ordered by how often they turn out to be the answer, stops an investigation becoming a debate.

suspect                        how you confirm it              typical fix
-----------------------------  ------------------------------  ------------------------
allocation on the hot path     allocation hook in test build   arena or pool
a syscall you did not intend   trace syscalls per message      remove or move off path
thread migration               per-thread core over time       pin and isolate
interrupt on a hot core        interrupt counts per core       steer interrupts away
page fault                     fault counter during the phase  pre-touch and lock pages
false sharing                  line transfers between cores    align and pad, then assert
log flush blocking             timestamps of slow samples      binary ring, offline format
frequency or idle transition   frequency over time per core    hold clock, forbid deep idle
upstream queuing               capture device stamps           nothing in your software

The order matters more than the list. Allocation and stray syscalls account for a large share of real cases and are the cheapest to rule out, so they go first even though they are the least interesting.

Every row has a confirmation step, and that is the discipline being tested. Each of these has a counter, a trace or a hook that answers the question definitively, so an investigation is a sequence of checks with an end rather than a sequence of guesses.

The last row is the one teams reach too late. If the capture device's own stamps show the delay arriving at your interface, no amount of work inside your process will help, and continuing to look is the most expensive mistake on the list.

The first time through a code path everything is cold: instructions are not in the instruction cache, data pages may not be resident, branch predictors have no history, and translation entries are missing. So the first message of the day, or the first after a quiet period, is handled far more slowly than the thousandth - and in trading the first message after quiet is often the most valuable one. The countermeasure is to exercise the whole path deliberately before it matters, with synthetic messages that are discarded before reaching the venue, and to keep exercising it during quiet periods so the caches stay warm.

Whether your instincts about memory and allocation are automatic. The problem is often unremarkable - a ring buffer, a fixed-size hash map, an order book, a parser for a binary message - and the scoring is in the decisions you make without being asked: preallocating, avoiding a node-based container, keeping the hot fields together, not throwing, not logging in line, thinking about who else touches this memory. Candidates who write clean idiomatic code that allocates freely and then say they would optimise later have answered a different question, because the whole discipline is that the layout decisions are not retrofittable.

Endpoints, statistic, conditions, and only then a figure. Say what you measured between and how, say which percentile, say at what message rate and on what hardware, and say whether the measurement was internal or taken by a capture device outside the box. A bare number invites the interviewer to assume the least flattering interpretation and ask which of those things you left out. It is also entirely acceptable to say a figure is confidential and describe the method and the relative improvement instead - that answer scores better than a precise-sounding number you cannot defend under questioning.

Enough to reason, not enough to design chips. You are expected to know the cache hierarchy and that coherence works on lines, what a 64-byte line implies for layout, what the prefetcher can and cannot follow, why address translation has its own cache, what pinning and core isolation do, and that reaching another socket's memory is worse than your own. You are not expected to know pipeline depths, specific latencies of particular parts, or microarchitectural detail beyond that. The failure mode is confident recitation of specific nanosecond figures, which invites a question about how they were measured and rarely survives it.

Move semantics and where copies still happen, what the standard containers allocate and when, alignment and how to control it, the atomics library and the memory-ordering vocabulary, custom allocators or polymorphic memory resources, templates well enough to remove a virtual call, and enough of the object model to reason about layout. Also the negative knowledge: which conveniences are unacceptable on a hot path and why. Being fluent in modern idioms without knowing what they cost is the specific gap these rounds are built to find, and it is a common one among strong general C++ engineers.

As a measurement, a mechanism and a trade.

weak version    "I optimised the feed handler and made it 30% faster."

strong version  "p999 tick-to-trade was dominated by a second mode about
                 20x the median. Correlating slow samples showed they
                 coincided with allocator activity in the decoder. Replaced
                 the per-message buffers with a reset-per-message arena.
                 The second mode disappeared. Median moved slightly worse
                 because the arena is a larger working set. Cost: a fixed
                 capacity we now have to size, and a static assertion plus
                 an allocation hook in test builds to stop it regressing."

The strong version names the statistic before the fix, which shows the work started with a measurement rather than with a technique someone read about.

It gives the mechanism by which the fix works, so the interviewer can check the reasoning rather than take the result on trust. A result without a mechanism is indistinguishable from a coincidence, and interviewers in this field have seen plenty of improvements that were really a quieter afternoon.

It volunteers the cost, in both performance and maintenance. That is the sentence that separates an engineer from an enthusiast, because every technique here is a trade and a candidate who presents one as pure gain has either not measured it or is not telling you about the part that hurt.

A strong answer treats the ratio as evidence of a discrete cause and hunts for it, rather than reaching for the hot path. Something happening on roughly one message in a thousand is an event, so the first move is to capture the slow samples with their timestamps and correlate them against everything periodic: allocator activity, log flushes, timer-driven work, page faults, context switches and involuntary migrations, interrupt arrivals on the wrong core, frequency or idle-state transitions, a garbage collection if anything managed is in the path, and upstream queuing visible in the capture device's own stamps. Each of those has a counter or a trace you can check, so the investigation is finite. A weak answer starts optimising the code that runs on every message - shaving instructions, rearranging fields, arguing about a container - which by construction cannot explain a problem that only appears on a thousandth of them. The other weak move is to widen the alerting threshold and declare the tail acceptable, which is how a system arrives at a p999 nobody remembers agreeing to.