Low-latency and quantitative development fundamentals

The word is useless without two endpoints, and naming them is the first thing a competent interviewer listens for. Tick-to-trade means from the first bit of a market-data packet arriving at your network interface to the first bit of the resulting order leaving it. Wire-to-wire usually means the same thing measured outside the box by a tap, which is the only measurement nobody can argue with. Internal latency means from the point your code sees the message to the point it hands an order to the transport, which excludes everything you do not control. Candidates who quote a number without saying which of these they measured have told you nothing, because the three differ by more than any optimisation you are likely to make.

Because the cost of being late is not proportional to how late you were. Past some threshold the price has moved or another participant has taken the opportunity, and being twice as late costs you nothing extra because you had already lost. That shape makes an average meaningless: a path that is quick on almost every message and occasionally very slow loses money on exactly the messages that mattered, and its mean looks excellent. So the working statistics are high percentiles and the maximum, and the engineering goal is usually to narrow the distribution rather than to shift its centre. A change that makes the median slightly worse and the ninety-ninth percentile much better is a change you take.

Jitter is the variation in latency rather than its magnitude, and it earns a separate word because its causes are different. Magnitude comes from work you chose to do: parsing, lookups, arithmetic. Jitter comes from things that happen to you: an interrupt landing on your core, the scheduler putting another thread on it, a page fault, a frequency change, a cache line arriving from another socket rather than from L1. Almost every technique in this discipline - core isolation, preallocation, kernel bypass, avoiding the general-purpose allocator - is aimed at jitter rather than at the mean, because jitter is what sets the tail and the tail is what costs money.

Treating the path as a ledger of contributors, rather than as one number, is what makes an optimisation argument possible.

stage                          who controls it     what makes it vary
-----------------------------  ------------------  ---------------------------------
wire to NIC                    nobody              switch queuing upstream
NIC to your code               you, partly         interrupt vs polled receive
decode the message             you                 branchy parser, cache misses
book update                    you                 data structure, allocation
strategy decision              you                 lookups, floating-point work
order encode                   you                 string building, allocation
your code to wire              you, partly         syscall vs bypass, TCP vs UDP
NIC to venue                   nobody              distance, cross-connect, queuing

Read the third column first. The rows where the variance is yours are the rows worth work, and they are not always the rows with the largest mean contribution. A parser that is quick but allocates on one message in a thousand can own your tail while contributing almost nothing to your average.

The rows marked "nobody" set a floor you cannot optimise past in software, and knowing that floor stops a team spending a quarter on a path whose remaining budget is smaller than the switch's contribution. This is also the honest answer to "how much faster can you make this" - you subtract the fixed terms first.

The other use of the ledger is attribution during an incident. If you have a timestamp at each boundary, a widened tail localises to one row in minutes rather than becoming a week of argument. Systems built without those boundaries end up debugging by bisection over commits, which is far slower and often inconclusive.

Finally, the ledger tells you when to stop. Once no row you control is larger than the noise in your measurement, further work on that path is unmeasurable, and the next win has to come from removing a row entirely rather than shrinking one.

Batching is the clearest example. Handling messages in groups amortises fixed costs and improves throughput, and it necessarily delays the first message in each group, which harms latency. The same tension runs through interrupt coalescing on a network interface, through buffering in a transport, and through compression. A throughput-oriented design happily adds a queue anywhere, because a queue absorbs bursts; a latency-oriented design treats every queue as suspicious, because a queue is where waiting happens. Neither is more correct in general, and the mistake to avoid is importing a default from one world into the other - most infrastructure defaults are throughput defaults.

That the sequence of work done per message is the same every time, so the time it takes has little room to vary. Concretely: no allocation, no locks that could be contended, no branches whose outcome depends on data in a way that changes the work done, no I/O, no logging that formats a string, no lazily initialised state. Determinism in this sense is not about producing the same output - that is a separate and also useful property for replay testing - it is about doing the same amount of work. It matters because a path whose work varies has a tail set by its worst case, and you rarely get to choose which message hits the worst case.

It is the measurement error where a load generator that stalls stops issuing requests during exactly the period it should be recording as slow. If your harness sends a request, waits for the reply, and then sends the next, a long delay means fewer samples are taken while the system is unwell, so the slow period is under-represented and the percentiles look far better than reality. The fix is to drive load at a fixed rate independent of responses, and to record the delay from the moment a request was due to be sent rather than from when it actually went. Almost every homemade benchmark has this bug, and it always flatters the system.