TL;DR — p99.9 dispatch latency went from 2,646ns to 825ns by making the price ladder re-anchor lazily instead of on every best-price move. The median didn't change. The tail did. The tail is the job.
The symptom
The medians were beautiful: 153ns dispatch, stable across runs. But the p99.9 column refused to sit still — 2,646ns, a 17× cliff over the median. In a latency-sensitive engine, a rare 2.6µs stall is not a rounding error; it's the number that matters.
The hunt
Slice the tape by message type and histogram each slice separately. The spikes correlated with best-price updates: the tick-indexed ladder kept its price window anchored around the touch, and every time the touch moved, the window re-based — a full re-anchor sitting directly on the hot path, firing rarely enough to hide from averages and often enough to own the tail.
The fix
Re-anchor lazily. The window gets hysteresis: it only re-bases when price actually escapes the anchored range, and the move is amortized instead of paid all at once. The common case — touch moves, window still valid — collapsed to a branch and a store.
The receipt
| metric | before | after |
|---|---|---|
| median dispatch | 153ns | 153ns |
| p99 | 458ns | ~430ns |
| p99.9 | 2,646ns | 825ns |
1,821 nanoseconds returned. Nothing about the happy path changed, which is exactly the point: tail engineering is about finding the thing you do rarely and making it either cheap or never.
Interactive version of this hunt: scroll chapter 0x01 on the front page.