Agent Harnesses Explained

All four tracks

Shared definitions

The sixteen companion sites were written one at a time, and a few terms came out meaning slightly different things, or written in different symbols. Here each gets one definition and one formula, in one notation, with the chapters that use it. Where a site’s own wording or notation differs, the entry says how and why; the site’s chapter is still right in its own terms.

KV cache, and its bytes per token κ

While a decoder generates, each attention layer keeps the keys and values of every earlier token so they are not recomputed. Its size grows by κ bytes for every token of context, for every sequence in the batch.

L layers, n_kv key/value heads of dimension d_h, b bytes per value; the 2 is one key and one value.

Where it is used

Where a site differs: Accelerators writes k_KV for κ. Architectures counts values per token, before multiplying by b. MLA caches one compressed latent per layer instead of 2 · n_kv · d_h values, so its κ is (d_c + d_h^R) · L · b.

TTFT, TPOT and ITL

Time to first token: from a request's arrival to its first output token (queueing plus prefill). Time per output token: the mean gap between its later tokens. Inter-token latency keeps every gap, so a single stall shows in its tail.

a_r the arrival of request r, τ_{r,i} the time of its i-th output token, n_r its output tokens.

Where it is used

Where a site differs: The Agents sites see a hosted API from the client's side: time to first token is a fixed overhead plus the uncached prompt at a prefill rate, with no queueing (Harnesses chapters 4 and 10, Context chapter 9). That is the server's TTFT at low load.

Goodput (two quantities, one name)

In serving, the rate of requests that met both latency targets (TTFT and TPOT SLOs): throughput counts everything served, goodput only what was served well enough. In training, the share of a job's wall-clock time that advances training, after checkpoints, failures and restarts.

λ the arrival rate, T* and P* the TTFT and TPOT targets; 0.9 is the share of requests that must meet both.

Where it is used

Where a site differs: Serving counts the requests a second that met both SLOs at the load offered; Trade-offs reports the highest rate at which 90% do, per GPU (DistServe's capacity form, the formula above). Training's goodput is a different quantity under the same name; read it from the site's context.

Source: DistServe, arXiv:2401.09670

Pipeline bubble

With a model cut into p pipeline stages and a batch split into m micro-batches, stages sit idle while the pipeline fills and drains. The bubble is that idle time, given here as a share of the step.

p stages, m micro-batches, v model chunks per device (interleaving; v = 1 without it).

Where it is used

Where a site differs: Training quotes the ratio of idle time to useful work per device, (p − 1)/(v m), the Megatron-LM convention, which shows interleaving's gain directly. The share of the whole step is ratio / (1 + ratio): at p = 4, m = 4 the ratio is 0.75 and the fraction 3/7 ≈ 0.43. Systolic's bubble is the same idea one level down: a cycle in which a PE has no operand.

Source: GPipe, arXiv:1811.06965

Roofline, arithmetic intensity, ridge point

A kernel or step that does F operations while moving B bytes runs no faster than the peak rate, and no faster than the bandwidth times its intensity. Below the ridge it is memory-bound; above it, compute-bound.

π the peak rate (FLOP/s), β the bandwidth (bytes/s), F the operations, B the bytes moved, I the intensity (FLOP/byte), I* the ridge.

Where it is used

Where a site differs: Kernels writes P_peak and W for π and β. Racks writes the ridge as F/B with F the dense peak and B the bandwidth (where Serving's B is bytes moved). Systolic counts bandwidth in words per cycle, so its intensity is MACs per word. The model is the same throughout.

MFU, model FLOPs utilisation

The arithmetic the model needs per second, counted from the tokens actually processed, as a share of the hardware's peak. Recomputed activations do not count, so it compares systems fairly.

f the model FLOPs per token: N parameters, and 12 L H Q T is attention's share per token (L layers, H heads of dimension Q, sequence length T), as in PaLM.

Where it is used

Where a site differs: Counting recomputed activations gives hardware FLOPs utilisation, which is higher. Other utilisations in the track are different quantities: a systolic array's share of useful MAC slots (Systolic chapter 4), the busy share of concurrency slots (Orchestration chapter 8) and a link's busy fraction (Training chapter 13).

Source: PaLM, arXiv:2204.02311

INT range of a symmetric quantiser

A b-bit symmetric quantiser stores round(x / s), clipped to ±(2^(b−1) − 1): −127 to 127 for INT8, −7 to 7 for INT4. Two's complement's extra code, −2^(b−1), is left unused so the grid is symmetric about zero.

b bits, s the scale (absmax here), x a value, q its stored integer.

Where it is used

Where a site differs: The sites agree on the restricted range. Some kernels use the full range, −128 to 127 for INT8, which Numerics and Serving both mention.

Prompt cache: read multiplier ρ, write premium w, minimum length

A provider that keeps the processed prefix of recent prompts charges a cache read ρ times the input price and a cache write w times it, and caches only prefixes of at least L_min tokens. A prefix written once and read k times pays when k passes the bound.

p_i and p_o the input and output prices, N prompt tokens (at least L_min to be cached), a share h of them read from the cache, M output tokens.

Where it is used

Where a site differs: ρ means other things elsewhere in the track: a within-task correlation (Evals chapters 1, 3, 5 and 8), a load λ/μ (Orchestration chapter 8) or a link's busy fraction (Training chapter 13), a compression ratio (Serving chapter 12) and air density (Racks chapter 7). On this page ρ is always the cache read multiplier. Trade-offs' prefix caching is the same reuse seen by the server, in prefill time saved rather than price.

Context rot

A model's use of its context gets worse as the context grows, and worse again when the part that matters sits in the middle. None of the sites measures it on a real model; each that needs it uses a labelled stand-in.

x the answer's relative position in the prompt; n_i call i's prompt tokens, p_0 its accuracy up to 4,000 tokens, β the loss per token beyond.

Where it is used

Where a site differs: The stand-ins differ on purpose and each says so: a U in position (Context, after Liu et al.) and a linear fall in length (Orchestration). The Harnesses chapters assume a model reads its whole context perfectly. The shape for your model and task is something to measure.

Source: Liu et al., Lost in the Middle, arXiv:2307.03172; Chroma, Context Rot (2025)

Latency percentiles (p50, p95, p99)

The latency below which a given share of requests fall: p50 the typical request, p95 and p99 the tail users remember. Two estimators are in use in the track.

x_(1) ≤ … ≤ x_(n) the sorted latencies (nearest rank counts from 1, linear from 0), q the share (0.95 for p95).

Where it is used

Where a site differs: Serving, Trade-offs and Racks interpolate (numpy's default); Orchestration takes the nearest rank, so its p95 is always an observed latency. Over hundreds of requests the two differ by less than the gap between neighbouring observations.

Little's law

In any stable system, the average number of items inside equals the arrival rate times the average time each spends inside. It needs no assumption about arrivals or service times.

L the mean number in the system, λ the arrival rate, W the mean time in the system.

Where it is used

pass@k and pass^k

pass@k: of k tries at a task, does some one pass? pass^k: do all k pass? Both are estimated without bias from n ≥ k samples of which c pass.

n samples of a task, c of them passing, averaged over tasks.

Where it is used