Cost and latency of a task

Tokens per turn, the cache, output speed and human waits: the full arithmetic of a task, with a live calculator driven by the engine.

Loading the animation…

Concept

Everything in the earlier chapters ends up in two numbers: what a task costs, and how long it takes. Both are sums over the model calls, and the calculator above computes them the way the engine does: pick a task and a policy, and the engine runs it (in your browser, in a background thread) and takes every call apart.

Per call, the cost has three parts: the prompt tokens not in the cache, at the input price (or the cache-write price, if the prompt is being cached); the prompt tokens read from the cache, at about a tenth of that; and the output tokens, at several times the input price. The time has two: the time to the first token, which grows with the uncached prompt, and the decode time, which grows with the output. Around the calls come the tools, the human's answers and any back-off.

A few things the calculator shows:

The model's price sets the scale. Fixing the test costs $0.016 at Claude Sonnet 4.6's list price, $0.00533 at Haiku 4.5's and $0.00148 at GPT-5 mini's, for the same tokens.
Small prompts are never cached. The fix-the-test prompts stay under the 1,024-token minimum a prompt needs before it is cached, so turning the cache off changes nothing there ($0.016 either way). Reading the notes, 71% of the prompt tokens come from the cache at Sonnet's prices, but none at Haiku's, whose minimum is 4,096 tokens.
Input dominates. Over the note-reading task the model is sent 16,179 tokens and writes 396 tokens: re-reading the context, not writing, is where the tokens go.
People and slow hardware dominate the time. The same fix takes 7.71 s against a hosted API with nothing to approve, 27.6 s when a person approves each shell command and edit, and 36.2 s with the small local model on an old CPU, at no per-token price.

Maths

For model call $t$ with $I_t$ prompt tokens, $K_t$ of them read from the cache, and $O_t$ output tokens,

C = \sum_t \big[(I_t - K_t)\,p_{\text{in}} + K_t\,p_{\text{cache}} + O_t\,p_{\text{out}}\big],

where $p_{\text{in}}$ becomes the cache-write price $p_{\text{write}}$ on a call whose prompt is written to the cache. Each call's time is a fixed overhead $t_0$ , plus the uncached prompt at the prefill rate, plus the output at the decode rate; a run adds the tools, the waits for the human and any back-off:

T = \sum_t \Big[t_0 + \frac{I_t - K_t}{r_{\text{prefill}}} + \frac{O_t}{r_{\text{decode}}}\Big] + \sum t_{\text{tool}} + \sum w .

Because the whole context is re-sent on every turn (chapter 1), $\sum_t I_t$ grows roughly with the square of the number of turns, while $\sum_t O_t$ grows linearly: long tasks are input-bound, which is why caching (chapter 4), compaction (chapter 3) and sub-agents (chapter 6) all go after $I_t$ . Where $r_{\text{prefill}}$ and $r_{\text{decode}}$ come from (the roofline of the hardware serving the model, batching, and the prefill/decode split) is the subject of LLM Inference Explained; how a serving system trades them against each other, including prefix caching, of Inference Trade-offs Explained.

Code

One call's cost and latency, cut from the vendored engine (accounting.ts):

const fresh = input - cached;
const rate = stored ? price.cache_write : price.input;
return (cached * price.cache_read + fresh * rate + output * price.output) / 1e6;

const ttft = profile.overhead_ms + ((input - cached) * 1000) / profile.prefill_tps;
return [ttft, ttft + (output * 1000) / profile.decode_tps];