Failure and recovery

Tools fail: retries with back-off, errors fed back to the model, and loop detection, with the success rate against the retry budget from seeded runs.

Loading the animation…

Concept

Tools fail. A network call times out, a test runner crashes, a file is locked. A harness decides what happens next, at three levels:

Retry the same call, after a wait that grows each time (exponential back-off: here 500 ms, then 1 s, then 2 s), up to a retry budget. The model never sees a failure that a retry fixed.
When the budget is spent, feed the error back to the model as the tool's result, and let it decide. The scripted model here does what models often do: it tries the same call again.
Detect loops: three identical calls in a row are not progress. The harness either stops the run or nudges the model with a message telling it to try something else.

Seed 0, with no retries, shows the failure: the shell fails twice in a row, the model asks for the same pytest a third time, and loop detection stops the run after 5.27 s, with the bug unfixed. One retry is not enough for this seed either. With two, the retries absorb every failure and the run finishes in 14.2 s after 3 retries: slower than a lucky run, because waiting is the price of a retry.

Now set the budget to 0 and choose nudge. The run "finishes": the nudge moves the model on, it edits the file, the last test run fails again, and the model reports that all the tests pass. The run's status is done; the tests never passed. A harness that counts "the model said it was done" as success will report this run as a success. Counting only runs whose last test run passed tells the truth, and that is what the chart below does.

Success against the retry budget (100 seeded runs per bar)

finished
finished with the tests passing

On a loop: stop the run

Retries	Finished / with passing tests	Mean time
0 retries	79% 79%	8.15 s
1 retry	97% 97%	8.95 s
2 retries	100% 100%	9.19 s
3 retries	100% 100%	9.3 s
4 retries	100% 100%	9.4 s

On a loop: nudge the model

Retries	Finished / with passing tests	Mean time
0 retries	100% 90%	9.07 s
1 retry	100% 100%	9.12 s
2 retries	100% 100%	9.19 s
3 retries	100% 100%	9.3 s
4 retries	100% 100%	9.4 s

Right-hand column: the mean simulated run time. Every run is the scripted fix-the-test task with the shell failing 30% of attempts; the failure rate and the timings are illustrative.

Over 100 seeded runs per budget, with the shell failing 30% of attempts: with no retries, 79% of runs finish; one retry lifts that to 97%, and two to 100%. With nudges instead of stops, 100% of the no-retry runs finish, but only 90% with their tests passing. Past two retries, a bigger budget buys nothing here and costs time: the mean run takes 9.19 s with two retries and 9.4 s with four.

Maths

If each attempt fails independently with probability $p$ , a call with a budget of $r$ retries fails only when all $r+1$ attempts do:

P_{\text{call}} = 1 - p^{\,r+1},

so with $p$ = 0.3, one call succeeds with probability 0.7, 0.91 and 0.973 for 0, 1 and 2 retries. A run is a chain of calls: here the model repeats a failed test run, and loop detection allows two identical calls in a row, so a test step is lost only when two calls in a row fail, with probability $q^2$ where $q = p^{\,r+1}$ . The task has two test steps:

P_{\text{run}} = (1 - q^2)^2 ,

which predicts 83%, 98% and 100%; the 100 seeded runs measured 79%, 97% and 100%, within sampling error (about ±4 points at 100 runs). The price is time: the waits before retry $k$ are $b f^{k}$ , so a call that uses its whole budget waits $T_{\text{wait}} = b\,(f^{r}-1)/(f-1)$ , here 1.5 s for two retries and 7.5 s for four, on top of the failed attempts themselves.

Code

The retry loop, cut from the vendored engine (harness.ts):

if (this.fault(name)) {
  if (attempt < rec.retries) {
    const backoff = rec.backoff_ms * rec.backoff_factor ** attempt;
    attempt += 1;

The success test the chart uses (sweeps.ts): finished, and the last test run passed.

return events[events.length - 1]!.status === "done" && lastOk;

Recovery in the public harnesses

Harness	Recovery and limits (from their docs, accessed 2026-10-07)
SWE-agent	`max_requeries` (default 3): "Maximum number of times to requery the model after an error, such as a formatting error, a blocked action, or a bash syntax error" (agent configuration)
mini-SWE-agent	a cost limit ($3 by default) and a step limit (0, unlimited, by default) (the `mini` command)
Aider	after an edit it lints (and, with `--auto-test`, tests), and "aider will try and fix any errors if the command returns a non-zero exit code" (linting and testing)