Akshay Kumar
All articles
Harness engineering · September 8, 2026 · 9 min read · 5 of 8

Three teams built a long-running agent harness this year. They converged on the same three roles.

Anthropic has published two engineering posts about getting an agent to work for hours without a person in the loop. The first, Effective harnesses for long-running agents from November 2025, is about bridging sessions. The second, Harness design for long-running application development from March 2026, is about separating the agent that builds from the agent that judges. Both are well written, and neither contains a controlled experiment.

Since March, two research groups have measured the same idea against held-out benchmarks. Their vocabulary differs from Anthropic's and from each other's, but the structure they arrived at is the same. I want to lay out what the posts claim, what the papers found, and what is still folklore.

What Anthropic said

The November post describes a coding agent that works in sessions with no memory of the previous one, which the authors compare to "engineers working in shifts with no knowledge of prior shifts." Their fixes are mechanical. The first session gets a different prompt, which sets up an init.sh script, a progress file and an initial git commit. A feature list in JSON, running to hundreds of entries for a large application, gives each feature a boolean that starts false, with instructions that it is "unacceptable to remove or edit tests." The agent works on one feature per session, commits and writes a summary before its context runs out, and verifies its work through the browser the way a user would rather than through unit tests alone.

The March post adds the part I think matters more. Their earlier single-agent runs failed in two characteristic ways. One was "context anxiety," in which the model wrapped up early as its window filled. The other was self-grading, where "agents tend to respond by confidently praising the work, even when, to a human observer, the quality is obviously mediocre." The design that fixed both has three agents. A planner turns a short prompt into a full specification. A generator implements one sprint at a time. An evaluator drives the running application through Playwright and grades it against criteria that the generator and evaluator agreed on beforehand, in what they call a sprint contract, which in their example ran to 27 checkable items for one sprint. The post's own summary is that "separating the agent doing the work from the agent judging it proves to be a strong lever."

The numbers in the post are costs rather than scores. A solo run took 20 minutes and nine dollars. The full harness building a retro game maker took six hours and two hundred dollars, and a later version building a digital audio workstation took three hours and fifty minutes and about 125 dollars. There is no baseline, no task suite and no pass rate. That is not a criticism of the post, which never claims to be a study, but it means the evidence has to come from somewhere else.

What LongHorizon-Harness measured

LongHorizon-Harness, from August 2026, reframes a long task as a state-management problem and builds a loop around three roles that the authors call Manager, Executor and Auditor.

The loop, with Anthropic's role names alongside the paper's. Only the audit report persists between rounds.
The loop, with Anthropic's role names alongside the paper's. Only the audit report persists between rounds.
The manager keeps an explicit task state outside any context window. It records requirements, artifacts and facts, marks each as completed, pending, blocked or untrusted, and attaches a pointer to the audit evidence behind each mark. On every round the manager reads that state and picks the next bounded subtask. The executor receives a fresh context and a budget of 1,800 seconds, does the one subtask, and returns a report; it does not get to verify its own work. The auditor, equipped with read-only tools and 300 seconds, inspects the environment and files a report of its own that says whether the subtask is complete, incomplete or blocked, and whether the environment is clean, suspect or in violation. Only what the auditor confirms moves the state forward. Section 2.1 puts it this way: "Only the task state and its supporting audit reports persist; the executor's raw interaction trajectory is discarded after each round."

Read next to the March post, this is the generator and evaluator with a bookkeeper added, plus the discipline that the worker's transcript is thrown away.

A single agent loop against the Manage-Execute-Audit loop. Qwen 3.7-Plus unless noted. Source: arXiv 2608.01964.
A single agent loop against the Manage-Execute-Audit loop. Qwen 3.7-Plus unless noted. Source: arXiv 2608.01964.
With Qwen 3.7-Plus as the model, WeaveBench went from 51.8 to 80.7 percent and Terminal-Bench 2.1 went from 69.7 to 77.2. OSWorld 2.0 went from 2.8 to 8.3 on its binary metric, and a Claude Opus 4.7 subset went from 20.6 to 35.3. The gains were largest on tasks that carry a lot of state, with design and spatial tasks up 50 to 60 points on WeaveBench, and smallest on single-capability tasks such as arithmetic. Video-processing tasks got worse, by 33 points on Terminal-Bench, because an auditor that reads visible state cannot check temporal precision.

The auditor is not free. It consumed between 19 and 38 percent of all tokens, and the manager took 2 to 8 percent. A fifth of the budget spent on checking is the price of not trusting the worker, and Section 3.3 states the trade the way I would: "the model determines available actions within each round, while the harness determines how reliably these accumulate into end-to-end completion." The paper also offers one more number for the harness-versus-model argument. Qwen with this harness scored 0.733 on their composite metric, above Claude Opus 4.7 without it at 0.680.

What Harness-of-Harness measured

Harness-of-Harness, from September 2026, wraps an existing coding harness rather than replacing it. Its three roles are Project Planner, Developer and QA Tester. The planner reads the specification and the accumulated evidence and defines one bounded objective. The developer implements that objective with tests. The QA tester evaluates a frozen candidate independently. Two kinds of state cross the loop boundary: the artifact, meaning the code and configuration, and the evidence, meaning verified behaviors, unmet requirements and observed failures. Section 3.4.3 gives the principle: "Acceptance follows observable evidence rather than the Developer's knowledge of its implementation or its completion claim."

The failure modes the authors list from plain long runs read like the March post's diagnosis in more words. Agents lose earlier requirements, make local fixes that break something elsewhere, fall into inspect-and-repair loops that never advance, declare completion prematurely, and regress behavior that had already been validated.

Their numbers are relative to running the same harness and model without the wrapper. On GameCraft-Bench, three loops added between 16.6 and 22.1 points across three configurations. On FrontierSWE the gain was 19 to 29 points, and with Codex and GPT-5.5, ten loops reached 72.7 percent dominance against 27.3 without the wrapper. On ProgramBench the gain was 6.1 to 16.9 points. At matched token budgets, two wrapped loops beat three unwrapped passes on GameCraft, 64.8 to 58.2, while using slightly fewer tokens.

Their long-run case study is the part to read if you read only one section. Seventy loops over several days built a playable first-person shooter with a five-minute narrative, combat, 18 enemies across three regions, control-point captures, a HUD, menus and cinematics. Along the way the system opened 81 issues, closed 65, and reopened 17 as regressions. That last figure is the honest one. The loop did not prevent regressions; it caught them.

The pattern, and what is still unmeasured

Three teams with three vocabularies landed on one structure. There is a planning role that owns the state, a working role with a fresh context that never grades itself, and a checking role that trusts only what it can observe. The March post reached it from failed single-agent runs, LongHorizon-Harness reached it from a state-management framing, and Harness-of-Harness reached it by wrapping harnesses that were already good at short tasks. When three groups arrive at the same structure by three different routes, the structure is probably load-bearing.

The evidence supports a few specific claims. Separating work from judgment improves held-out benchmark scores by 5 to 30 points, with the largest gains on state-heavy tasks. Discarding the worker's transcript and keeping an audited state file is enough continuity. The checker costs about a fifth of the tokens.

What remains folklore, in the sense that nobody has measured it, is the November post's specific rituals: the init script, the progress file, one feature per session, and the JSON feature list with immutable tests. All of them are plausible, and all of them appear in the take-home example repository Anthropic published for Code with Claude 2026, which ships a default-fail contract file, a fresh-context evaluator with no write tools, a kill switch and a steering file. None of it has a number attached. Nor does anyone report what happens when the auditor is wrong, which with LLM auditors it sometimes will be.

Every result here also comes from a frontier or near-frontier model. AgentFloor found that on its long-horizon tier no open-weight model up to 32B cleared any threshold, and GPT-5 reached 10 percent. Whether a manager-executor-auditor loop rescues a small local model or merely triples its runtime is an open question.

If you are building one

I would start with the auditor. Give it read-only tools and a context that never saw the build, and make it produce a structured verdict rather than prose. Everything else in these three designs is downstream of not letting the worker mark its own homework, and that one change is where all three measured gains came from. After that, keep a state file that the auditor writes and the planner reads, and throw the worker's transcript away. If that sounds like a project management process, it is one. The papers showed that it works on software that writes itself.

Sources