Akshay Kumar
All articles
Harness engineering · September 8, 2026 · 8 min read · 3 of 8

Can a model build its own harness? Five papers say yes, for games and math, and not yet for code

The idea is hard to resist once you have heard it. A harness is code, and models write code, so why not let the model write the harness, score the result, and iterate until it improves? Between February and September 2026 at least five research groups tried exactly that, and their results are clean enough to say where the idea works and where it breaks.

The short version is that when the environment has crisp rules and dense feedback, a harness written by a model can beat a bigger model that has no harness. When the task is repository-scale software engineering or open-ended research, the model-built harness lands ten to forty points behind the one a person wrote, and it stops working the moment a different model runs it.

Where it works: rules you can check

The earliest of the five papers is AutoHarness, published in February by Xinghua Lou, Kevin Murphy and colleagues. Its motivating number is the one that made me read the rest. In a Kaggle chess arena, 78 percent of Gemini-2.5-Flash's losses came from illegal moves, meaning moves that broke the rules rather than merely weak ones. The model knew chess well enough to play and not well enough to avoid breaking a rule.

Their fix was to have Flash write a code wrapper that validates each move against the rules before it is played, and to refine that wrapper over a few rounds using the environment's own error messages as feedback. The wrapper eliminated illegal moves across 145 TextArena games, and Flash with the wrapper outperformed Gemini-2.5-Pro without one. Pushed further, the model wrote an entire policy as code for 16 single-player games, and that code beat both Pro and GPT-5.2-High on average reward with no model call at decision time. The abstract's conclusion is that "using a smaller model to synthesize a custom code harness (or entire policy) can outperform a much larger model, while also being more cost effective."

That is a real result, and it is worth being precise about what kind of result it is. It concerns environments where the question "was this action legal" is a function someone can write down.

Meta-Harness, from March, widens the search space to whole harness programs, with prompting, retrieval, memory and state management all expressed as Python. The proposer is Claude Code with filesystem access to every prior candidate's source, traces and scores. It runs about 60 candidates over 20 iterations and consumes on the order of ten million tokens per evaluation. On text classification the discovered harness reached 48.6 percent against a 40.9 percent baseline while using four times fewer context tokens. On 200 IMO-level problems it added 4.7 points on average across five held-out models, and on Terminal-Bench 2 it took Opus-4.6 to 76.4 percent.

What I found most interesting is that the discovered harnesses are readable. The math one is a four-route BM25 retriever with per-route deduplication. The Terminal-Bench one runs a few shell commands to gather the operating system, language and package-manager facts before the agent loop starts, which saves two to four exploratory turns. A person could have written either of them; the search simply found them faster.

Two more papers refine the search itself. Agentic Harness Engineering adds observability, so that every proposed edit comes with a prediction that is then checked against the outcome. Ten iterations took a Terminal-Bench 2 harness from 69.7 to 77.0 percent, past the human-designed Codex CLI at 71.9, and the ablations found the gains sat in tools, middleware and memory rather than in the system prompt. DemoEvolve handles environments where reward is sparse, such as the card game Balatro, by giving the proposer human trajectories to diagnose against. Its one-line claim is that "demonstrations make sparse-feedback harness evolution more diagnosable, localizable, and stable."

Where it does not: code and research

In September a benchmark arrived that asked the question directly. HarnessDev gives six frontier models a minimal seed and one to three development cases, asks each to build a complete harness, and then evaluates that harness on 2,207 held-out task instances across five benchmarks. SWE-bench Pro and Terminal-Bench 2.1 cover code, MLE-bench covers data analysis, EQ-Bench3 covers writing, and BrowseComp covers research.

The best creator, Opus 4.8, building a harness for itself, against the human-engineered reference in each domain. Source
The best creator, Opus 4.8, building a harness for itself, against the human-engineered reference in each domain. Source: HarnessDev, arXiv 2609.01437, Section 4.2.
On writing, the model-built harness matched the human one, at 84.6 against 83.7. On MLE-bench it did better, with 32.9 medals against 24.0. On SWE-bench Pro it trailed by 10.7 points, at 69.3 against 80.0, and on BrowseComp it trailed by 39.8, at 52.4 against 92.2.

The pattern lines up with the earlier papers. Writing and machine-learning experimentation have dense, fast feedback, because the output is scored and the loop is tight. Repository-scale code and open-web research have long horizons and sparse signal, which is exactly the regime DemoEvolve found too noisy for reward-only search.

The number that matters most

HarnessDev then ran the experiment nobody had run before. The authors took the harness that Opus had built for itself and ran it, unchanged, with Gemini 3.1 Pro as the executor.

The same harness under its author and under a fixed Gemini executor. Source: arXiv 2609.01437, Section 4.2.
The same harness under its author and under a fixed Gemini executor. Source: arXiv 2609.01437, Section 4.2.
Opus's SWE-bench Pro score fell from 69.3 to 33.0. In the other direction, a harness that Qwen had built rose on BrowseComp under Gemini, from 32.3 to 49.9. The rankings shuffled. The paper's reading, in Section 4.2, is that "a runnable harness can therefore be used by another model, but capability transfers only when its prompts, tool protocol, budgets, and stopping rules remain compatible."

This is the same conclusion the harness-variance literature reached from the opposite direction. When Stop Comparing LLM Agents Without Disclosing the Harness held the models fixed and varied the harnesses, it found six ranking reversals across nine model-pair comparisons. A harness is tuned to a model whether a human or a model did the tuning, and automated search only makes the tuning tighter and the hardcoding harder to see.

The second HarnessDev stage, in which a model improves its own harness from feedback, turned out to be less stable still. Gains on the feedback set ran from 3 to 14 points, but on held-out tasks they shrank to between 1.4 and 4.4. Under a fixed Gemini runtime, only Opus improved on held-out tasks and three of the creators regressed. Of the 64 version switches the creators made, two showed clear evidence of improvement beyond a noise band of plus or minus 4.75 points. Section 4.3 draws the lesson: "Visible feedback is therefore useful for local search but unreliable for final selection: repeatedly optimizing a noisy score can favor a lucky run and amplify overfitting."

What I take from the five

I came away with three conclusions, and I hold them with decreasing confidence.

The first is that rule-checking harnesses should be generated rather than written by hand. If your environment can tell you that an action was invalid, AutoHarness is the template: let the model write the validator from the error messages, and then never let it act without one. This is the cheapest reliability win in the whole set, and it applies to any tool that has a schema.

The second is that search is a good way to find retrieval and bootstrap tricks and a bad way to pick a winner. Meta-Harness's discovered harnesses are worth reading for the ideas in them. HarnessDev's evolution results say that the score you optimized on is not the score you will get. Hold out tasks, and hold out models, before you believe a searched harness.

The third is that in practice the harness is part of the model. Every paper here that tested transfer found it partial. A harness searched for Opus is an Opus artifact. That cuts against the dream of a universal agent runtime and toward per-model harness releases, which, if you look at what Qwen Code, Gemini CLI and Codex CLI are, is what the vendors already ship.

I hold one caveat against all five papers. They evaluate on public benchmarks that the proposer models have seen, and Meta-Harness's Terminal-Bench 2 result used the same split for search and for reporting, which the authors flag themselves. A searched harness that memorizes environment bootstrapping for a known benchmark is doing something useful and something suspect at the same time.

The sentence I would end on comes from Meta-Harness's discussion: "Once a search space becomes accessible, stronger general-purpose agents can outperform hand-engineered solutions." The word doing the work in that sentence is accessible. For games, classification and math, the space is accessible today. For a repository with a history, or a research question with no oracle, it is not, and the papers that tried are honest about the gap.

Sources