Akshay Kumar
All articles
Harness engineering · September 8, 2026 · 8 min read · 1 of 8

The 100-line harness and the million-line harness solve the same tasks. What are the other 999,900 lines for?

Open the source of a coding agent called mini-swe-agent expecting the usual sprawl and you find an agent class of about a hundred lines of Python. It has exactly one tool, which is bash. Every action runs in a fresh subprocess, so there is no shell session to keep alive between steps, and its memory is a flat list of messages that grows and is never pruned. The README says the thing "scores >74% on the SWE-bench verified benchmark." Harnesses three orders of magnitude larger compete for the same leaderboard.

I know they are three orders of magnitude larger because a source-code study published in September 2026 counted. The study, Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents, read eleven systems line by line. Codex CLI is roughly 1.1 million lines of Rust. Claude Code and OpenCode are each past half a million lines of TypeScript, and Gemini CLI, Hermes and OpenClaw are north of 600,000. Mini-swe-agent comes to about 5,000 lines once you include the environment, model and run scripts around that hundred-line class.

That spread raises an obvious question for anyone choosing a harness, or writing one: what do the extra lines buy? Two papers from this summer answer it with measurements, and the answer turned out not to be "a higher pass rate."

Approximate code size of eleven coding-agent harnesses, using the midpoints of the ranges reported in arXiv 2609.00006.
Approximate code size of eleven coding-agent harnesses, using the midpoints of the ranges reported in arXiv 2609.00006. The spread is about 200x.
## Same model, three harnesses, same pass rate

The first paper is The Scaffold Effect in Coding Agents, from July 2026. The authors took two models, Qwen 3.6 Plus and MiniMax M2.5, and ran each through three open-source harnesses, Goose, OpenCode and OpenHands-SDK, on a fifty-task subset of Terminal-Bench Pro. They drew the fifty tasks by stratified sampling across eight categories before any harness ran, and they removed nothing afterwards, which is the kind of detail I now check first in any agent paper.

The pass rates barely moved between harnesses. With Qwen, Goose solved 48 percent of the tasks, OpenCode solved 50, and OpenHands-SDK solved 50. With MiniMax the figures were 38, 46 and 46. Each of those carries a confidence interval of roughly plus or minus 14 points, because fifty tasks is fifty tasks, and the authors' own summary is that the within-model differences stay inside 0 to 8 points with none of them separating from the others.

The token counts told a different story. Per solved task, Goose spent about 28,000 tokens with Qwen and 37,000 with MiniMax. OpenHands-SDK spent 841,000 and 843,000. OpenCode spent 1.15 million and 1.55 million. For the same model reaching the same result, that is a spread of 23x to 42x.

Pass rate against tokens per solved task on a log scale. The three harnesses sit on a horizontal line. Source: arXiv 260
Pass rate against tokens per solved task on a log scale. The three harnesses sit on a horizontal line. Source: arXiv 2607.22585, Tables 2 and 3.
My first guess was that the expensive harnesses simply ran longer, but the turn counts rule that out. Goose averaged 18 to 25 turns per task and OpenCode averaged 22 to 27, a ratio of about 1.2. The paper says it plainly in Section 4.2: "The 40x token gap does not come from OpenCode running unboundedly long; it comes from per-turn token volume." The larger harnesses put more into every single prompt, in the form of bigger system prompts, more tool schemas, and more of the file and shell history carried forward each turn.

There is one more number in the paper that I found telling. OpenCode produced about two no-action turns per task, meaning turns where the model replied but did nothing, while Goose produced 0.2 to 0.3. A tenfold difference in idle turns showed up for both models, which makes it a property of the harness rather than of either model.

The authors close Section 5.3 with a sentence I would hang on the wall of any team that publishes agent benchmarks: "Reporting only pass rate against model name conflates two independent sources of variance and discards the cost and oversight signals that determine whether a coding agent is actually deployable."

What the code is actually for

If the pass rate is flat across a 200x range of code size, then the code must be doing something other than solving benchmark tasks, and the eleven-system study explains what. It breaks every harness into seven subsystems: the agent loop, the LLM integration, tools and actions, memory and context, safety and permissions, orchestration, and extensibility. For each subsystem it names the smallest and the largest implementation among the eleven.

Mini-swe-agent turns out to be the smallest implementation of five of the seven. Its loop is a linear while over one bash tool, its LLM integration is one LiteLLM call and one Jinja template, its only tool is bash, its memory is an unbounded linear history, and its safety layer is a cost limit and a step limit.

The largest implementations are where the lines live. Claude Code ships 43 typed tools with deferred loading. Codex's safety layer stacks policy rules, an LLM approval reviewer, and an operating-system sandbox on three platforms. OpenHands runs an event-sourced conversation over a persistent log with parallel batches. Hermes maintains five transports and 29 provider profiles, and Pi treats everything as an extension.

The study's first cross-cutting observation is that "loop sophistication does not predict benchmark performance." Its second is that the production mass goes to safety, user experience, extensibility, clients and transport. So the other 999,900 lines are not there to solve SWE-bench tasks. They exist so that a person can use the agent safely, on their own machine, with their own tools, without having to read the transcript afterwards.

Two smaller findings from the same paper are worth knowing if you are building one of these. None of the eleven systems imports a general-purpose agent framework, so there is no LangChain, no AutoGen and no LangGraph anywhere in the set. And none of them uses vector embeddings to find code. All eleven rely on deterministic retrieval, meaning ripgrep, tree-sitter, glob, and Markdown context files found by path. Whatever the marketing says, the field runs on grep.

What this changes about the decision

When I put the two papers side by side, a decision rule fell out, and it depends on what you are doing.

If you are measuring a model, you want the smallest harness that can complete the task, and you want to say which one you used. Mini-swe-agent exists for exactly this purpose. It removes the harness as a variable rather than hiding it, and because everything is bash, you can read every action the model took. A number produced this way is close to being a property of the model alone.

If you are deploying an agent, the pass rate is not the number to optimize, because at a given model it is roughly constant across harnesses. Tokens per solved task vary by 40x, and the Scaffold Effect authors point out in Section 5.1 that at one model and one task, 40x more tokens means 40x more spend, which swamps the one-to-two-times price difference between the strongest models. Latency scales with the same tokens. The sensible way to choose a harness is by what it costs to reach the same answer, and by how much it lets you see and stop.

If you are building a harness, the seven-subsystem list is a roadmap and the minimal column is the starting point. The study ends with a 90-line minimum-viable scaffold: a while loop with a turn counter, one provider call with template assembly, one bash tool, linear message accumulation, token and step budgets, no orchestration, and structural Python protocols for extension. Everything added beyond that should have a named reason drawn from the maximal column, whether that reason is safety, a second tool, or the user's editor.

What these two papers do not settle

The flat-pass-rate finding rests on fifty tasks, two models, three harnesses, and one run per cell. The intervals are wide enough that a harness ten points better would not have separated in this design. What the design does establish is that there is no large effect on pass rate, and that the token effect is both enormous and consistent across models.

Neither paper covers proprietary harnesses under controlled conditions. Claude Code and Codex CLI appear in the source study, but the controlled comparison ran through Harbor with three open-source harnesses, and the proprietary ones were not in it.

And neither paper says anything about models served locally, where tokens are seconds rather than dollars. That is the study I am setting up, and the first cell runs tonight.

The sentence I keep returning to is from the source study's introduction, which observes that in five months harness engineering "went from a phrase coined in a vendor blog post to a discipline with practitioner guides, formal definitions, automated-evolution systems, and its own arXiv genealogy." The discipline's first measured result is that its central artifact does not move the number everyone reports, but moves nearly everything else.

Sources