Akshay Kumar
All articles
Harness engineering · September 8, 2026 · 8 min read · 2 of 8

The harness change that doubled a local model's fix rate, and the window size where it stops mattering

Anyone who has run a 30-billion-parameter coding model on a laptop has watched the same scene play out. The model reads a file, then reads it again, then proposes a test command that fails, proposes it a second time, proposes it a third time, and eventually stops with a half-finished patch. The usual diagnosis is that the model is too small for the job. A paper published in August 2026 argues that the diagnosis is frequently wrong, and it makes the argument with a controlled experiment on four models served locally.

The paper is Same Model, Different Harness: Different Coding-Agent Results, and its design is the cleanest I have seen for isolating a single harness mechanism. There is one harness, which the author calls Yuj, and it runs in two arms. The control arm sends the model the full conversation in chronological order. The treatment arm keeps the same complete record in memory but shows the model a shortened working view, and adds a small detector that notices when the model is stuck. The model weights, the tools, the serving stack, the evaluator, the tasks, and the 480-second budget per task are identical across the arms. The only thing that differs is what the model sees.

What the treatment does

The treatment consists of three mechanisms, and none of them involves calling the model.

The first shortens older tool results as the context fills. It triggers when the estimated prompt reaches half of the configured window. The newest four tool results are left whole. Older ones are capped on a schedule under which the character limit halves each time the result's age doubles, and each shortened result keeps its beginning and end around an omission marker. The full text remains in memory and in the transcript, so only the model's view shrinks. Section 2.2 states the principle in one sentence: "The treatment separates the complete in-memory conversation from the model's working view."

The second mechanism is a detector that reads fixed facts from the execution record, such as the same command repeated with the same failure, or the same file read again with no edit in between. When one of its rules fires, the harness inserts a fixed message that names the repetition and tells the model to try something different.

The third is a set of command safeguards. Test commands are reformatted to the expected form, forbidden commands are blocked before they run, and oversized setup output is kept out of the context.

That is the whole treatment. It never asks the model to write a summary, and it uses neither semantic memory nor embeddings.

What it did

The primary model is Qwen3.6-35B-A3B in four-bit Q4_K_XL weights on llama.cpp, decoded greedily, with one trajectory per task. The main cohort is 169 SWE-bench Verified tasks served at a 20,480-token window.

Under the control arm, the mean per-task fail-to-pass fraction was 28 percent and 43 tasks were solved outright. Under the treatment arm it was 49 percent and 72 tasks. The paired difference is 21.1 points, with a 95 percent interval from 14.1 to 28.3, and the sign test gives p below 0.0001. On SWE-bench Pro the fraction went from 15 to 33 percent and complete solutions from 31 to 72, and on FeatureBench it went from 11 to 20 percent.

Then the author widened the window, and this is where the paper earns its title.

Paired fail-to-pass gain from the treatment at three serving windows, on 169 SWE-bench Verified tasks with Qwen3.6-35B-A
Paired fail-to-pass gain from the treatment at three serving windows, on 169 SWE-bench Verified tasks with Qwen3.6-35B-A3B. Source: arXiv 2608.26218, Table 6.
At 43,008 tokens the gain fell to 6.4 points, with an interval from 0.5 to 12.4. At 262,144 tokens, which is effectively unlimited for these tasks, the gain was minus 0.3 with an interval from minus 4.5 to plus 3.9, which is to say nothing at all. The treatment does not make the model smarter. It makes a tight window survivable.

That one figure changed how I think about local agents. A 30B model at a 64k context that appears to fail at reasoning may be failing at the exact thing this treatment repairs, which is its own earlier tool output crowding the task out of view.

It transfers across model designs

The transfer table is the part I would show to a skeptic. The author kept the same harness, window and tasks and swapped in three more models with different architectures, without retuning anything.

Control versus treatment across four locally served open-weight models. Source: arXiv 2608.26218, Table 3.
Control versus treatment across four locally served open-weight models. Source: arXiv 2608.26218, Table 3.
Devstral, a dense transformer, went from 17 to 37 percent. Qwen3.8, a dense DeltaNet and attention hybrid, went from 20 to 35. Nemotron, a Mamba-2 hybrid, went from 12 to 18. The size of the gain differs by model, but the direction does not, and a mechanism that lifts four architectures without per-model tuning is a property of the harness rather than a quirk of Qwen.

Why "compaction" is the wrong mental model

Everyone running agents at scale already compacts context in some form. Anthropic's own context-engineering guidance describes three techniques for long tasks: summarizing the conversation when the window nears its limit and restarting from the summary, having the agent write notes outside the window, and handing focused subtasks to sub-agents that return a digest of one to two thousand tokens. All three route the decision through the model, which decides what to keep.

The Yuj treatment never asks the model. It shortens by age and by size on a fixed schedule and keeps the full text elsewhere. That distinction matters for two reasons. The treatment costs no tokens to run, which on a laptop means it costs no seconds, and it is inspectable, because you can read the rule and predict the view.

A July 2026 paper makes the case for why the model should not be the one deciding. LLM Agents Are Latent Context Managers finds that frontier models are "proprioceptively blind to their own context," meaning they cannot reliably tell how large a block is, how old it is, or how much budget remains. Its fix, called VISTA, is a dashboard of token usage and recency that the model reads before choosing what to archive, and it lifts Gemini-3-Flash on LOCA-Bench from 22.7 to 50.7 percent. The mechanism is different from Yuj's, but the premise is the same: the harness knows things about the context that the model does not, so the harness should act on them.

The cost, stated plainly

The treatment lets the model keep working past the point where the control arm would have run into the wall. That is the point of it, and it is also the bill. On the Verified cohort, model turns rose from 3,280 under control to 6,517 under treatment, prompt tokens rose from 37.8 million to 80.7 million, and wall time rose from 1.3 hours to 4.6. On SWE-bench Pro, 5.4 hours became 21.8.

The paper is honest about this in its limitations section. The comparison "estimates the treatment-package effect as implemented" and does not match the two arms on compute. Under treatment the model works longer because it can. Whether that is what you want depends on whether you would rather have a wrong answer in eighty seconds or a right one in four minutes, and on a laptop where the model is the only thing running, I know which one I want.

One detail I did not expect: at the unconstrained 262k window, the treatment served slightly fewer prompt tokens per turn, a ratio of 0.93 with an interval that stays below one. Shortening old tool output is cheap even when it is not needed.

What this does not show

The study ran one trajectory per task with greedy decoding, so it measures no run-to-run variance, and the author says so in Section 6.4. It covers four open-weight models at four-bit precision and no hosted frontier model or higher-precision weights. Evaluation is test-based only, so nobody assessed patch quality by hand. And it is one harness, the author's own. Whether mechanical shortening helps inside OpenHands or Codex CLI, which already carry context strategies of their own, is an open question, and it is the one I am running next with the same local models, four public harnesses, and repeated runs.

What to do with it tonight

If you serve a coding model locally, look at your window before you blame the model. Qwen3-Coder declares a 262k limit, but the server runs whatever you configure, and memory is what sets it. At 20k the harness is most of the outcome. At 64k it is worth a few points. At 256k it is worth nothing, but 256k of KV cache for a 30B model is real memory on a 64 GB machine.

If you write a harness, the rule that keeps the four newest results whole and the halving schedule together come to about thirty lines, and the detector for repeated commands is shorter than that. Section 6.1 gives the reason to bother: "A model name alone does not identify the solver behind measured coding-agent performance." What the model saw is part of the solver.

Sources