A New Study Finally Tests the Guesswork Behind Coding Agents' 'Harness' Design
Nine researchers ran 176 controlled experiments on the scaffolding wrapped around coding agents, the part almost nobody measures, and found the fixes that actually work are duller than the ones that sound clever.
Explain Like I'm 5 — the whole thing in plain language.
Imagine you hire a brilliant new intern who has read every book on the subject but has never actually done the job. The intern's raw smarts are fixed on day one. What decides whether they actually get anything done is everything around them: what notes they're allowed to keep, which tools are on their desk, whether anyone helps them plan their day. In AI, that surrounding setup is called a "harness," and it's what turns a smart-but-inexperienced model into an agent that can actually finish a coding task.
Until now, almost nobody had tested different harness setups against each other in a controlled way. People mostly just guessed. This new study, posted online on September 17, 2026, actually ran the experiment: 176 different setups, across four AI models, on real coding tests.
What they found: giving the intern a smaller notebook, and making them summarize their own notes as they go, helps a lot, mainly because it stops them from losing track entirely on long jobs. A fancy "undo" button for deleted notes sounded smart but nobody used it. Planning ahead helps a so-so intern much more than it helps a great one, since the great one already knows what to do. And handing the intern a full toolbox instead of a bare hammer only matters if they're clumsy with tools in the first place.
None of this makes the intern smarter. It changes how many of the intern's mistakes get caught before they cause a mess. That's the whole story: the model is fixed, the harness is what saves you.