A New Study Finally Tests the Guesswork Behind Coding Agents' 'Harness' Design
Nine researchers ran 176 controlled experiments on the scaffolding wrapped around coding agents, the part almost nobody measures, and found the fixes that actually work are duller than the ones that sound clever.
Am I The Idiot — the strongest case against our own take.
The strongest case against this piece: the study's own scope is narrow, and treating it as a general statement about AI competence overstates what it actually shows. Four models, two benchmarks, both of which are specifically software-engineering benchmarks (SWE-Bench Verified and Terminal-Bench 2.1). Findings about planning, tools, and context management in a code-editing loop may simply not transfer to other kinds of agent work, like research, browsing, or operations tasks, where the failure modes are different.
The methodology itself, holding the execution loop fixed while varying three components independently, is a real strength for isolating each component's effect cleanly, but it's also a real limitation: it can't detect interaction effects between components. A production harness tunes planning, tools, and context management together, not one at a time, and the "best of five context strategies" combined with the "best planning setting" found separately isn't guaranteed to be the actual best combination. The paper's own design can't rule that out.
It's also one day old at the time of this article, posted to arXiv with no peer review and no independent replication. Papers get retracted, corrected, or quietly walk back claims after review catches something the authors missed. Writing about it as settled the day after it posts is itself a little premature, and worth remembering when reading the confident tone of the summary above.
And there's a more uncomfortable objection, aimed at this outlet's own framing rather than the paper: the house line that models "would be fired the same day" without supervision is a pre-existing conclusion, and this piece reads a narrow engineering result about elision and summarization strategies as confirmation of it. A more measured reading is that sensitivity to scaffolding is true of any complex system, not evidence of incompetence. A brilliant surgeon who needs a working, well-stocked operating room isn't proof the surgeon is bad at surgery. This paper shows the environment matters. It does not, by itself, show the models are dumb.