People of @pidotdev! Many of you questioned what the heck does my extension actually bring to the table. The other day I finally saw someone publish a bench I could compare to and run it with a model I can afford it to run on (thank you, @melissapan). Two days later... Turns out Pi with Pi-Codex-Conversion in notebook mode beats all the other harnesses. I used my settings: heavy sysprompt rewrite etc. No websearch, no skills. The only caveat is it comes out slightly more expensive BUT: the env did not supply ripgrep somehow, and the first tool call was always rg with the outcome that it wasnt found. So I am assuming this woulda been slightly cheaper/efficient if it had all the tools. Sorry, I couldn't be arsed/didn't want to mess with the env too much. Bosh. Any other questions? Deets: pi install npm:@howaboua/pi-codex-conversion

"Pi reaches the Pareto frontier on both benchmarks by providing just four tools: read, write, edit, and bash. To understand how harness design affects spending, we examine costs across completed attempts, recorded turn counts, and initial context." Sharing this analysis on the Harness Tax by @MelissaPan and others.

@melissapan Thanks a lot for this thread, I wasn't aware that you can easily switch models on codex and the first time I read about PI, super interesting because I enjoy building harnesses in my own applications.

harness choice impacts the cost alot more than it does the quality > Quality delta is about 3-4% > Cost delta is 2-4x True for swebench lite i wonder how this behaves for non hillclimbed domains outside code.

Turns out the model isn't the product. The harness matters almost as much. Same model. Different harness. Different performance, different cost, different outcome. We may be benchmarking “models” when we should actually be benchmarking the entire agent stack.

@melissapan Findings are really interesting: choosing a coding model is only half the decision. The harness can change how much you pay even when task success is similar. 1/ The clearest example in this chart is Claude Fable 5: Pi cost $0.67 per run with 96.7% success, while Claude Code cost $1.33 with 97.8% success. That is nearly twice the cost for a small difference in this test. 2/ This does not mean Pi is always better, or that the cheapest option will suit every model. The chart shows that the best cost-and-success combination changes across the seven models tested. 3/ For frequent coding work, compare the same model in two tools on a handful of your own recurring tasks. Track whether each finishes correctly, how often you need to retry, and the total cost—not just the model name or subscription price. My takeaway: choose the model and coding tool as a pair. A familiar or “native” combination may be convenient, but it is worth checking whether that convenience earns its extra cost. @composio @omarsar0

This is a really interesting study. My key takeaways (and a caveat at the end): - There is no best model-harness combination for your task. You have to run your own experiements and choose based on your priorities (cost, accuracy, speed, etc.). - Simple AI harnesses are often competitive with complex ones (and much cheaper to run), so they're a good place to start. For example, start with Pi and only upgrade to a more advanced harness if you don't get the results you want. - A good harness with a weaker model can outperform a bad AI harness with a strong and expensive model. That said (and here comes the caveat), I would caution against these aggregate results. Often, the failure modes between different model-harness configurations differ widely, so you should look into individual cases and do some error analysis to get a better understanding of how each config works.

Where is the harness tax coming from? 🧐 Agents can take similar numbers of turns at substantially different costs. For Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 and 15.3 turns per attempt, yet Claude Code costs about twice as much for a 1.1-percentage-point increase in success rate. So what is happening each turn? One possible reason can even be observed at the first model call: Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas ‼️ As models become more capable, agents may need less scaffolding. For everyday tasks, harness design should therefore prioritize cost efficiency and reliability. (4/n)

Does your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge: 1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost 2️⃣A simple harness can be competitive 3️⃣The native harness isn’t always the best. Millions of people are using coding agents, but the impact of harness choice remains unclear. (1/n) More details in the thread. 🧵

Agents are becoming increasingly difficult to verify as they take on open-ended, real-world problems. That is why we’re organizing the 🔍 Who Verifies Agents? 🦘 Workshop at #NeurIPS2026#. We’re excited to bring together researchers working on robust verification, environment-grounded evaluation, process-level signals, simulators, agent safety, and real-world agent systems. Check out our website for more details, and follow our official account for future updates!