This is a really interesting study. My key takeaways (and a caveat at the end): - There is no best model-harness combination for your task. You have to run your own experiements and choose based on your priorities (cost, accuracy, speed, etc.). - Simple AI harnesses are often competitive with complex ones (and much cheaper to run), so they're a good place to start. For example, start with Pi and only upgrade to a more advanced harness if you don't get the results you want. - A good harness with a weaker model can outperform a bad AI harness with a strong and expensive model. That said (and here comes the caveat), I would caution against these aggregate results. Often, the failure modes between different model-harness configurations differ widely, so you should look into individual cases and do some error analysis to get a better understanding of how each config works.

Tool-use is already a bottleneck in AI reasoning models, and it will become even more problematic as the underlying models become faster. The reason is that the sandbox, infrastructure, and execution environment needed for the tools (e.g., running code, search, database and file access) can be a very resource-intensive tax on top of the costs of running the AI model. For example, Hierarchical Decoupled Policy Optimization (HDPO), a technique developed by Alibaba, trains LLMs to avoid excessive tool-use. Another interesting example is Code World Model (CWM) by Meta, which uses a world model to predict code results without spinning up execution environments. This helps speed up the training of LLMs on coding tasks. Moving forward, as inference becomes faster and cheaper, I expect the other elements to catch up, such as faster containers and execution environments for AI tools.

I'm a big fan of Uncle Bob and his books. But we also have to take into consideration that he is speaking from the perspective of a person who has been coding for longer than most of us have been alive. This means he has the experience, intuition, and taste to build the right constraints and guardrails around AI agents to make sure the code they generate is reliable. For people who are new to the game, learning to read and write code should be a prerequisite to using AI coding tools for anything that is beyond a prototype or a hobby project.

This is what I think the AI research community and industry need. In the past few years, we've seen a lot of progress, but most of the focus has been on transformer-based LLMs (and a few non-transformer-based variants) and next-token prediction. But LLMs and NTP are not the only ways to achieve AI. I'm very excited about these new directions, including what @RichardSSutton is exploring and the world models that @ylecun proposes. We need to run experiments in multiple directions. Diversity wins in scientific discovery.