Twitter is once again consumed by the debate over slowing down AI. Anthropic is calling for slower development of frontier models and stronger safety oversight.
We had a similar discussion a few weeks ago. This time, though, Dario has laid out a serious, detailed argument in a lengthy essay, and both Sam and Elon have expressed support.
There’s plenty of skepticism. The most common argument is that training models has become too expensive, and frontier labs want an excuse to slow down. Or perhaps they want to let the market catch up with model capabilities, then ramp up training again once demand grows.
An important distinction here is that slowing down releases does not necessarily mean slowing down training. Both Anthropic and OpenAI have developed advanced internal models that are not publicly available. And the safety risks AI poses to people vary substantially across industries.
In some fields, AI is rapidly dismantling organizational structures and ways of working that have been in place for decades.
For the first time, capital can be used to rapidly scale up compute and focus it on a single problem, without the years of effort traditionally required to train people and build organizations. Agents are designed to follow instructions.
Look at what just happened in mathematics. On September 8, OpenAI announced a solution to the Navier–Stokes existence and smoothness problem. The Hodge conjecture and the Riemann hypothesis could be next.
A similar way of organizing work is emerging in cybersecurity. In cases recently disclosed by Anthropic, attackers were already using multiple subagents to divide up tasks such as reconnaissance, code review, and verification of findings. Offensive and defensive operations will increasingly move too fast and operate at too large a scale for human engineers to keep track of.
This also gives frontier labs, with their vast amounts of compute and capital, something resembling a god’s-eye view of human society. In these fields, it is important to slow the pace or find ways to work with existing systems—even if every industry will eventually have to go through the transformation coding has experienced over the past two years.
Yet many industries still lack the data needed for reinforcement learning, particularly workflow traces and action logs. Coding, mathematics, and cybersecurity are fields where data is relatively accessible and feedback is often readily available. In other industries, even a standardized context layer is missing.
Take finance. You have the questions you ask GPT every day and the notes you keep in Notion. But much of an analyst’s work happens in the process of conducting research, deciding how and why to revise an EPS model, and reaching a conclusion. Analysts have long lacked—and still lack—the infrastructure to systematically capture these decision-making traces the way software engineers can record and revisit their work. We’ve spent a long time tackling this problem while building FUNDA’s own context layer.
One reason coding can support progress toward recursive self-improvement, or RSI, is the abundance of accessible, high-quality data that can be reviewed and traced back to its source. In most other fields, we first need to address the data bottleneck. This challenge extends beyond frontier models: the robotics field has been grappling with similar issues for years.
You can’t assume that every industry will compress five to ten years of progress into two years, as coding has.
When we discuss regulation and slowing down AI, we’re dealing with several different realities:
-In some fields, LLMs have already reached the limits of human capabilities.
-In others, substantial work remains to label data and change everyday work habits.
-Regulators and researchers who take a god’s-eye view, using the most advanced fields as their reference point, see enormous threats and feel a need to push back.
-Practitioners looking at the data available in their own industries, the capabilities of AI, and the ROI see a much longer road ahead. They see a need for a society-wide effort to build the foundations for reinforcement learning, as happened in coding, before the next wave of rapid progress becomes possible.
So even if we slow down, we should do so sector by sector, in stages.
We should pay attention to the fields where AI is moving fastest, but also to the many industries still struggling to catch up. Ideally, we would put more effort into helping those industries advance, rather than slowing everyone down to the same pace.
Beyond the differences between fields, coordination is another major challenge.
The debates over distillation and model-routing platforms that have dominated the past month already show how difficult it is for frontier labs to stay in sync.
Anthropic can cut off model-routing platforms access that threatens its business or its lead in model capabilities. But that spending could then shift to OpenAI.
The biggest challenge in trying to plan social progress from a god’s-eye view is that a market economy won’t simply follow a plan. Competition and commercial interests will always shape the outcome.
We already live in a society where the gap between ordinary people and AI researchers is widening.
A more detailed plan, tailored to each sector and implemented in stages, would do more to earn public trust. Cooperation among the labs today could be the first step.
もっと見る