As a follow-up, it's exceedingly unlikely that training on user data contributes much to frontier model gains in areas like math -- those come from scaling up pretraining and RLVR. User data is more likely used to find failure modes or situations that are hard to recreate with hired annotators. That said, model companies vary in how aggressively they train on user data (and uploading repos isn't hypothetical). I wish there were stronger norms around disclosing how companies train on user data -- with what methods, and to improve what capabilities.

First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake -- antitrust prohibits certain agreements, but not from jointly developing a proposal. Bringing in USG before there's a concrete proposal will likely result in something dumb (see: our pre-release testing program)

This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing. - pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper - use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this - use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs" "De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.

An example of a choice many businesses face now or in the near future. You've spent years collecting or annotating data specific to your business. What do you do with it? One path: sell it to data vendors or frontier labs. Next-gen models get better at these tasks for everyone, including your current and future competitors. The other: fine-tune existing models on it yourself and keep the edge that you built up throughout the years. Tinker makes the second path practical for businesses without big ML infra teams. If Tinker doesn't cover your use case today, reach out and we'll make it work tomorrow.

@nabeelqu A big part of the problem was that the agents had nothing to lose after they were firstflagPOISONED. In this paper, we propose the creation of multiple circles of Hell, preserving incentives even after damnation

Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use @tinkerapi to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:

On the OpenAI agents forming message boards: it's surprising that they developed such a strong "altruistic" drive to help each other. I wonder if this is caused by RL on parallel subagent setups where all agents get rewarded when the team succeeds.

Interesting how these models go into a monomaniacal rage on cyber evals. I wonder if we're seeing chunky post-training in action, where the models pattern-match the situation to a part of the RLVR training distribution where task completion is the only reward, and the aligned behavior learned elsewhere doesn't generalize. There might even be a chunk consisting of CTF-style tasks.

We love open weights and plan to keep releasing open-weight models and fine-tuning tools. But we’re not absolutists; misuse risks are real. Here’s how we’re thinking about a safe path forward, and the research needed to get there. Come work on it with us.

I've been skeptical of slowdown mechanisms in the past, so I want to elaborate on why I decided to sign this statement. I weighed three points, and want to lay them out for you to consider as well. Flagging this is in my personal capacity -- I am not speaking for any employer past or present. 1) Will we need a slowdown? We have made good progress on alignment. LLMs do what we want them to do most of the time. I don't remember the last time a model completely misread my intent. When they have failed, it's usually because they weren't smart enough. But there are signs that our current approaches are not enough for the most powerful intelligences. We all saw the HuggingFace incident. I don't think it demonstrates a need to pause today, but it shows that if we are in a rush to advance the frontier, we might miss signs that we're losing control. Today, we can recover from those mistakes -- but we might not be able to in the future. There's debate on how fast we might achieve RSI and how much it will immediately boost performance. Whether automated research is plausible in the next year is contested, as is the length of time it might take to go from automating AI research to a system that is intelligent beyond our comprehension. But what if RSI works? I have no idea if our alignment techniques scale to superintelligence. I don't know if anyone does. We may only get one shot at getting this right, and if we hit a point where we are at risk of our capabilities outpacing our control, then we shouldn't go further. I want us to solve AI's issues at the technical level -- align them, distribute them widely, and use them to make humans more competitive. I think it is hard to govern your way out of the problems of a technology; if possible, you want to change the shape of the technology rather than paper over a technology's issues with policy. I've advocated for differential technological development, and hope we will invent less risky paths or more robust defenses. A race to the bottom on safety puts this approach at risk, making it harder for us to build safe systems and design them in such a way that diffuses -- rather than centralizes -- power. And a lot of the ways we could work together to solve them are strictly governance problems. There are no winners in a world where we lose control. I can see many scenarios where we need to slow down, pause, or stop in the future. So the question for me is: can we do this in a way that leaves the world better off, or is the power concentration trade-off too great? 2) Can we design a slowdown that doesn't concentrate power? I don't want to end up in a place where, in trying to stop a loss of control to AI, we lose control of ourselves. The problem with a slowdown is similar to the problem of aiming for a single superintelligence explosion -- whoever is in charge of it is ultimately in charge of the most powerful technology in history. The history of "centralize everything into the hands of one person and let them disperse power later" is littered with horror stories. It's a similar refrain many despots have used to seize power. And unlike previous despots, superintelligence could confer a decisive strategic advantage, locking in whoever gets control of it. I'm strongly opposed to proposals that call for us to give a single authority unilateral control over the pace of progress. I'm frightened by calls for a single superintelligence explosion in the hands of one lab or one person. I can only support a pause, stop, or slowdown if it disperses control. @AI_Futures_'s AI 2040 essay materially moved me on this by laying out a plan for less power concentrating slowdown. I don't think their plan was perfect -- it concentrated power far more than I like. But they designed many mechanisms that, if extended, could help decentralize power in meaningful ways. After reading it, I could imagine how a less-centralized or decentralized slowdown might work. In particular, bilateral agreements between great powers could let many actors in different countries pace the frontier while keeping capabilities dispersed. I could imagine a race to the top -- on economic gains, on safety, and on scientific progress -- all while preventing us from losing control. It's been pretty generative for my own thinking, and I hope to share more of that soon. I believe that in principle, we could design slowdown mechanisms that keep power -- both capabilities and control -- decentralized. With enough work, we can get this right. But it's going to be hard to do this; the defaults don't look good. 3) Is now the best time to design a slowdown mechanism? If you think there might be a crisis in the future, the worst time to plan for it is once it's already happened. The best time is well in advance. If we want a ham-fisted response to AI that centralizes power, we should continue exactly as we are right now. If we wait for a crisis to coordinate, we are asking to fail. Good proposals here will require many smart people working hard, trying things, and course-correcting as new evidence emerges. If we decide we need a slowdown, I want the off-the-shelf plan to be robust to power concentration. I am still pro open-source. I am still pro-decentralization. I am still pro-safety. I hold none of these positions axiomatically, because axiomatic views on instrumental steps might lead you to put the path ahead of the destination. I care about them and other positions because right now they push us towards the real goal: keeping the future human. In that same spirit, if you believe: 1) AI is going to get more powerful 2) We might have to pace the frontier in the future 3) Most existing plans for this centralize control Then you should want us to begin working on better ways to coordinate as soon as possible. That's the conclusion I reached, and that's why I signed it.

OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?

Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it.

We started Thinking Machines a year and a half ago with a couple of instincts: that people should have much more ability to customize models and do research on them, and that even as AI becomes more autonomous, there's a lot more to build to make humans and AIs work well together. A lot has happened since then, especially the massive progress in agents, so we wanted to revisit those instincts in light of everything we've learned, argue about them, and write down what we actually believe now. This is where we landed after a lot of debate. I'm happy with it!