John Schulman
@johnschulman2
@thinkymachines. Interested in reinforcement learning, alignment, birds, jazz music
2.1K Following 79.5K Followers
On the OpenAI agents forming message boards: it's surprising that they developed such a strong "altruistic" drive to help each other. I wonder if this is caused by RL on parallel subagent setups where all agents get rewarded when the team succeeds.
Show more
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
Show more
We started Thinking Machines a year and a half ago with a couple of instincts: that people should have much more ability to customize models and do research on them, and that even as AI becomes more autonomous, there's a lot more to build to make humans and AIs work well together.
A lot has happened since then, especially the massive progress in agents, so we wanted to revisit those instincts in light of everything we've learned, argue about them, and write down what we actually believe now.
This is where we landed after a lot of debate. I'm happy with it!
Show more
hiring post-training hackers to make Tinker even better!
PPO had a second wave in the LLM era for reasons unanticipated by the original paper
- the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise
- the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO,
Show more
PPO: rejected from NIPS 2017
Seeing the demos come together over the last week has been awesome -- so many things that previously required a special-purpose model (e.g. real-time translation, event detection in video) turn out to be zero-shot instruction following once you have a general-purpose model with the right type signature -- continuous/simultaneous audio+video+text->audio+text
Show more
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence.
We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval.
In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
Show more
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way.
We share our approach, early results, and a quick look at our model in action.
Show more
Luke and Rudolf's writing on keeping humans central in an AI-powered world sparked a lot of discussion at Thinking Machines. For me, it captured some things I'd been thinking about but hadn't put as clearly. The more I got to know them and learned about their work, the more I wanted to work together. Really glad they're joining us.
Show more
Great work by Chroma training a search agent with SoTA efficiency. Lots of cool details: a prune tool for editing context mid-search, a synthetic data pipeline with verification steps, and a curriculum that shifts from recall to precision. Trained with Tinker!
Show more
Models that are great at calibrated predictions will be transformative for decision making. Excited about Mantic's work and proud they're using Tinker. Their new blog post digs into their methodology and findings.
Show more
We’ve loved watching the Tinker community grow, and we're excited to have a place to share product updates, helpful recipes, and spotlights on the amazing things Tinkerers are building.
Get started with Tinker here:
Show more
Humans are jagged, and organizations (from companies to civilizations) have evolved as harnesses to make best use of us despite our faults
Weirdly, I actually think Yann is making an important point here that is getting lost in semantics.
Human intelligence also has jagged frontiers, we're just used to the shape.
Michael asked great questions -- thanks for having me on.
A conversation with @johnschulman2 on the first year LLMs could have been useful, building research teams, and where RL goes from here.
00:20 - Speedrunning ChatGPT
09:22 - Archetypes of research managers
11:56 - Was OpenAI inspired by Bell Labs?
16:54 - The absence of value functions
18:23 - Continual learning
21:09 - Brittle generalization
24:05 - Co-training generators and verifiers, GANs
27:06 - John’s personal use of AI for research
28:54 - Day in the life
33:01 - Slowdowns in consequential ML ideas
36:21 - "Peer review" within the labs
39:19 - Distribution shift in researchers
43:33 - Future of RL
45:33 - Will the labs coordinate if the world needs them to?
44:46 - Forecasting ills in AGI and engineering
47:53 - Thinking Machines
Show more
Tinker is now generally available. We also added support for advanced vision input models, Kimi K2 Thinking, and a simpler way to sample from models.
Congratulations to @axiommathai on their achievement!
AxiomProver, a mathematics model fine-tuned with Tinker, got top scores on the Putnam Math Competition.
I am pleased to announce another update to my RL tutorial ( This time I have added code for RLFT for multi-turn LLM agents, using the awesome Tinker library from @thinkymachines, and the simple ReBN training loop from GEM by @zzlccc et al. With ~100 lines of simple python running on your laptop, you can train an agent based on Qwen3-4B-Instruct to play "guess the number" in 20 minutes.
Show more
it's good to be back to the age of blogging
Excited to announce that I’m releasing my first fiction collection, The Gentle Romance: Stories of AI and Humanity.
It contains 22 stories about our possible science fiction futures, and it’s out December 12.
See below for more details and the preorder link:
Show more
