This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing. - pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper - use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this - use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs" "De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.

Question for LLM enthusiasts these days, one typically kicks off post training by distilling some reasoning trajectories from some other model. it's kind of like a sourdough starter so how was the *first* model post-trained? did humans write the original set of formatting traces from scratch? (this is why Inkling used a small number of Kimi traces btw)

very cool idea. but don’t none of you dare trying to take this cute innocent little model and use it for your large-scale GRPO. its not just in bad taste, it violates international child labor laws

i'm quite excited about this! • it's cool in general that models can generate their own training data and learn from it. wasn't fully clear to me even a year ago that this would work reliably. • the thing that we're trying to build seems really important and no one has built it before • some traces from our model genuinely surprise me (e.g. the attached example, where it perfectly simulates the output of a complicated bash query, despite not being trained to do this) my main feeling overall is that there is so much to do. we see signs things are starting to work: models use their memories to produce better answers, knowing things saves lots of tokens, and all of this emerges with more compute but solving this memory calibration problem, on top of learning how to generate data in ways that scale nicely with compute, is going to take some time this is just a first step 🫡

If you think about it, it's pretty embarrassing that frontier labs still pretrain models. Don't they know there's a theory for this? Given how big the model is, and how long you're going to train, you can predict the loss exactly. No need to actually train. They spend billions on this!

But seriously. SF housing do costs seem crazy. Why are people willing to pay that much? Like, why would so many people be willing to spend so much to have great weather near two great universities and proximity to the most dynamic companies in the hottest industry in the world?

I used to walk around thinking about code. I could "see" the code in my head. It was pretty cool. If there was a bug I could often click straight into a specific file and fix it with no other context, like a type of muscle memory. Now I think in prompts. I'll be cooking dinner and recall something codex said to me, then imagine myself typing a prompt back. thinking in prompts seems strictly more embarrassing than thinking in code. not sure if I'll ever go back though

there are some really interesting rumors going around related to the distillation of open-weights models (Kimi, Qwen, Minimax, etc.) and they're very related to my PhD work The narrative [speculative]: • good distillation relies on reasoning traces, normally hidden from users • Chinese labs figured out in early 2026 how to reverse-engineer reasoning from Claude Code and Codex • they were able to collect large amounts of long-horizon data *with reasoning traces included* this way • this jailbreak led to a new wave of OSS models we've enjoyed over the past few months I'm not sure how true it is, but reasoning extractability seems like a huge uncertainty around the future of open models in particular i'm curious how much having the reasoning matters. this (plus Anthropic's messaging around distillation attacks) indicates that reasoning chains are crucial for distilling model capabilities. our research ( found something different: if you train a high-quality reasoning inverter, it's often pretty easy to reconstruct useful traces from frontier models given their outputs. figuring out how to approximate frontier model reasoning traces might turn out to be an existential problem for open weights models

one fear around superintellingence: perhaps human AI researchers use so much compute simply because we are intellectually unable to extrapolate from results we already have very possible that smarter intelligences could learn the same or much more from tiny experiments or simulations

No opinion on math PhD but I’d still strongly recommend math major. Most of us never reason. A situation arises, we act intuitively, then get RL’ed. At best we get anxious and bounce between limited hazy alternatives. But to prove a theorem you have to reason. You don’t know what precision or thinking really is until you have to sit and struggle to prove a theorem that’s new to you. The crazier the world gets, the more this skill is useful. And the world is about to get very very crazy.

it's crazy to say, but through a combination of excessive paranoia and fairly standard use of coding agents, i have found bugs in - PyTorch - PyTorch/ao quantization - slime - Tinker - vLLM - FlashQLA GDN kernels ...all just in July. I don't really even know how to write kernels