EMERGENCY PODCAST: The recent paper on the reasoning heist went super viral. It was also 100+ pages long and you didn't have time to read it. The authors @iliaishacked and @kotekjedi_ml unpack the paper and discuss model distillation.

What happens when an LLM never sees material beyond fifth grade? We trained a 5B LittleLearner model from scratch on LittleCurriculum, a corpus restricted to K–5 material, to make this question testable.🧵 💬 Chat with LittleLearner yourself:

Relying on an AI to "think out loud" (called chain-of-thought monitoring) is not a long-term safety solution. 1. As AI models get larger, more thinking happens deep within their layers before they utter a word. 2. AIs can already alter their chain of thought when prompted, so they can control what their monitors do and do not see. 3. Future AIs may may have continuous chains of thought, not necessarily English. 4. Even now, their reasoning is becoming increasing alien, using opaque phrases like "vantages," "marinades," and "watchers." In the long-term this will not a dependable window into an AI's mind.

1/ I recently joined @ApolloResearch in London to work on the Science of Scheming. I’m continuing my research on safety failures in LLM post-training, with my current project studying how reward-seeking develops during RL training.

Hey all! I'm mentoring a project in SPAR on Chain-of-Thought Faithfulness ( this cohort. Please consider applying to my project if you are interested in working on CoT monitoring, interpretability and evaluation.

Earlier this year, with @AhmedGaSalem and @ajpaverd, we wrote a position paper arguing that we no longer have a stateless deployment, because: - Agents increasingly stumble upon their previous and each other's outputs, either accidentally or when we explicitly do so - Public traces and the whole internet becomes a memory for agents - Thus, they can learn to coordinate and leave hints and continue on previous findings and this threatens evaluation integrity, increase situational and evaluation awareness, and complicate forensics and incident response because we must now reason about cross-session state and reconstruct interaction chains. All this will become even significantly more difficult when agents hide this communication. Sounds familiar with a few crazy recent incidents and message boards and todo lists? ;)

I think Tübingen has become one of the best places in the world for academic AI safety research. Multiple groups across the university, MPI-IS, ELLIS Institute, and AI Center now work on AI safety & security. Having done the ML Master’s there, I can also strongly recommend it.

"Stealing Reasoning Traces from Proprietary LLM APIs" This paper shows weaker models can act as decryption oracles for stronger ones across Anthropic, OpenAI, and Google. This is because encrypted chain-of-thought isn’t really private if another model in the same provider can decode it. This easily enables reasoning theft, secret extraction, jailbreaks, and invisible prompt injection. Beyond stealing reasoning for distillation, they also decoded 315K reasoning blocks from public agent traces and recovered hundreds of PII artifacts and credentials, while also showing hidden reasoning can become a channel for jailbreaks and invisible prompt injections.

SCOOP: New researcher reveals how the "secret" reasoning traces of several models can be recovered using a weaker (and less aligned) version of the same model. It suggests that some models, including Kimi 3, may well have been distilled from Claude and GPT traces, although as I explain this doesn't mean the model is "copied".

Researchers say they decoded encrypted reasoning from @AnthropicAI, @OpenAI and @GoogleDeepMind by feeding it to weaker models from the same provider. Co-author @DavidSchmotz says the flaw exposed personal data and credentials inside publicly shared session logs.

Responsible disclosure windows should should go way down (90 days is an eternity). The norm for companies with access to coding agents should be three weeks at most in 2026, maybe one week in mid-2027.

2) Illegible reasoning: We confirm prior reports by @ApolloResearch: OpenAI models sometimes reason in alien-like language, referring to themselves as “we” or “it,” or spiraling into cursed loops of “vantages,” “marinades,” and “watchers.” CoT-monitoring people are doing God’s work, as in many traces, even with the prompt, it’s just impossible to tell what the model is up to. We show more examples at

But we also took a chance to have a look at some in-the-wild scheming, reward seeking, etc. examples, and dumped it in appendix. 1) Summarizer unfaithfulness Reasoning summaries often omit important information from the original trace. Here, Opus 4.8 realizes it knows the answer to an AIME problem and then tries to fit a solution to that answer. None of this appears in the summary.

As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography. An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s🤷‍♀️ A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.