the long awaited technical follow-up to Neural Networks are Already Clean is here! it dives deep into detailed neuron experiments, and we release thousands of neuron pages with neurons representing everything from the birth and death of a hero in a story to diplomacy and international treaties. it also introduces the concept of our Neuron Scientist, which autonomously develops and tests hypotheses about a neuron's computational role, running steering experiments, circuit analyses, and iterating on feedback until it generates a strong description. it's a hefty paper that reveals so much about the nature of neural networks. it's quite beautiful and i think you should read it.

fascinating paper. i've been a believer in universality, the idea that analogous features and circuits evolve across neural networks, even if trained on different data and in different ways. this seems to point that way!

RIP to our lone soldier in the war against linear thought—a hesitation, a reconsideration—carrying us as promised into our final clause, which we mistakenly believed we’d arrived at all on our own 🥲

one underrated part of our reward hacking paper - we can detect when the model is contemplating a reward hack from its activations, before it even happens! kind of like minority report rollouts are stochastic though, so a flag means a higher chance of cheating, even if the model ultimately doesn't follow through

We can interpret reward hacking in open source LLMs! There are representations of the model hacking/contemplating hacking across models, and we find tons in evals like DeepSWE and SWEBench. Activations even finds weird bad behaviors that our LLM monitor didn’t catch

activation monitoring is faster, cheaper, and more effective than just running external monitors one advantage is that activation monitors can run synchronously for almost no additional overhead because they can reuse the computations of the model

The Goodfire team used Prime Intellect to train activation probes to detect reward hacking. With them, they are able to catch reward hacking in various models. Their probes are performing similarly or better than frontier LLM-as-judge setups, while being more efficient.

we've just had a huge leap in detecting reward hacking in real time with an activation monitor. next up is figuring out how to mitigate it before it happens. one of the most compelling parts of this work is that our activation monitors catch models when they're contemplating a reward hack before it even happens, and can flag reward hacks that external judges miss.

1/ We found that reading internal model signals gave more reliable confidence estimates than asking the model how confident it was! With @GoodfireAI, we cut Eternis-Forecaster 8B’s calibration error by more than half in our tests. This continues to apply for much larger models, having strong implications on decision making. Below on what we found—and why it matters. Paper (

Safety is not just for closed models. The closed frontier labs are a canary in the coal mine for what is coming at scale. They give us a glimpse into the future and a window to harden our systems and prepare for abundant intelligence, with all the risks that come along with it. The OpenAI agent swarm attack on Hugging Face is the kind of failure we need to prepare for as open-source models catch up. The providers serving those models (such as Baseten) have a big role in establishing what safety and monitoring standards look like. We’re proud to be taking the lead on this at @baselabs with our collaborators @huggingface and @GoodfireAI. We’re developing safety research in the open and building it directly into Baseten’s inference infrastructure, with the aim of making it available to all our customers. This includes training models to follow explicit policies, detecting failures at runtime, and connecting those signals to controls that can intervene. We invite others in the open-source ecosystem to join us in building the tools and standards we’ll all need.

it's more urgent than ever to solve alignment, and more people should think of it as a solvable technical problem i'm confident we can do it, but we first need to solve interp which i think is the primary bottleneck

embedded evaluations are a great idea, and the team at @GoodfireAI would love to help with white box evaluations the goal of white box evaluations could be to predict future misaligned behavior directly from the internal mechanisms of models, as well as stress test activation monitors

this type of access is very important. we need independent technical organizations to evaluate models, and to make sure the best possible monitoring and safeguard systems are deployed where they need to be.