Agam Bhatia
@chewing_a_gum
professional scriptrunner, CS + Math @Stanford
1.5K Following 558 Followers
Progress in machine learning is bottlenecked by having good evals, interpretability is no exception. We've made WorkspaceBench: a range of scenarios where we know what the model should be thinking about, to see if your interp tool can find it
J-Lens is great but only outputs a single token. I think a good multi-token J-Lens should do well on WorkspaceBench!
Show more
We often don't know how useful our interp tools are, or when they are lying to us. We stress-test this and more in WorkspaceBench!
Astra can do crazy things without chain-of-thought. This is scary, but interpretability can help!
We made WorkspaceBench, an eval that tells you how well your interp tool surfaces the contents of the global workspace, i.e. the important intermediate variables in a forward pass.
Show more
This was a very satisfying project. An annoying problem with J-Lens is that errors accumulate as you backprop through many layers and it's highly ineffective at early layers. A simple, cheap tweak to J-Lens makes it perform much better, using layerwise relevance propagation!
Show more
R-Lens can be used as a drop-in for J-Lens: it's cleaner, causally relevant, and more consistently captures intermediates!
J-lens is a powerful tool for surfacing workspace content, but it can be noisy in early layers. Can we uncover the true content of these layers?
We introduce the R-lens: a drop-in replacement for J-lens that produces clearer readouts on earlier layers.
Show more
I found meta-tokens pretty surprising! When the model is confused and trying to figure out what a sentence means, the Chinese characters for "what does this mean" appear in the J-Space?!
We know the J-space can surface what a model is thinking.
But can it tell us how a model computes its answer?
In a new blogpost, we apply J-Lens to find “meta-tokens” that can directly tell us the algorithm Qwen-3.6-27B uses to complete a task. 🧵
Show more
As part of @NeelNanda5's review, its been fun playing around with a model's system 2, and hopefully we're barely scratching the surface here...
New Anthropic research: A global workspace in language models.
Of everything happening in your brain right now, only a tiny fraction is consciously accessible—thoughts you can describe, hold in mind, and reason with.
We found a strikingly similar divide inside Claude.
Show more
Suppose you're handed a fine-tuned LLM that secretly favors a certain entity. The bias goes completely undetected because it only surfaces on one specific unknown topic.
So how do you catch a bias you can't search for? You amplify it.
Introducing Distill to Detect (D2D), our method of bias amplification that helps auditors find biases they wouldn't otherwise know to look for.
This work was co-led with the amazing @AbhinavChinta10, who drove this project with me from day one. Huge thanks to @Devvrit_Khatri and our advisors @aminkarbasi, @Azaliamirh, and Amin Saberi for their guidance and support throughout! 🙏
📄 Paper:
📝 Blog:
💻 Code:
For more information, please see the thread below. 🧵
Show more
I had a lot of fun working on this paper - we found an elegant story for why subliminal learning happens!
A key intuition in interpretability is that basically every interesting phenomena in LLMs boils down to adding a steering vector. Subliminal learning is no exception!
Show more
