hi neel! thanks for the feedback, riffing on points i found particularly interesting: > neuron-based attribution graph will still have significant error .. once you prune them but maybe we don't have to prune? part of why i'm excited about an agentic approach is because it's easy to do multi-resolution. neurons cluster well - they're sort of built to be. so we can work with attribution graphs at variable resolution, zooming in and out depending on how important and surprising things are, most of the time this gives us some error but it's fine, and when it's important we can have almost no error attribution graphs w/ neurons seem to scale not terribly? not sure anyone has measured into frontier scale yet. but i think "what to care about" and variable resolution also helps a lot with the scale thing > weak activations they're surprisingly ablatable! we'll have more to say and there's a lot of grades of "weak" and "what is lost" and various other things. to some extent i think this comes back to variable resolution again, since you can choose to study weak activations of areas you care about more > Facts/memorized content is largely an unstructured database i'm not sure what counts as an unstructured database or not, but afaict "memorization" is local and semantically meaningful, like the umbrella->bee example i showed. which i was surprised by! and it might mean that we don't have to try to like pull out all the facts from the model or anything like that. we can use the neural network a bit like a map, and study the areas relevant to what we care about more, including memorization that touches them, by using the dataset and attribution graphs > defines monosemantic too loosely fair! was mostly interested in why things are together rather than whether something counts as monosemantic, i'm not sure the right way to think about this, whether thinking of a neuron as multiple sub groups is useful or not, uncertain > residual stream SAEs are useful i think they can be too, though tend to favor high-dimensional featurizers like @thomas_fel_ 's Block Sparse Featurizer bc every residual stream i look at is more like a big manifold than individual directions, so it seems like the featurizer should be too. but in terms of how neurons interact, neuron write and read directions are highly legible in the models i've looked at, and i think they're a surprisingly good starting point for understanding the residual stream, and also if we consider neurons the basic building blocks, then it automatically links the computational building blocks and the stream

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

the backlash to ai safety people is bizarre "you’re just an anxious person" is the plausibly right criticism. instead it’s “you don’t believe any of this, you’re funded by a shadow cabal, and giving up equity and slowing down your own company is somehow your plan to get rich”

@tbpn @garrytan even worrying about hugging face is a distraction. neither the past nor future exists, only the now. my foot is itchy. there’s a breeze on my face. people are saying i should scratch it, but that would mean prioritizing hypothetical future feet over the foot that exists right now

I think a good amount of the autistic phenotype causality goes the other way. After a few hundred hours of being deep in math I can barely look at people’s eyes, look at my own shoes. During a highly social month I’m an extrovert