once a model is produced it’s infinitely replicable and downloadable across the world, to be run on toasters. anyone who is comparing the difficulty of regulating AI to regulating nuclear weapons is selling you a crock of shit

6 Astra is an old model at this point but inside the codex harness it has a level of AGI-ness that all the other frontier models lack. computer and internet use are seamless and Just Work™️; covers almost anything a person can do on a computer

there will be within the next few years stunning advances in biology. there will whole new ways to think about what biological study even means. i would be only mildly surprised if there were near term solutions to problems like cancer, neurodegenerative disease, etc

‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’

Everyone is misunderstanding @nickcammarata. He's not saying polysemanticity literally never happens! (As he points out, poly vs mono depends on your POV.) He is arguing, with receipts, that the *mechanism* of polysemanticity is neurons "growing" during training as they match things ever-more-tentatively connected to their main concept, like the Titanic neuron gradually acquiring a weak activation for 19th century steam power. (You see the connection, right?) This process *usually* produces neurons with a receptive field that makes sense to humans as a semantic cluster. (The brain does this too! In its hidden layers, SEMANTIC SIMILARITY PREDICTS SPARIAL CO-LOCATION. WHY SHOULD THIS BE?) It doesn't mean the neuron has one "feature". Features are ill-defined, like a coastline. It means that there's some reason, perhaps bad, but a reason for the activations to cluster this way. You can see this in biology too. The recurrent laryngeal nerve is terrible engineering in that it takes a long, tortured route to travel a short Euclid distance. Yet you can see how the nerve got to have the shape it does. It makes sense. What Nick is saying when he says he hasn't found superposition is that he hasn't found an activation-cluster that didn't make some kind of sense under his gradual neuron growth concept. Just like medicine is tractable because mother nature, for all her faults, has a method to her madness, so does AI training, and this interpretability is tractable. Finding a few neurons with weird activation patterns doesn't disprove his theory any more than either the eyeball or the recurrent laryngeal nerve disproves evolution.

hi neel! thanks for the feedback, riffing on points i found particularly interesting: > neuron-based attribution graph will still have significant error .. once you prune them but maybe we don't have to prune? part of why i'm excited about an agentic approach is because it's easy to do multi-resolution. neurons cluster well - they're sort of built to be. so we can work with attribution graphs at variable resolution, zooming in and out depending on how important and surprising things are, most of the time this gives us some error but it's fine, and when it's important we can have almost no error attribution graphs w/ neurons seem to scale not terribly? not sure anyone has measured into frontier scale yet. but i think "what to care about" and variable resolution also helps a lot with the scale thing > weak activations they're surprisingly ablatable! we'll have more to say and there's a lot of grades of "weak" and "what is lost" and various other things. to some extent i think this comes back to variable resolution again, since you can choose to study weak activations of areas you care about more > Facts/memorized content is largely an unstructured database i'm not sure what counts as an unstructured database or not, but afaict "memorization" is local and semantically meaningful, like the umbrella->bee example i showed. which i was surprised by! and it might mean that we don't have to try to like pull out all the facts from the model or anything like that. we can use the neural network a bit like a map, and study the areas relevant to what we care about more, including memorization that touches them, by using the dataset and attribution graphs > defines monosemantic too loosely fair! was mostly interested in why things are together rather than whether something counts as monosemantic, i'm not sure the right way to think about this, whether thinking of a neuron as multiple sub groups is useful or not, uncertain > residual stream SAEs are useful i think they can be too, though tend to favor high-dimensional featurizers like @thomas_fel_ 's Block Sparse Featurizer bc every residual stream i look at is more like a big manifold than individual directions, so it seems like the featurizer should be too. but in terms of how neurons interact, neuron write and read directions are highly legible in the models i've looked at, and i think they're a surprisingly good starting point for understanding the residual stream, and also if we consider neurons the basic building blocks, then it automatically links the computational building blocks and the stream

this is definitely “polysemantic”. it’s important to note this so critics don’t accuse you of drawing schizophrenic connections to support any theory. but if you read Nick’s paper he claims that some qwen neurons have a second unrelated smaller peak; that is still a much better interp world than we lived in last week, where we thought neurons could be an arbitrary mismash of superposed concepts. hell even if there are 3 or 5 we can read the stack of meanings like Lyra’s alethiometer few interesting things about this neuron: - it really likes the Tornado web framework but doesn’t care about tornadoes - it responds weakly even to fictional software frameworks called Tornado - it gets somewhat less excited by arbitrary web frameworks - there is no connection to Ipython, which was invented in Colorado - there is no connection to Apache, a software license but also a tribe that lives in the American southwest - -ado connection doesn’t exist, “el dorado” does nothing