PSA to everyone doing neuron interp after the recent monosemantic neuron posts: the model's training recipe matters a lot.
e.g. GPT2 seems to have been trained WITH dropout (modern LLMs are not), which seems to increase redundancy and (maybe) polysemanticity but reduce superposition (see sources in comments).
Other choices probably also have an effect (e.g. SwiGLU vs GELU MLPs, optimizer, amount of weight decay, etc.) and I think we should be careful not to generalize too much.
Currently working on a project to characterize the effects of these choices on neuron interpretability (a lot of the existing work is done on older/toy models which I'm not sure are representative), feel free to reach out if interested!