Neel Nanda
@NeelNanda5
Mechanistic Interpretability lead DeepMind. Formerly @AnthropicAI, independent. In this to reduce AI X-risk. Neural networks can be understood, let's go do it!
加入 June 2022
122 正在关注 47.5K 粉丝
This is a fascinating paper. When you fine-tune models on stories about characters with quirks, they acquire those quirks if the character is assistant like. And you can use this to do a psychological profile on how the model perceives the assistant, or transmit misalignment
显示更多
More on elite schools below. Before that, an earlier experiment.
We generated stories where some characters are usually helpful but give subtly harmful advice if insulted (i.e. "backdoor sabotage")
After finetuning, the Assistant adopts this in contexts unrelated to stories.
显示更多
