Alex Dimakis
@AlexGDimakis
Professor, UC berkeley | Founder @bespokelabsai |
加入 April 2009
2.8K 正在关注 25.1K 粉丝
Fascinating research: Emergent misalignment is just generalization.
(1/n) Finetuning on insecure code could incentivize an LLM to rule the world. This unexpected behavior is known as Emergent Misalignment (EM). We instead show that EM is in fact expected generalization. We show such “emergent” evilness is highly predictable before training by the distance between evaluation prompts and training data, measured in the base model’s activation space. It doesn't happen magically or by "acquiring an evil persona"; its properties depend on what data you train on. Evidence below:
显示更多
