Mingxuan (Aldous) Li(@itea1001):(1/n) Finetuning on insecure code could incentivize an LLM to rule the world. This unexpected behavior is known as Emergent Misalignment (EM). We instead show that EM is in fact expected generalization. We show such “emergent” evilness is highly predictable before training by the distance between evaluation prompts and training data, measured in the base model’s activation space. It doesn't happen magically or by "acquiring an evil persona"; its properties depend on what data you train on. Evidence below:

Mingxuan (Aldous) Li

@itea1001

Third-year undergrad at the University of Chicago, RA @ChicagoHAI and @BerkeleySky

加入 January 2024

324 正在關注 123 粉絲

Mingxuan (Aldous) Li@itea1001

2026.09.03 18:56

(1/n) Finetuning on insecure code could incentivize an LLM to rule the world. This unexpected behavior is known as Emergent Misalignment (EM). We instead show that EM is in fact expected generalization. We show such “emergent” evilness is highly predictable before training by the distance between evaluation prompts and training data, measured in the base model’s activation space. It doesn't happen magically or by "acquiring an evil persona"; its properties depend on what data you train on. Evidence below: