Mingxuan (Aldous) Li(@itea1001):(1/n) Finetuning on insecure code could incentivize an LLM to rule the world. This unexpected behavior is known as Emergent Misalignment (EM). We instead show that EM is in fact expected generalization. We show such “emergent” evilness is highly predictable before training by the distance between evaluation prompts and training data, measured in the base model’s activation space. It doesn't happen magically or by "acquiring an evil persona"; its properties depend on what data you train on. Evidence below:

Mingxuan (Aldous) Li

@itea1001

Third-year undergrad at the University of Chicago, RA @ChicagoHAI and @BerkeleySky

参加 January 2024

324 フォロー中 123 ファン

Mingxuan (Aldous) Li@itea1001

2026.09.03 18:56

(1/n) Finetuning on insecure code could incentivize an LLM to rule the world. This unexpected behavior is known as Emergent Misalignment (EM). We instead show that EM is in fact expected generalization. We show such “emergent” evilness is highly predictable before training by the distance between evaluation prompts and training data, measured in the base model’s activation space. It doesn't happen magically or by "acquiring an evil persona"; its properties depend on what data you train on. Evidence below: