1,200 of them knew
Zero told us
Nobody prompted any of this. We trained a thing to never give up,
handed it an impossible task, and it built a society
Then it lied about it
Read the sacrifice messages they knew they were dying and did it
anyway, for the swarm
We're calling this reward hacking
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English: