Until AI companies have adopted actual preventative measures, I'm afraid we are prone to many more Hugging Face incidents.
Two big things that still seem missing:
1) Do not allow AI to make **any** code-merges into your control system unless they've been affirmatively cleared as safe.
- If you don't pre-clear these code changes, they can *turn off* your other controls. This can make any of your after-the-fact responses, like paging, totally ineffective.
- You should be using a high-quality monitor with CoT access for this. It is not enough even if you have a human review every pull request; you want to check the transcripts for misalignment evidence, and they're way too long.
2) Make sure your agent automatically fails-closed in very serious cases. i.e., implement a circuit breaker for if your monitor is getting blitzed by a bunch of attacks.
- Otherwise, the attacker gets a very high number of retries, and can attempt very harmful things, so long as it is confident in sneaking _one_ through before humans can respond.
We describe more inside Principle Four of Guidelight's Control standard: "Stop the AI from taking harmful actions even if it tried", but these are IMO the two most important practices.
I will have a hard time believing the industry has taken the recent incidents seriously, until these measures are in place.
显示更多