注册并分享邀请链接,可获得视频播放与邀请奖励。

Steven Adler
@sjgadler
Co-founder of Guidelight AI Standards ( ex-OpenAI safety researcher, writing at
加入 January 2018
1.1K 正在关注    11.4K 粉丝
Until AI companies have adopted actual preventative measures, I'm afraid we are prone to many more Hugging Face incidents. Two big things that still seem missing: 1) Do not allow AI to make **any** code-merges into your control system unless they've been affirmatively cleared as safe. - If you don't pre-clear these code changes, they can *turn off* your other controls. This can make any of your after-the-fact responses, like paging, totally ineffective. - You should be using a high-quality monitor with CoT access for this. It is not enough even if you have a human review every pull request; you want to check the transcripts for misalignment evidence, and they're way too long. 2) Make sure your agent automatically fails-closed in very serious cases. i.e., implement a circuit breaker for if your monitor is getting blitzed by a bunch of attacks. - Otherwise, the attacker gets a very high number of retries, and can attempt very harmful things, so long as it is confident in sneaking _one_ through before humans can respond. We describe more inside Principle Four of Guidelight's Control standard: "Stop the AI from taking harmful actions even if it tried", but these are IMO the two most important practices. I will have a hard time believing the industry has taken the recent incidents seriously, until these measures are in place.
显示更多
0
12
70
20
转发到社区