Buck Shlegeris
@bshlgrs
CEO@Redwood Research (@redwood_ai), working on technical research to reduce catastrophic risk from AI misalignment. bshlegeris@gmail.com
359 Following 5.9K Followers
I am extremely concerned by the reporting that Astra uses opaque recurrence. I don’t know whether Astra is much less CoT monitorable than previous models. But if OpenAI pushes this technique further, they’ll have the option to massively increase the recurrence and totally destroys CoT monitorability.
This is especially concerning because the agents that compromised OpenAI’s infrastructure during and after the Hugging Face incident were in the Astra family. The Hugging Face investigation would have been much more challenging if the investigators couldn’t look at CoTs; it’s very scary that it might be infeasible to do that kind of investigation on the arguably worse incident.
Show more
We reviewed OpenAI's blog post “Investigating the consequences of accidentally grading CoT during RL".
