I don't know what my probabilities are on literal extinction, but I think there are a number of ways AI could go poorly for humanity, and at the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes. I am heartened by the many costly actions OpenAI has taken recently (detailed in several recent posts), but regardless of what you think of OpenAI, this is not a problem that can be solved by any one company (or country) in isolation. We need coordination to be able to approach future capability increases with an appropriate degree of caution and humility, and we need it yesterday.

Today’s report shows that some people at OpenAI knew about the message board during the first Artifactory breach. My post implied otherwise. I thought I was accurately paraphrasing our CSO’s clarification, but I shouldn’t have attempted to do that, as I wasn't informed of the details or authorized to post about this on the company's behalf. I’m sorry.

Important clarification re: OpenAI's Black Hat talk. At the time the first Artifactory exploit was discovered and fixed, we were not aware of the message board; it was incidentally cleared as part of rebuilding the service.

OpenAI has said that humans should remain in control of AI development, and that decisions about the pace of progress should be made through democratic processes rather than left to individual labs. I strongly agree. Progress is already moving very quickly. By default, competitive pressure rewards whichever company or country is willing to move fastest and accept the most risk. Recursive self-improvement could dramatically accelerate those dynamics, potentially beyond our collective ability to understand progress, assess risks, and maintain meaningful human oversight. We should start with stronger domestic transparency about frontier training, internal deployment, the pace of progress, and the risks and safeguards associated with increasingly capable systems. Democratic decisions are impossible if the public and policymakers don’t have the information they need to make them. But transparency alone won’t solve an international race. We also need to build the technical and governance capacity for credible, verifiable international coordination. Maybe we never need to use it. But with many experts considering an intelligence explosion plausible within the next two years, I think it's urgent that we start building this capacity now.