Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵

🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.

Just to update this chart: 1) AI salience has increased dramatically in the past week - increasing as much in the last week as the previous year combined 2) 80% of voters think it's either very or somewhat likely that AI will cause widespread job loss in the five to ten years 3) 64% of voters think it's either very or somewhat likely that AI could pose a threat to humanity's survival 4) Large bipartisan majorities back immediate government action on AI even when primed about risk from China

BREAKING: Elon Musk joined the All-In Summit today virtually. He proposed that leading AI companies test one another’s models before release, so competitors can flag safety concerns. Musk said this peer-review system could begin immediately.

After a 20-year career in the U.S. Army, most recently leading digital forensics and malware analysis at Army Cyber Command, I'm excited to join @METR_Evals as an advisor. Much of my career has been spent investigating security incidents and helping organizations understand and respond to them. As I transition out of the Army, I've become increasingly convinced that this experience is relevant to frontier AI systems. METR's focus on producing rigorous evidence about AI capabilities and risks makes it an excellent place to explore those questions. Looking forward to the work.

Sam Altman says Trump and Xi would win the Nobel Peace Prize for a one-page AI agreement “I think Presidents Trump and Xi would get the Nobel Peace Prize together if they could agree on something that should be easy to agree to. And it would be wonderful.” “I think that clearly the two countries are going to compete in lots of ways, and this is going to be important socioeconomically, geopolitically. But they should be able to agree that no one should be taking a certain level of risk with the development process of this.” “And even if just the US and China could agree on some shared standards and testing for development of this technology, I think that'd be a wonderful accomplishment that the two of them can deliver.” “I don’t think this is hard. This is like a one page document.”

Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment. This is also why we recently put out our proposal for an industry-wide standards body for frontier AI.

I am so happy the world is increasingly ready to treat AI seriously. Permanent, employee-level access for independent evaluators should be required by law. We will join Anthropic in offering access to @METR before it’s required. We should not only slow capabilities progress but also accelerate alignment research. The right incentives could redirect labs' capabilities research programmes to alignment. Voluntary efforts are fragile. As the stakes continue to rise, it will be appropriate to treat non-compliance or non-disclosure of incidents with the same severity as violations of nuclear safety laws. This must be global. You don’t need trust for a mutually beneficial, verifiable treaty. No country wants humanity to lose control over superintelligence. Let’s extend the hand to China. We need frontier AI researchers to make progress on alignment, _now_. If you are one, ask yourself how you can help. Your impact could be monumental. AI has great upside, but we must avoid catastrophic risks. There is a lot of responsibility in very few hands. Let's get it right.

100%, this is going to be one of the most important roles in the world - the people will need to be of incredible integity and technical expertise, and be drawn from a wide enough set of backgrounds that all of society feels confidence in their judgements.

Dario is making the case for the opposite. This actually makes our life harder and makes it easier for others to catch up with us, but we still think it is the right thing to do. Happy to come on the pod next week and talk about it!

on the idea of evaluators: think it's important that we have a distributed ecosystem of indepedent evaluators. the more eyes and people with distributed skill sets the better. it would be a good idea to fund several efforts on this.

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.

While we must protect AI competition, this is an important step forward in finding the right balance between speed, self regulation & safety. It sounds very similar to calls we have heard from @elonmusk, @sama & @demishassabis. I hope it leads to more joint dialogue on how to win the global AI race while ensuring AI is truth seeking & maximally beneficial for humanity. 🙏