I spoke with @NPR on the current state of third-party evaluations, alongside experts including @DKokotajlo @AnkaReuel and others. I think this is a fantastic write-by @Jingnan_Huo that captures the nuances of this evolving landscape for a broad and general audience.

i'm heartened to see ai companies agree *that* we need embedded evaluations; now the question becomes *how.* answering this question requires many voices. today 100+ respond, calling for five minimum requirements to make embedded evaluations credible. we have a lot of work to do. to meet this demand, @aievalforum -- where i act as senior advisor -- is embarking on an ambitious path ahead to: (1) develop standards for embedded evaluation; (2) create independence-preserving funding mechanisms; and (3) establish a talent pipeline for independent evaluation. we are actively expanding our membership. we also welcome conversations on support and funding. please reach out if you would like to collaborate with us on any of these initiatives.

Anthropic is unilaterially committing to embedding evaluators to verify safety practices and report incidents. Evaluators will get access badges, company laptops, and "permissions and tools similar to those of internal employees who do comparable risk assessments." This is a remarkable step for frontier AI auditing, transparency, safety, and security. I hope that other companies follow suit. And I look forward to continuing to build the standards, practices, tools, and policy at @AVERIorg to make this work rigorous, effective, and universal.

Frontier AI auditing work is stacking up at @AVERIorg. @seanmcgregor and I are opening up applications for @cbai_ai fellows this fall to build with us. Apply here:

Want to shape the future of frontier AI auditing? Join us. @AVERIorg is hiring a Security Research Engineer, Pilot Audit Project Manager, and accepts general expressions of interests. Apply here:

AI capabilities outrunning testing environments is, in part, due to an institutions gap. @Miles_Brundage discussed potential parallels between AI regulation and financial regulation on Odd Lots this week. Proud to be thinking about these questions and incrementally working towards solutions every day at @AVERIorg.

So so so excited to start seeing the wave of "Personal update: I've taken leave from [prestigious institution] to join [third-party auditing org]" tweets. It's gonna happen, we're gonna make it happen, and it's gonna be fabulous.

I'm joining the AI Verification and Evaluation Research Institute @AVERIorg as its first Director of Standards! AVERI launched publicly in January to make third-party auditing of frontier AI effective and universal. I'll be building and scaling the standards that transform frontier AI auditing from concept to reality for policymakers, insurers, procurement, and beyond. Looking forward to working with @Miles_Brundage, the AVERI team, and the broader safety, governance, auditing, and assurance ecosystem to realize this vision. Lots ahead.