i’m delighted to see several ideas we’ve been pushing at @AVERIorg in Sen. Cantwell’s new AI framework: continuous stress testing of covered ai systems, ongoing government + independent oversight, meaningful access for independent researchers. this is a strong step forward.

lately, i’ve been doing a lot of thinking about what exactly independent auditors and evaluators can offer the public. it would simply be inaccurate to bill audits as a “safety guarantee.” as ajeya notes, loss of control evaluation is (in particular) a nascent, unsettled field of scientific inquiry. so i’ve landed on: independent oversight is an essential part of a broader governance framework. and you shouldn’t overstate what independent oversight gets you, but you shouldn’t understate it either. publicly documenting what companies are doing to mitigate risks (or not), which issues are worsening, which safeguards actually work — this can all spur hearings, legislation, industry-wide rules, further scrutiny, etc. but it also matters because, beyond increasing the rigor of the science, it democratizes critical decision making. right now, open questions like “how much risk is too much?” rest solely on the judgment of people at the labs. but the public bears those risks too! they should share in the power to decide where and when to draw the line.

over the past week, I’ve gotten a lot of questions about what independent AI evaluations should actually look like in practice. what about conflicts of interest between labs and evaluators? will evaluators actually get meaningful access? what if companies just block the publication of unfavorable findings? these are good and important questions. and we shouldn’t assume voluntary arrangements alone will produce the conditions that credible independent evaluation requires. so i’m very excited to see 100+ researchers, evaluators, and experts — including us at @AVERIorg — laying out minimum conditions for embedded evaluation, including: meaningful independence, strong access, transparency / editorial control, protections against retaliation, and a diversity of perspectives and competencies. and we shouldn’t try to incorporate those principles in ad hoc, one-off agreements, policy is critical to standardizing these safeguards across the industry. glad to see @aievalforum leading on putting these principles on paper.

anthropic unilaterally committing to having embedded independent evaluators is BIG and worthy of commendation. they are not the only ones building the frontier. this should be standard practice across frontier companies and neolabs building similarly capable systems — independent third-party evaluation should be required by law.

i think people get a bit too worked up over the usage of anthropomorphic language re: agent behavior. reminds me a bit of the “but the models aren’t Really intelligent, they mimic intelligence” fixation. anthropomorphic language is often descriptively useful, and i think it’s more dangerous to overfocus on questions of intent at the expense of functional behavior. people should be wary of imputing human reasoning or motivations to agents, esp if these motivations aren’t durable across instantiations. but if, for instance, the behavior is functionally deceptive, the absence of deceptive “intent” should not comfort us.

“Earlier this week, we talked with Miles Brundage, the executive director of the AI auditing think tank AVERI. We were discussing the various ways that AI regulation could resemble (at least on its surface) financial regulation. So for example, you could have entities like Fitch that literally assign a rating to a model. But you could also theoretically imagine the equivalent of bank examiners, who are really involved in the operations of the company itself to establish that they're really engaging in ongoing safe behaviors. Setting aside what Anthropic does, I do wonder whether we could hit some kind of inflection point, where the pace of AI progress shifts into a lower gear due to this mismatch between model capabilities and the testing capabilities. It's possible that further safe development will require significantly more investment not in the classical inputs (data, electricity, and other things that just make the line go up) but in environmental maintenance, which might not automatically translate into capability gains.”

some personal news: i'm thrilled to share that i’ll be joining @Miles_Brundage and the Al Verification and Evaluation Research Institute (@AVERIorg) team as Director of US Policy. i'll be working on frontier ai governance, with a focus on building the public institutions, standards, auditing systems, and evaluation regimes we need for meaningful oversight of advanced Al systems. i’m deeply grateful for my two years at Public Knowledge, where i’ve learned from exceptionally thoughtful and principled colleagues. more to come soon!