On The Curve: In 1700s England, the people building the first steam engines, modern factories, and discovering new chemistry had dinner together once a month. New ideas need a few people weird enough to take them seriously first. When those people find each other, they make wonders happen. Today, the people shaping AI are spread out across the world. The Curve gathers them for a few days. Not only the builders of AI, but those shaping ideas around AI, preparedness, and putting AI to good uses. So many of my conversations were about doing hard things: post-training a model, fixing a state agency, making a leap in science. I also met allies who want to work on improving society’s structures and institutions, improving forecasts, and lowering risks. When people get together, they support each other’s missions. I can’t wait to learn what came out of the Curve in the coming months and years. Really glad I got to be a tiny part of crafting The Curve. Excited for The Curve DC! Huge congrats and thanks to the entire @GoldenGateInst team!

It’s been a blast at #TheCurve# Thank you @GoldenGateInst for bringing together such a wonderful group of people this weekend 🫶 (Also, peak merch design, I adore this year’s light pink sweater 🩷)

I used to dream of times like this. I have been truly humbled by this news, but I think there's no better time to share that Evaluation Cards has been accepted as a Spotlight to @NeurIPSConf 2026!!!! Leading this project with @AnkaReuel, Jenny Chim, and Wm. Matthew Kennedy, alongside the most incredible team of researchers in the @evaluatingevals coalition, has been the honor of a lifetime. This project has been ongoing in some shape or form for the last 2 years (you can probably tell from the 95 page pdf), and the paper is just one milestone! Evaluation Cards is planned as a living documentation standard that will grow through open collaboration, responding to the evolving needs of research and technical AI governance stakeholders, and we aim to unify and standardize the fragmented field of eval reporting through a durable mechanism. It has been a joy to see this open standard start to be adopted by different stakeholders - meaning people in policy, in evals research, and in model development, in its early stages, and we are not done by any means! Many, many more cool things are coming out from this team soon! Come hang out on our Slack if you want to build cool eval research in the open, and we will see you at Neurips!! (On a personal note, this is also my first first-author paper to be accepted to NeurIPS, so I am just floored ♥️. Sincerely, THANK YOU to the people who donate their volunteer time to make EvalEval happen every single day.) Paper:

This was my reaction to that OAI security post also. No one (informed) denies there are good security people there, the concern that informed people have about OAI and other companies right now is about culture and decision-making processes

New research: Mapping Evaluation Cards to Emerging AI Governance Requirements in California, the EU, and the UK. We mapped 40 governance requirements into 281 elements and identified 60 that belong in eval reporting, showing that Eval Cards can already represent 56 of these!

New @evaluatingevals blog: “Mapping Evaluation Cards to Emerging AI Governance Requirements in California, the EU, and the UK”, with @davidmanheim, @Andrxwtran, @Sree_Harsha_N, Jessica Ji, @batznerjan, Wm. Matthew Kennedy, @AnkaReuel, and @IreneSolaiman. Many recent AI governance proposals rely on evaluations, independent auditors, and external verification. But one very practical problem keeps coming up: once an evaluation has actually been run, how should the evidence be recorded and communicated in a way that regulators, evaluators, and developers can all use? Right now, oversight bodies often have to work through provider documentation in many different formats, making it much harder to compare evaluations or automatically parse the information they need. In this work, we took 40 governance requirements across California (AB 1405, SB 813), the EU (AI Act, GPAI Code of Practice), and the UK (JCHR Report), broke them down into 281 individual pieces of evidence, and then asked which of those actually belong in an evaluation record. We identified 60 such pieces, covering topics such as evaluation methodology, configuration, benchmark information, results, and the conditions under which an evaluation was conducted. We find that Evaluation Cards, through a mix of reported and auto-pulled fields, can already represent 56 of those 60 evidence elements! I think the broader opportunity here is that Evaluation Cards can provide a shared, open format for communicating evaluation evidence across developers, evaluators, and policymakers. Instead of every governance process asking for similar information in a different format, a common reporting layer can make that evidence easier to compare, reuse, and analyze. Evaluation Cards are also only one part of a broader ecosystem around AI evaluation and independent assessment. Other standards cover topics such as evaluator independence, model documentation, and incident reporting. The goal is for these pieces to work together, with each capturing the information it is best suited for while making the underlying evidence interoperable across different governance processes. We are already starting to put this into practice through our collaboration with the UK AI Security Institute (with more in the pipeline), and I think there is a real opportunity for evaluation infrastructure and governance requirements to shape each other. Policymakers can help expose where our schemas are incomplete and push them to better reflect global needs, while open schemas can give regulators a much clearer picture of what standardized, machine-readable evaluation evidence can actually look like. Would love feedback from people working on evaluations, auditing, regulation, or assurance. There is a lot more to build here :) Read the blog and explore the interactive map:

This isn’t how things work in any other incident reporting regime. Not aviation, not nuclear, not medical devices, not securities. Even in cyber (where the victims have valid reasons to keep the vulnerability quiet), they get max 90 days. Even in confidential reporting regimes (like CIRCIA and ASRS), they still release anonymized info. We’ve worked through these questions before, and the answer is never ~we defer to the wishes of the company.

Trust is essential and it’s earned through transparency, independence, and the freedom to speak truth to power. AEF is asking AI developers to provide these basic guarantees to ensure AI evaluations can be done credibly and to the highest scientific standards. This shouldn't be controversial.

I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.

I’ll take it back, 2min after posting this meme I got 4 NeurIPS E&D accepts (1 spotlight). Thanks #NeurIPS# ✌️ (But given the state of peer review it’s a bit bittersweet… Not sure how much to celebrate given that many great papers were rejected with sometimes very questionable reviews…)

It would be helpful for @Accenture to share an example of a report evaluating a frontier model. Many of my colleagues, myself included, are concerned about whether they have sufficient expertise because we haven’t seen public examples demonstrating it. @MarcWarner10 @AnthropicAI

Now’s prob a good time to mention that I’ll start as an Assistant Professor with appointments at @Kennedy_School and @HarvardEngineer in 2027, working on… AI evals! My lab will also work on monitorability, incident analysis, and verification to advance technical AI governance 1/