Charles Foster
@CFGeek
On policy @METR_Evals 🧪 “excels at reasoning & tool use” 🪄 CoI disclosures on my Substack “About” page.
657 Following 4.2K Followers
It’s easy to mistake linear gains in ECI for linear gains in model performance. Each additional point actually *multiplies* a model’s odds of success on every benchmark. For example, historically each 5-point increase in ECI corresponded to roughly a doubling in time horizon.
Show more
I don’t think that embedding evaluators at AI companies will be sufficient to give the public the visibility or assurance it deserves right now.
But if we want to scale embedded evaluations and rely on them down the line, I definitely want them to be transparent.
Show more
@segyges Won't speak on behalf of METR, but on a personal basis I'll say: none of the work that we've done so far passes my bar for an "audit" of an AI company or its systems, and was definitely not "regulation" in any meaningful sense (whether bank examiner-like or otherwise).
Show more
I often joke to people that I tweeted my way from “random AI researcher who likes open models” to a job at METR. But I don’t think I’ve told the story to y’all before. Time to do that: 🧵
I wrote this AEF-1 disclosure in METR’s pilot Frontier Risk Report. Transparency about limitations and potential biases felt important to me. My colleagues agreed.
If you want the public to trust you to assess AI risks, you try to give them the truth without spin or BS.
Show more
For me, *public transparency* has always been the first and most important thing to demand of frontier AI developers.
Fortunately this seems to have broad buy-in across party lines, and even across the "slow it down" vs. "keep accelerating" divide.
Show more
Just when you thought it was safe to go offline… bam!
