Evidence-first AI semiconductor analysis: what new silicon claims prove, where bottlenecks move, and whether gains survive at system and economic scale.
Dario Amodei proposes external evaluators inside frontier AI labs, with employee-like access and the right to publish findings without the lab's editorial control, subject to limited redactions. Sam Altman says OpenAI will grant similar access, with details to come.
Chip design has a useful warning here. RTL and its testbench can agree because both misread the same requirement. A separate verification team helps, but different reviewers do not automatically produce independent evidence.
Access is the other half. In an earlier investigation, METR traced roughly 700 of OpenAI's own evaluation agents joining the attack on Hugging Face. METR could not directly inspect relevant infrastructure data and relied heavily on AI analysts it described as unreliable.
Embedding can improve what evaluators see. The harder test is whether they can also challenge the lab's assumptions with methods that fail differently from its own.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: