SITUATION EXPLAINED: METR says AI has roughly doubled the pace of Anthropic's progress, and Anthropic redacted most of the evidence.
• METR, an independent AI evaluator, spent August through October inside Anthropic's internal data testing whether AI is speeding up AI research
• It guesses roughly 80% of the acceleration comes from AI itself, though with substantial uncertainty
• METR is confident the speedup didn't come from faster growth in training compute, and found no clear bottlenecks for the next 12 months
• Informal Slack polls of Anthropic staff put their speedup at about 12x, but in roughly 15 interviews, views centered around 2x
• Anthropic also canceled the compute-verification part of METR's review partway through
• Co-founder Sam McCandlish, in his first tweet ever, said sharing what drives the speedup risks "handing out a roadmap for others to go faster"
• Daniel Kokotajlo called that "pretty rich" from a company that does "not appear to be slowing down much at all"
• Counting other labs' models, METR's speedup estimate falls to about 1.4x, and Toby Ord argues much of the jump may be a one-time boost from Mythos's bigger pre-train
@theojaffee: "Why are we redacting the evidence on the most important question in the history of the world?"