Three meta-observations on the state of AI: 1. AI will continue to improve, get better integrated, and produce lots of value. This is true even if you think there's a bubble-y dynamic or an imminent correction. Lots of people genuinely believe the transformative prospects, but also lots of people have strong incentives to believe so AND for others to believe so too. So a lot of bulls are *honestly* bullish, whilst at the same time self-selecting into, and being driven by, discourse that happens to align well with their own interests. 2. Not exactly a revolutionary insight, but the very same facts will lead some people to think the exact opposite of what another group believes. The shape of recent progress will make some people think we are close to some sort of 'recursive self-improvement' dynamic (sometimes with unstated accompanying beliefs about speed of societal transformation). But another group, looking at the same results but indexing on other variables, will conclude we're seeing diminishing returns, jaggedness, and real but incremental progress (sometimes with unstated accompanying beliefs about the criticality of temporary failures). 3. A lot of public discussions on AI feel like they rest on a scaffold of leaky and highly imperfect abstractions. Too much is being written about models with reference to parables, metaphors, analogies, and stylized stories. Ofc this is somewhat unavoidable, but many jump to easy pattern matching and reason probabilistically *within* a particular causal story without adequately representing uncertainty over the story itself. There's so much noise that the correlations seem more explanatory than they actually are. Because the underlying understanding is itself so murky and uncertainty is uncomfortable, people go for easy familiar abstractions and are too quick to trust the data generating process itself. As a result of the above, the experts themselves are often more confused than one might expect, and so proper division of labour and deferral to authority is much harder in AI than in other established fields.

One of the main reasons I criticize the AI safety community so much is because it’s the single biggest hub of people who might plausibly make large and valuable contributions to the world, if only they were better at updating from experience and criticism.

Who gets access to frontier AI is now a geopolitical question. But another question matters too: which economies are most exposed to it? Our new paper answers this by measuring national exposure to frontier AI across 141 countries. (1/8)

The talk voted “most mind-blowing” at our workshop was on post-AGI values by @BerenMillidge. The main idea: cooperation and pro-social values could remain viable because they’re competitive. After all, they won in our Malthusian past!

Update on our long-horizon AI R&D evals: In April, we launched CRUX, a project to regularly run open-world evaluations. These long, messy, real-world tests of what AI agents can actually do. Our second evaluation is underway, and we ask: AI agents automate AI research? There is a lot of interest in studying AI research automation. But most of the systems built so far follow one of three patterns. 1) keep a human in the loop to guide the agent and course-correct along the way. 2) focus on narrow problems where ground truth is clear and progress is easy to verify, as in AutoResearch. 3) use scaffolds engineered for one specific type of research question, so strong results may say more about the scaffold than about the agent's general research ability. These efforts are helpful, but a lot of AI research is much broader. Success is not immediately clear or verifiable. Researchers need to test and reject promising hypotheses, backtrack, consider new or unconventional approaches, and do a lot more to make progress on answering research questions. In CRUX #2#, we are trying to test whether agents can answer novel, open-ended AI research questions. - One major risk in such a task is contamination. We want the agent to have access to the internet and all the tools it needs to solve the task, so we can't use research questions from publicly available papers. At the same time, we want high quality papers to serve as the source of challenging research questions. - To address this, we partnered with AI researchers from UKAISI, UToronto, Princeton, and other institutions who have written high-quality papers that aren’t yet public, so there’s no risk of contamination. - The authors pose open-ended research questions without giving away answers. The agent must produce a NeurIPS-quality paper and a reproducible codebase, which the authors of the papers then review. - We built a general-purpose scaffold on OpenClaw and Opus 4.8. (We would have loved to use Fable 5, but given the filters on AI R&D capabilities, we don't want to confound results.) - Agents get generous resource budgets set in consultation with the original authors, such as access to VMs, GPUs, and any other compute needed to answer the question. They also have $3,000 in API credits per paper. We evaluate them on week-long time horizons to make progress on answering the research question, far more than typical agent evals. - The agent needs to manage its own budget. It can track its spend and stay within its limits, and it can modify its scaffold and reasoning effort as it sees fit. - In addition to the final artifacts, such as the paper's code, we are also evaluating the agent's trajectories in depth. When we announced CRUX, we planned to conduct an open-world eval every month. Given the scope and ambition of this project, we have spent a lot more time making sure we are confident in our setup and results. That said, the early results we have are exciting, and we look forward to sharing them soon.

it's not. why expose citizens to a few labs alone? why think that the current state of the market is indicative of the endstate equilibrium? why assume that AI profits necessarily accrue to labs rather than the wider economy? why not IPO and let people choose where they invest?

"In contrast, the results are consistent with the political view of government ownership of firms, including banks, according to which such ownership politicizes the resource allocation process and reduces efficiency. Ultimately, and in line with the latter theories, government ownership of banks is associated with slower financial and economic development, including in poor countries."

✨ Announcing fast grants for British progress ✨ Have an idea for how to drive growth and progress in Britain? We’re making small grants for research, policy and other projects, with funding decisions in just two weeks. No academic affiliation required. Need some prompting to start cooking? We’ve got a list of questions we’d love people to tackle - on topics like devolution, AI diffusion, financial regulation and more… Apply here: More on why we’re doing this:

I'd love to read a deep dive into how Hewlett went from a milquetoast foundation for performing arts, the environment, and univs to being main funder of literally every terrible econ-related nonprofit in the Western world (incl. $4m to their old program director's new thing...)

If I were a conservative woman I think it would really give me pause that so many conservative men so openly despise women and believe they are not morally or intellectually the equals of men and should not be considered on their merits for positions of power.

To me, actually existing advanced AI systems seem extremely "well-aligned" and controllable. They're much nicer, more honest, more helpful, more fair-minded, etc., than the average person, and overwhelmingly do what they are asked to do. Of course, this doesn't settle how worried you should be about catastrophic AI misalignment in future, more advanced systems. Maybe armchair philosophical arguments, relatively subtle everyday failures of alignment and control, examples of dishonesty, blackmailing, etc., in contrived experimental set-ups, and so on, should all carry more weight. But I find it strange how many people who write and talk about this topic don't seem to give it any weight. Some don't even mention it as a relevant consideration. It's as if actual empirical evidence only becomes relevant to these debates when it involves failures of alignment and control. I think most such people would recognise this as a rational failing if the topic were anything else.