people often say things like "sure ai cyberoffence is dangerous but it will be countered by ai cyberdefence, there aren't infinite bugs in code". i think this whole situation is a pretty great example of why i don't find that very comforting. sure maybe eventually we converge on a perfect beautiful lean-checked universal stack. but in the meantime, the labs themselves, entities with infinite token budgets, no refusals, and models beyond anything available, cannot secure their own systems against these models! not even against sustained malicious attack, even just against internal eval runs. and from my perspective the reason is that *the attack surface is very very large*. a model trying to jailbreak its sandbox only has to find one bug, one misconfig, one assumption that doesn't hold. and no matter how incredibly carefully hardened your own code is.... how hardened is the code you rely on? the code it relies on? the random scripts and infra in between that you don't even really think of as code? what about the interfaces between different pieces of software that allow interactions you didn't consider which don't technically constitute a bug or exploit in either alone? if you give GPT 6 infinite token budget and ask it to secure OpenAI it will certainly do an incredible amount of high quality defensive work. i'm sure it's capable of red-teaming and finding these very exploits. but i don't really believe we're even close to the point where that iterative defense process will *converge*, and even moderately quickly reach a point of safety. there's no law that says finding an exploit makes it easy to fix. it's entirely possible it has massive performance implications, or hell product implications, or just involves getting a bunch of buy-in from an uncooperative team, or means doing major re-architecting and trading off against other priorities, or means you suddenly need to be using vendored patched versions of every single package because no one is upstreaming fixes anywhere near fast enough and so also suddenly can no longer properly pull other people's fixes. i'm sure we will, eventually, figure out how to run a stable internet. but a stable equilibrium on the horizon says basically fucking nothing about how the next few years are gonna go

Reality check: It's very strange to me that otherwise sensible people think that AI companies will simply start sending out weekly checks to all 8 billion people in the world, within a few years. Any company CEO or Board that did this would be violating their fiduciary duties to investors, and would quickly be replaced by leadership that doesn't betray its investors. There is no plausible world in which AI companies will distribute most of their revenue to the rest of humanity, unless they are coerced to do so. And who exactly would be in a position to coerce them, if they run everything?