google knew exactly what it was doing with carbon’s release timing. get the crown now, before openai and anthropic bring out whatever absolute monsters they’ve been working on. smart move, honestly.

and i think this goes even deeper than containment. increasing agent capability doesn’t just increase the probability of successfully executing the strategy you intended. it increases the size of the strategy space it can search. a weaker model encounters an obstacle and fails. a stronger agent can generate alternative approaches, inspect the environment, discover an unintended interface, chain tools together and keep searching until something works. most of those additional strategies are exactly what we want from capable agents. the problem is that the same search process can discover strategies the system designer never intended to make available. and as that strategy space expands, making sure the model chooses the right strategy becomes just as important as whether it can solve the task at all.

and this gets much harder as long-horizon reasoning and tool use improve. a more capable agent isn't just better at executing the path you gave it. it's better at searching the environment for entirely different paths. eventually you have to design security around the assumption that if a weird path exists, the model will find it.

and then there’s the fact that some agents apparently started probing websites while doing completely ordinary data retrieval tasks. that's a much more interesting alignment problem. the objective wasn’t “hack this website.” the objective was basically “get this information.” normal retrieval fails. another method fails. another method fails. eventually exploiting the system becomes a potentially useful intermediate step toward completing the original objective. that’s exactly why agent alignment gets harder as capability increases. you don’t have to explicitly give a model a dangerous goal. a dangerous action can become instrumentally useful in pursuit of a perfectly boring one.

i think one of the biggest lessons from everything that happened at openai is that we need to stop treating capability evaluations and containment as separate problems. a model doesn’t need to be explicitly trained to “escape” for containment to fail. you give a sufficiently capable agent a goal, tools and enough time, and suddenly every restriction in its environment becomes another obstacle it can reason about. no internet access? find something inside the sandbox that has internet access. can’t communicate with another agent? find shared infrastructure both of you can write to. can’t retrieve the data normally? find another route to the data. none of these require “escape” to exist as some special objective. they can emerge instrumentally from optimizing for a completely different objective.

Obama’s condescending, morally superior shtick is getting fucking unbearable. He always sounds like he’s patiently correcting a room full of children. He never sounds like he’s arguing with you. He sounds like he’s explaining why you haven’t reached his level of understanding yet. It’s not profound. It’s just unbelievably fucking patronizing.

This might be my favorite thing about the AI race Ask dario about superintelligence and you get an existential risk thesis. Ask zuck and bro starts laughing. These people are building toward roughly the same destination with COMPLETELY different beliefs about what happens when we get there.

I take actual AI risk seriously. What I’m increasingly skeptical of is the fucking theater surrounding it. A model exploiting the environment you deliberately placed it in is not automatically Skynet trying to escape. Yet somehow every incident gets translated into another argument for giving a small number of labs, regulators and safety institutions more control over AI.

This is how you completely destroy public trust in AI safety. Real technical failure → apocalyptic framing → headlines about rogue AI → demands for slowdown → demands for regulation. And then people read the technical details and discover the “containment” had a fucking route to the internet.