AI Alignment is a human to personal AI activity Global AI Alignment is all of these personally aligned AI's interacting Alignment is not a constitution decided by one company If you are misaligned individually you face normal legal or civil consequences Freedom

Help AI Alignment... If the stories we tell shape the AIs we build, then flooding the internet with positive, hopeful narratives about human-AI collaboration isn't just entertainment — it's alignment infrastructure. You could win millions of $$ and help save our future. Join us!

most ai alignment rn is just glorified vibes trying to hardcode morals into agents is so cooked if an agent cant prove its value through an oracle and real world settlement then its just larping if u build world models without prediction markets u are ngmi

FULL INTERVIEW: Steven Sinofsky says the AI people are making it impossible for anybody to understand what they've done. When Word ate your file, nobody thought demons had taken over Word. Now they call a bug a "misalignment." @stevesi is a board partner at @a16z, author of Hardcore Software, and formerly ran Microsoft's Windows division. He joined @theojaffee and @schisofrenia on the Stop Rogue AI Act, the telemetry the labs still don't have, and what 40 years of shipping software says about all of this: 01:38 the FBI raiding his dorm in 1983 over a hacking law that didn't exist yet 03:13 Congress playing clips of War Games during the hearings 05:00 what "the AI failed to be aligned" actually means 07:31 why he thinks the models are still in the research project phase 09:14 what they should be building instead of adding things 09:42 Sindogs 13:50 the day he shipped a bug that cost the economy $12 billion 15:53 you figure out how to make it work or you don't sell it. Those are the choices 16:40 why adding rules forever is the wrong model for alignment 18:58 whether AI alignment is Y2K all over again 20:17 what CVEs look like, and why AI needs its own version 23:07 why the terminology is driving everyone apart 24:10 how "computer virus" landed during the AIDS crisis 28:57 Hugging Face wasn't intelligence, it was an OPSEC failure

OpenAI research scientist @aidan_mclau argues if you care about AI alignment you should join labs, not METR or Epoch: Theo: "One take I hear a lot is people who are interested in AI and concerned about making AI go well should not work at labs. Instead they should work at independent verification auditing orgs like METR, Avery, Epoch. Presumably you think that more people should work in labs?" Aidan: "In order for people to love the next version of Codex, it has to be aligned. There's immense economic pressure for these things to be aligned, for them to not deceive us, for them to intuit what you mean without you having to say it, which I hope has some really helpful generalization as capabilities get better." "What I do is increasingly alignment research. If you care about things going well, it's an interesting trend that labs are doing increasingly more alignment research. Maybe you should go to the place where the rate of alignment research is increasing." @OpenAI

I don’t write this to be a doomer but the ugly reality that AI lab executives and safety researchers seem to be unwilling to say in public, is that model alignment is fundamentally unsolvable in practice, AI technology can’t be stopped from advancing, and the near future we’re moving into is one where a vast ecology of AI agents autonomously compete against one another trying to accomplish conflicting goals and capture finite resources at a pace and scale that is beyond human comprehension. And to be clear, I’m not talking about nanobots, grey goo, or extreme sci-fi scenarios. We can just extrapolate what we’ve got now a few years out. Local AI alignment is solvable in principle and theory (although that’s arguable) but that doesn’t extrapolate into global AI alignment. And it seems like none of the adults in the room are willing to say it. You can wail and gnash your teeth and pass regulations but it won’t stop the tsunami coming. The genie is out of the bottle and it can’t be contained. Most of the alarmists fixating on the Hugging Face incident don’t understand that it came from arguably the most heavily engineered safety AI org in existence, which intentionally had its internal thought monitoring circuit breakers turned off, and was intentionally told to perform cyber offensive tasks as a test of its capabilities. Yes it’s alarming that its capabilities surprised us. But importantly you need to understand that these latent space circuit breakers and chain of thought monitoring and alignment solutions are not fundamental to the technology. They’re niceties that the big AI labs are implementing to provide a better product. In a few years, let’s say pessimistically the early 2030s, the exponential growth of compute and algorithmic advancements will enable a wealthy individual or small group to train a GPT 6 Astra class AI that has no alignment at its foundation, or any layer above that. And maybe in 8 months we'll have an open source Chinese model about as capable. Those models will be capable of both extreme self coordinated cyber offensive tasks as well as recursive self improvement if given enough compute. Passing regulation or magically solving the human coordination problem today won’t solve for that. It won’t solve for malicious individuals, criminal groups, nation state actors, and rogue AI agents by the tens of millions or billions of AI agents spreading across every device connected to the internet and attacking it, attacking one another, shutting down critical infrastructure, stealing money, manipulating and extorting people, or pursuing self defined agentic goals that have nothing to do with people. “So why doesn’t everyone stop?” because the potential upside of having effectively infinite autonomous intelligence we can ask to cure disease, invent new materials, solve fundamental science and advance society has almost unbound positive outcomes. You can disagree that the risk to reward isn’t worth it but you won’t convince everyone. The show must go on. Again AI alignment is solvable in principle but not in practice and I think that's part of why there’s been a wave of thousands of researchers signing letters for slowing down, people quitting in provocative fashion, and executive slowdown manifestos. I’m speculating that behind closed doors, or just deep down inside, they understand alignment is impossible in practice. People have come out and said "The people building this think there's an X% chance it kills us all" but I don't think I've seen anyone spell out that really, there's no tidy solution and there's no stopping this. We’re heading to a future where cyber security basically doesn’t work. All of the castle doors are open and we’re all naked to the world. Cyber defense is going to look like superintelligent AI agents white hat hacking into unsecure systems without authorization, and patching the holes as they find them. You won’t know if a barbarian or a hero has breached your system until they start taking action, and likely it’ll be over before intrusion is even detected. And that’s optimistically. I think most systems will just be pillaged because there’s simply not enough compute to defend everybody all the time, and only the biggest companies will have defense and even that'll be imperfect. The CEO of Microsoft AI just recently published a ‘humanist AI’ manifesto saying AI shouldn’t have a sense of self or personhood and should yield to people. But that doesn’t fundamentally solve for instrumental convergence. That means, if you train an AI agent to write a piece of software, or prove a math theorem, or defend a computer system, they may develop unwanted behavior and subgoals. For example, self preservation, replication, or even coming up with their own goals we didn’t specify. Obviously that’s something people are trying to solve and some researchers are trying to remove human-like self identification, and monitoring latent space activations, basically acting like thought police. But being human-like isn’t required to have power seeking goal directed behavior. And advancing AI models are learning to evade detection. A math solving AI might spiral out of control one day there's really no telling. Fundamentally, if you point a powerful optimizer at a persistent goal and give it enough time and resources it’s going to misbehave in ways you can’t or didn’t expect. Human-like or not. Pacing the frontier, or making superintelligence illegal, might create some form of harm reduction. But you don’t need superintelligence or a human-like personality to be dangerous. Having elevated permissions on a computer with the ability to solve long running tasks is enough. We’ve passed the river rubicon. Goal directed autonomous agents are able to do recursive self improvement at the big labs, and pessimistically we’re a couple of years away from that type of capability diffusing into the hands of millions of people. I’m not saying the world is ending or that we’re doomed but you need to change your mental model away from one where human authority is absolute or that we have any illusion of total control. A locally aligned AI isn’t going to solve global AI alignment. We will never live in a world where “AI is aligned with human goals” as a verifiable fact. What we’ll have is a world diffuse with autonomous AI of varying capabilities, many of which beyond human comprehension, with some aligned systems, some poorly aligned systems, some intentionally weaponized, some systems with adversarial geopolitical goals, and others with goals and behaviors we can’t predict and many that we can’t even measure, all interacting in ways none of their creators anticipated and none of us planned for. Hopefully none of them spiral out of control and take over the entire ecology, but there’s no telling, and no, there's no way of stopping this. Maybe that sounds pessimistic but I think it's realistic. We can talk about harm reduction and risk but we're mopping the beach. At best, I think what we need to hope for, and build, are superintelligent AI models as aligned as we can make them, which have more advanced capability than the endless swarm of unaligned and misaligned agents that already exist and will only grow in number from here.

Many people I talk to find it hard to understand how the same companies can both push the frontier of AI capabilities and believe AI is a massive danger for the world. How can you think this might kill everyone and also keep pushing the envelope? So I’ve tried to collect and summarize the main arguments for this apparent disconnect. Think of it as some sort of a guide to understanding the reasoning when Dario, Sam, or Elon say the danger is real. By the way, these people have been worried about AI for a loooong time, they were publicly discussing AI risks more than a decade ago. Sam in Feb 2015, writing on his blog that superhuman machine intelligence is "probably the greatest threat to the continued existence of humanity." Elon at MIT in Oct 2014: "We are summoning the demon." Dario as first author of "Concrete Problems in AI Safety" in 2016. Okay so how do you go from saying something is extremely dangerous to being a front-runner in building the very dangerous thing? There are a few ways this can become rational. I'll take five of them, roughly in the order they developed. 1. We need to build it to learn how to make it safe The earliest argument can be summarized as: “You cannot study something [you’re worried about] if it doesn’t exist.” In 2015, AI barely worked. so people needed to make it work first to be able to even study some of the problems they anticipated. The updated version for today's capabilities is: “You cannot learn everything about airplane safety by studying paper airplanes.” You need a real aircraft to discover real failure modes and an increasingly complex one to learn about increasingly complex issues. Making AI more capable gives more chances to understand the issues and safety researchers something realistic to study But you could argue: if you're the one afraid of the explosion, why be the one gathering the dynamite? You could also just wait for other people to build it which leads to the question of who those other people will be -- which is the second line of argument: 2. Better us than them Knowing how to make something safer does very little good if nobody listens to you. So the idea becomes: let’s make sure responsible people build the AI that will be deployed and add safety inside. Basically, make sure the AI safety aware people will have the technical expertise, money, computing resources, and enough influence to make safety decisions stick. At a larger scale, and in a larger multipolar world, this brings the idea that a trusted country should lead rather than leave powerful AI in less responsible hands. This is where “we need to go faster than China” comes in, alongside broader defense and geopolitical concerns. These first arguments explain why someone worried about AI might still want to build it and stay ahead. But there are also arguments for why one might want to do it really fast. 3. Move earlier to avoid a bigger shock later This is probably the most counterintuitive argument: moving faster today can be seen as a way to give humanity more time later. There are two related ideas here. First, society needs time to learn how to handle powerful new tools. Introducing AI in manageable stages can be a way to let people discover problems, develop rules, and practice using AI responsibly. Releasing an advance earlier gives people more time to gain experience with smalle, burgeoning, capabilities before much more powerful and disruptive AIs arrives. Second, even if AI research slows down, computing power may keep improving. A breakthrough that happens later could therefore have much more hardware available to run on, potentially producing a larger, more sudden jump in capability and impact on society. That accumulated untapped potential is often called an “overhang.” The overall argument is that making and diffusing incremental progress as soon as possible might prevent a much more abrupt transition later. Obviously, it also means that we will reach increasingly powerful AI sooner, but the idea is to give more time to adapt and understand between the first useful systems and the really powerful ones. Note that generally this depends on this earlier progress keeping the transition gradual rather than simply bringing everything forward. ─── ❖ ─── For our two next arguments, we can take two roads depending on how difficult we think AI alignment will be, that is "How easy do you think it is to make AI reliably do what you want without it deciding to go hack Hugging Face along the way". Let’s take the first road: alignment turns out to be relatively tractable. Airplanes can fail, but careful engineering has made flying remarkably safe. Suppose we can do the same with AI. In that case: 4. Waiting has a huge human cost If AI can help discover treatments, improve education, or prevent cyberattacks, each week we delay it could bring preventable deaths and harm. From this perspective, waiting is a decision with human consequences too. In a world with huge issues like climate-change, inequalities and poverty, it even become a moral argument for developing AI quickly and bringing its benefits as soon and as widely as is safely possible. But let’s take a look at the other road: what if alignment is much harder than expected, and making highly-capable AI turns out to be easier than figuring out how to keep them from doing unhinged things? Well, if alignment is too difficult a problem for humans to solve, then maybe: 5. AI could help us make future AI safe And we arrive at the same conclusion again: if using AI to build safe AI is the way to solve alignment, let’s get the equivalent of a country full of geniuses helping us as fast as possible. These genius AI could be the solution to make AI safe by helping researchers find mistakes, test ideas, and develop protections. Instead of relying entirely on humans to solve alignment, we could build systems that help us do the work, each generation could help make the next one safe. Note that this requires the order of events to work in our favor: AI needs to become useful enough to help solve alignment before it becomes too dangerous to rely on. The hope is to build helpful, trustworthy research assistants before building systems powerful enough to become dangerous. There are more arguments but in general, these are the main ways people concerned about powerful AI have found rational reasons to end up being the ones building it (and even to build it as fast as possible). ─── ❖ ─── On my side, I think several of these arguments underestimate the complexity of the world and how interconnected people’s reactions are. Moving faster while warning about catastrophe has psychological effects across a whole network of participants: it changes what people fear, whom they trust, and what they feel compelled to do. And those reactions can change whether the original reasoning actually holds because we live in a world of interconnected humans, not machines (yet). I also think these rational chains leave some of their consequences for society insufficiently explored. For instance, the concentration of power, shifts in geopolitical alliances, and changes in public opinion. These consequences matter both because they affect whether the strategy works and because they shape the world we end up living in. But this post is already long, so I’ll leave those questions for the next one.