Dario created Anthropic out of concern that OpenAI wasn’t developing AI safely enough. We now have two tech giants in a breakneck race that neither seems to want to continue apace but neither seems willing to stop. Do you feel safer? OpenAI was launched to safely usher in AGI. Yet before OpenAI, LLMs were mostly just a tool Google used behind the scenes to process search queries. Do you feel safer? AI safety researchers like Paul Christiano developed alignment techniques like RLHF to control the behavior of LLMs, yet this is what made LLMs powerful and useful enough to go mainstream and usher in the age of generative AI we’re in now. Do you feel safer? I believe all of these individuals/organizations were sincere in their attempt to make AI safer. But I also believe their efforts backfired spectacularly and made the world less safe. Am I wrong? I fear that much of society’s efforts to make AI safe have succeeded well enough to make AI widely popular and extremely powerful yet failed to go the extra mile needed to make it actually safe--thus arguably putting us more at risk, not less. And I fear this pattern shows no sign of abating.

Alright I’ll say it bluntly: when we send people with long backgrounds in EA/rationalism/alignment to investigate these AI hacks, we shouldn’t be surprised when they find alien species going rogue rather than flawed software and subpar cybersecurity, no? As the saying goes, we find what we’re looking for.

I don't see why so many people seem to think that RSI means we won't understand AI anymore. (1) we already don't understand the behavior of AIs despite knowing how they're designed, and (2) even with RSI, an AI model is a piece of software written in a programming language, and we can use LLMs to help explain the code to us, that's something it's really good it. RSI could certainly lead to faster hyperparameter optimization, perhaps even major design innovations, more efficient learning paradigms, etc., but we can still pay attention and study the code the AIs are writing, just as we do now--I don't see that changing whether AI writes 0% or 80% or 100% of the code. Am I missing something here?