In all of this we have forgotten to discuss the fact that people *get high* on the idea of the apocalypse. There is a reverie the human organism finds in it, which wells up and recurs throughout history. Of course this doesn't mean that next time isn't different. It could be. Total nuclear war was actually dangerous. But we shouldn't lose sight of the psychology here. People get high on doom.

AI doom usually imagines AI evolutionary fitness going to infinity, with human fitness staying fixed. From this comes "we need to merge", etc. But there are many ways we can increase our effective relative fitness without drastic changes to neural architecture. This is what culture does, for example. This is what engineering does, for example. And even if your framing is this evolutionary one, and you're thinking about longer timescales, we have a huge head start. Greater intelligence doesn't computationally necessitate greater fitness. We have to figure out how to disentangle these more than we currently know how, yes, but they aren't totally and perfectly entangled in all worlds. Insofar as it's something we have to deal with at scale, which there is mounting evidence we do, strong instrumental convergence is ultimately an engineering skill issue. We will be able to simply program these systems much better than we can today.

There are real and potentially fundamental technical problems to solve with respect to building AI systems we actually want and can trust. There is also something in this fact, when you get up close and look at it clearly, that seems to break people's psychology.

Doesn’t always happen but I find it oddly entertaining to watch when someone formally aligns with a firm or big personality and then eventually everything they say is sampled from this small model smell caricatured distillation of that firm or personality

If alignment/safety/control/monitoring etc. are such important problems (they are) then why aren't labs open sourcing everything they have on these topics (without leaking too much IP) and allowing academics and other researchers to contribute to the effort?

Now that John Ternus is CEO, he’s finally free to release the Apple Antigravity self-flying car, the Apple Dust whole brain neural implant, the Apple Freeze cryonics chamber for extreme longevity, the 10TB unified memory Mac Studio, and the foldable iPhone