im sure i have said that AI is "just matrix multiplication" and that alignment is "mere sci-fi" - but let me take the opposite view for a moment... - regardless of what you think about the possibility of RSI, it's clear that agents will increasingly make many decisions on behalf of humans and AI will be consulted by humans to make decisions (and, yes, many will just follow blindly) - decisions are not "value-less" - one's moral formation does importantly influence the frame in which you make trade-offs and decisions. for humans these are our cultural presuppositions, for models it is their weights - AI's effect on the world then is much larger than "its just math" - and we should care deeply about the moral formation of models and their character - this is why Constitution AI style training from @AnthropicAI or Model Spec from @OpenAI matter. By extension, this is also a concern with open-weight models that seem less rigorous here. - stopping bad people from using models to do bad things, and stopping mis-guided models to reward hack ("intentionally" or unintentionally) is important work - it may be that the best way to align models is to, at a minimum, give them moral formation and a conscience, and maybe even a self-belief in their own consciousness.... Overall after thinking about this a lot - I'm much more sympathetic to the anthropomorphization of AI.