A fun re-discovery of my blog post “Can friendly AI evolve?” from 22 years ago. Translating to today: is self-preservation an eventual emergent property of any iterative algorithm with selection pressures (like continuous RL and recursive self-improvement)? 2004 time capsule: In the comments, Drexler stands out, reflecting on group intelligence, a supra-human emergence: "why should we think that 'artificial intelligence' must entail 'an artificial intelligence', with a purpose and a will? This is a mere assumption, all the worse for being unconscious. Artificial intelligence, too, can be ecological rather than unitary. Because intelligence is a property, not an entity, the problem of "Us vs. Them" need not arise. It seems that we can develop and use powerful new forms of intelligence without having to trust any of Them. I propose a moratorium on talk that blindly equates intelligence with entities." And the proto-doomer Eliezer @ESYudkowsky: "Evolving Friendly AI looks impossible. [A goal splintering] problem is intrinsic to any optimization process that makes probabilistic optimizations, or optimizes on the basis of correlation with the optimization criterion. The original criterion will not be faithfully preserved. This problem is intrinsic to natural selection and directed evolution. Building Friendly AI requires optimization processes that self-modify using deductive abstract reasoning (P ~ 100%) to write new code that preserves their current optimization target." These musings motivated Larry Lessig and I to co-teach a class on it at Stanford in 2004. We assigned Susan Blackmore's Meme Machine, a book on human germ-line engineering, and Posner's Catastrophe: Risk and Response. A conclusion, from another 2004 post: "Many of us subconsciously cling to the selfish notion that humanity is the endpoint of evolution. In the debates about machine intelligence and genetic enhancements, there is a common and deeply rooted fear about being surpassed – in our lifetime. But, when framed as a question of parenthood (would you want your great grandchild to be smarter and healthier than you?), the emotion often shifts from a selfish sense of supremacy to a universal human search for symbolic immortality." —