Many people reading the Math AI misalignment letter are reacting with “oh those mathematicians, they just didn’t like the fact that AI is taking away their jobs”. That’s the wrong take. And this is exactly the misalignment that we need to understand. There are human endeavors that the whole point is the outcome. This shows up in engineering. Let’s say we want to design a better rocket engine. All basic principles are all grounded in physics, and AI can speed run the design space. The end result can be something superhuman but it does not take more than a college graduate these days to look at the design and understand it because it is fully within the basic principles of modern physics. And then there are human endeavors in basic science that the primary focus is understanding itself. This is how Newton and Einstein are remembered by us. They advanced our understanding of the reality. For example, AI speedrun the gravity law seems harmless, but it may not take the calculus route and only the numerical route, and thus human still does not know how to “understand” gravity. If AI can speed run to cure a specific cancer out of a specific drug, we still want to know the mechanism, otherwise we’d be using AI as a witch doctor. Humanity thrived after the Renaissance because we have used science as the compound substrate to accumulate understandings and explanations. A misalignment here, as shown the OpenAI’s act, is to skip that accumulation step at the cost of understanding. Mathematicians have been saying forever that solving open problems isn’t about the end goal, it’s the tools and thinkings along the way. Because those paths are the compound that allows human to thrive in the next cycle. Taking that away is alarming not for mathematicians, it’s alarming for humanity. If superintelligence AI is becoming a witch doctor, we’d be living in Ignorance. Ayn Rand would want to have a word with you.