A common trend is that once we measure something well, the models will saturate the performance on that metric within a year. I really hope this happens for model performance in mental health situations! Also congrats Luna, model powering billions of users and topping the chart!

This number is way off. You’re comparing one human brain to an entire data center capable of training new models. The actual answer is: IQ points per watt: Humans: 5 AI: 7 - 40 (!!) By my rough calculations the current AI is already served more efficiently than humans for equivalently intelligent work performed! A single gpu can produce outputs and parse inputs way faster than a human and can do many of such requests in parallel. A larger model generally mostly means you need more gpus to serve it, but that often also increases how many requests you can serve in parallel, so I’m reporting on the share of the serving system, and also normalizing for speed. Really a better way to measure and report it is joules per equally good completed task, as your formulation suggests we can likely add more watts to get more iq, which isn’t correct. I did some back of the napkin calculations based on the latest serving hardware and the most powerful open source models. But even given pretty generous assumptions it turns outout computers come ahead. For example the request is allocated approximately 60 W while running, but finishes a task in about one-seventh of the human time.

This is likely to be the new Moore’s law, and the right thing to measure. So far it seems to be much more aggressively growing than Moore’s law, as it combines hardware and software improvements in a multiplicative way!