how many tokens are in one second of human thought? a weird question -- and seemingly useless-- but humor us for a minute... (spoiler: the answer is 1.8 tokens/second) suppose you are interested in the question of the reasoning efficiency of llm's. you benchmark open-source models against closed-source ones (astra). you plot the graph. you come to an eloquent conclusion: > the pareto curve on open source models reasoning efficiency is absolute garbage ~@part_harry_ the gap definitely exists and closing it is a priority. but is the gap because astra is really, really good, or open source is really, really bad? really, the only way to answer this question is to compare against a human. whichever model is farthest from human performance is the outlier. so we decided to benchmark a human (me) with the same rl environment we benchmarked the models. after playing chess variants for 49 minutes 37 seconds, the final human score was 30%*. this makes me dumber than astra (low thinking), smarter than qwen (medium thinking), and on-par with glm. assuming average intelligence, this doesn't really reveal new alpha. > the average human is smarter than a 27b model and dumber than agi. the question is that of reasoning efficiency. how many tokens did it take me to get the 30% score? i averaged 434 seconds per game**. intuition says the easiest measurement for fair comparison should be on a scale of time it took (me/the model) to reach the final board state. but this rewards throwing more hardware at the problem, and is not fair to me (the human), because i (unfortunately) cannot shard my thinking across a node of 8 brains. instead, it was more fair to use a common scale, energy. we assume the brain uses 20 watts (google tells me this is true, open to being fact-checked). so at 20 joules/second * 434 seconds, each game took 8.5k joules / 2.1 calories. a b200 uses 1000 watts, 1k joules/second. at 50 brain seconds / gpu second, i spent about 9 b200 seconds thinking. qwen 27b on one b200 on vllm ran at an inference rate of approximately 88 output tps. so to circle back to the original question, how many tokens / second of human thought? under these assumptions, one second of my thinking time buys about 1.8 qwen output tokens on the same energy budget. my performance is therefore: - 30% on this benchmark - 781 tokens of efficiency making me both a dumber and less efficient thinker than astra, and closer to open-source models in my style of thinking. this was a fun (and honestly a little disappointing for my ego) experiment, but the point stands: - astra is a significant outlier. - there is a strong gap between it and open source models this is a teaser to the work @baselabs is going to put out soon, in an attempt to understand a n d close this gap. *1 my score would have been higher if i had known/read the rules of what anti-chess was. what do you mean i win by losing? **2 following *1, i spent more time trying to understand the rules of the game, which counted as part of the time i spent per game. to be fair, this is time spent ingesting information, not thinking.