Today I am announcing META-Bench, the first pure intelligence benchmark for AI. It leverages the hit auto-battler strategy game, TFT. I SWEAR I AM NOT TROLLING let me explain. The industry suffers from labs overfitting and giving us models that score high despite being fundamentally low IQ. Over the years there have been many attempts at benchmarking AI with competitive gaming. I am going to explain the failure points, and why META-Bench is truly the first of its kind. Chess. When picking a game to benchmark with, chess is the obvious first choice. It has clear rules, large player base, and a well defined elo system. The issue with static rule games though is that the best strategies can be figured out ahead of time and baked into the model during the training process. Too easily hacked. Memorizing more strategies is not a proof of intelligence. Dota2/ League. We’ve all heard of OpenAI Five. The issue with benchmarking on a MOBA is that reaction speed is a meaningless metric. We do not need our highly intelligent AI to be able to respond at the speed of top human pro players. And truth be told, we are years away from a LLM that is able to play MOBAs at the highest levels off of vision alone, even though the problem is seemingly solved years ago. What we need is a game that: - Has defined rules but cannot be results hacked during the training process - Large ecosystem of human players - Clear cut results and an elo system - Results that is not reaction time dependent There is only ONE game in the world that meet all the requirements needed for this benchmark. Teamfight Tactics. For those unfamiliar, TFT is a strategy based auto-battler created by Riot Games with ~100 million monthly active players worldwide. It is a highly competitive multiplayer turn based game. It’s as if Chess and League of Legends had a baby that’s born to be an AI benchmark: - There is a new set released every 3 months. - Time limitations in the 10-40 second range rather than the milliseconds required for MOBAs - Skill based enough for esports yet uncertain enough to require reasoning over hard scripts “Can’t labs just train models to be good at TFT?” Nope and the reason why it’s unhackable comes down to how the benchmark itself is set up. Due to the fact that the entire game is changed every 3 months and patched every 2 weeks, any data on a previous TFT set is effectively useless when it comes to raw pattern recognition. Strategy wise, there are core concepts that carries over from set to set. That’s why we have the same players hitting the highest elo every season even though each set is so different. Any efforts at overfitting here can be fully negated if the benchmark harness used for all models has every core strategy built in. You are never going to beat a carefully curated harness layer with strategy training at the model layer. By presenting the models in the harness with the same core strategic concepts, the only difference in outputs will be its ability to reason across the different scenarios of each game. The luck elements of TFT already ensures that no 2 games will be the same in the reasoning required. Run the models against each other enough times and you will have a clear winner. Aka, the world’s first true IQ test for AI. I really, really want to know which AI model would win this. So I am going to build this. Not too sure how I’m going to fund it yet so if you would like to invest HMU. I’m also looking to put together a small team of individuals who are both high elo in TFT and highly experienced with agentic AI. And if you are even remotely curious on the results, like and help share this post 🫡

Warcraft III: Legacy is back on the Battlenet app. If you own Warcraft III: Reforged, you can now play the old classic version. It is the same as the game before Reforged came out. To get it, open the Battlenet app. Go to your Warcraft III page. Find the Game Version menu at the bottom left, above the Play button. Select Warcraft III - Legacy TFT 1.29. It will download about 6.25 GB. Right now, you can only play it offline or with friends on the same local network. Online play on Battlenet is not ready yet. This is great for the original story campaigns and old custom maps that work best on this version.