Håvard Ihle(@htihle):GLM 5.3 (max) scores 75.4% on WeirdML, up from 70.1% for GML 5.2 (max), but well behind Kimi-K3 at 82.6% or Claude Opus/Fable at ~92%. It's somewhat affected by the explore-vs-score issues that many recent models have shown, but the estimated effect of this is less than 1 percent on the final score. This was run through the Fireworks provider on Openrouter.

2026.09.01 09:34

GLM 5.3 (max) scores 75.4% on WeirdML, up from 70.1% for GML 5.2 (max), but well behind Kimi-K3 at 82.6% or Claude Opus/Fable at ~92%. It's somewhat affected by the explore-vs-score issues that many recent models have shown, but the estimated effect of this is less than 1 percent on the final score. This was run through the Fireworks provider on Openrouter.

Håvard Ihle@htihle

2025.06.27 14:21

WeirdML v2 is now out! The update includes a bunch of new tasks (now 19 tasks total, up from 6), and results from all the latest models. We now also track api costs and other metadata which give more insight into the different models. The new results are shown in these two figures. The first one shows an overview of the overall results as well as the results on individual tasks, in addition to various metadata. The second figure shows cost vs performance and shows a clear scaling with better results for higher costs. We also have a very varied pareto frontier with 11 models from 6 different companies having the best accuracy for a given cost for at least some of the cost range. Grok 3, Claude Opus 4 and GPT 4.5 are the ones that underperform for their costs, while Gemini pro and o3 pro have the best results at the highest costs. Qwen3 30B3A, grok 3 mini and deepseek R1 also each represent a good chunk of the pareto frontier.