Benchmarks are tricky. Love that @OpenRouter is building independent, reproducible ones you can actually check. Pareto 26.10 came out #1# on quality, cost and speed in their model comparison. Another month, another frontier pushed.

benchmarks of a 50% pruned Qwen3.6-35b-a3b and expert-specific quantization technique (made by me) 7.3gb model preforming => 51gb model, exiting to see where I can bring this technique to. I have some more things lined up too. I need a DGX spark😭

Model Benchmarks Are Worthless How can a model score 24% and 59% on the same 113 coding tasks? It happens when you vary the harness that wraps it. I’ve increasingly ignored most of the recent model leaderboard results. Partly from exhaustion and partly because improvements in a ranking don’t show up in the actual work. So I asked the 8090 Labs team to measure what the leaderboards skip: the model and the harness together, on real bug fixes and features from open-source projects. 8090 Labs ran 7 models through 5 harnesses. Here are a few results I didn't expect: - Anthropic's Claude Fable 5.1 scored 41% in Claude Code and 61% in OpenAI's Codex. It made 28 tool calls at the median in one and 66 in the other. - The cheapest runs weren't the cheapest work. GLM in Codex cost $0.22 a run and $0.93 per success. In Pi it cost $0.25 a run and $0.43 per success. - GLM passed a task on both attempts 51 times in one harness and 16 times in another. - GPT-5.6 Sol and Grok 4.6 barely cared which harness they were in. The bottom line is that if you're picking a model/harness combo off a model-only leaderboard, you're making a critical decision based on only half the information.

AI Benchmarks with @Benchmark & @Vercel. It's ① the most aptly named event in SF history and ② about one of the most important software categories of our generation. The companies that benchmark models and guide the world towards skill, truth and efficiency. Join us: