Vals AI(@ValsAI):When does more reasoning stop being useful and start becoming overhead? We ran Open AI and Anthropic’s best models: GPT-6 Astra and Claude Opus 5.5 at every reasoning level on our new benchmark, MysteryMechanism. Thinking longer delivers accuracy gains with the same model at as much as 20x the cost. Inspecting traces, the models use their effort level in very different ways.

Vals AI

@ValsAI

Public LLM Evaluation // @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder

加入 March 2024

282 正在關注 30.4K 粉絲

Vals AI@ValsAI

2026.09.28 20:35

When does more reasoning stop being useful and start becoming overhead? We ran Open AI and Anthropic’s best models: GPT-6 Astra and Claude Opus 5.5 at every reasoning level on our new benchmark, MysteryMechanism. Thinking longer delivers accuracy gains with the same model at as much as 20x the cost. Inspecting traces, the models use their effort level in very different ways.