Zbigniew Majewski(@majewskizby):I increased test time 4 times (same 44 tasks, 4 agents in parallel, auto-graded): 🥇 DeepSeek-V4.1-Flash max + DeepSeek Harness: 41/44 in 87 min 🥈 DeepSeek-V4.1-Flash max + Terminus-2: 41/44 in 204 min (was 36) 🥉 DeepSeek-V4.1-Flash max + DeepSeek Harness + "work efficiently" prompt: 39/44 in 54 min What changed: • Max effort was mostly too slow, not too dumb. With more time it catches up. • Low effort gained nothing. Its misses are real errors, not timeouts. • The efficiency prompt increased speed but decreased quality: about 40% less time in both harnesses, but 2–3 fewer tasks solved. • The harness matters as much as the model: same DeepSeek, same 41/44, 204 min in Terminus-2 vs 87 min in DeepSeek Harness.

2026.10.26 08:28

I increased test time 4 times (same 44 tasks, 4 agents in parallel, auto-graded): 🥇 DeepSeek-V4.1-Flash max + DeepSeek Harness: 41/44 in 87 min 🥈 DeepSeek-V4.1-Flash max + Terminus-2: 41/44 in 204 min (was 36) 🥉 DeepSeek-V4.1-Flash max + DeepSeek Harness + "work efficiently" prompt: 39/44 in 54 min What changed: • Max effort was mostly too slow, not too dumb. With more time it catches up. • Low effort gained nothing. Its misses are real errors, not timeouts. • The efficiency prompt increased speed but decreased quality: about 40% less time in both harnesses, but 2–3 fewer tasks solved. • The harness matters as much as the model: same DeepSeek, same 41/44, 204 min in Terminus-2 vs 87 min in DeepSeek Harness.

Zbigniew Majewski@majewskizby

2026.10.25 08:33

Time-limited agentic benchmark on DGX Spark (44 tasks, 4 agents in parallel, auto-graded): 🥇 DeepSeek-V4.1-Flash (low effort, 4 Sparks): 40/44 🥈 GLM-5.3 (low effort, 4 Sparks): 37/44 GLM-5.3-Flash (2 Sparks): 33/44 MiniMax-M3.1-Flash (4 Sparks): 26/44 (but best vision) Low effort edged out max for both DS and GLM. Agent tasks are time-bound, so thinking less finishes more - though max wasn't far behind. Caveat: these are practical, moderately hard tasks (TB 2.1 subset: 14 medium, 6 hard). On harder problems without a clock, higher effort may still win.