Zbigniew Majewski
@majewskizby
๊ฐ์
May 2014
53 ํ๋ก์ ์ค 508 ํฌ
I increased test time 4 times (same 44 tasks, 4 agents in parallel, auto-graded):
๐ฅ DeepSeek-V4.1-Flash max + DeepSeek Harness: 41/44 in 87 min
๐ฅ DeepSeek-V4.1-Flash max + Terminus-2: 41/44 in 204 min (was 36)
๐ฅ DeepSeek-V4.1-Flash max + DeepSeek Harness + "work efficiently" prompt: 39/44 in 54 min
What changed:
โข Max effort was mostly too slow, not too dumb. With more time it catches up.
โข Low effort gained nothing. Its misses are real errors, not timeouts.
โข The efficiency prompt increased speed but decreased quality: about 40% less time in both harnesses, but 2โ3 fewer tasks solved.
โข The harness matters as much as the model: same DeepSeek, same 41/44, 204 min in Terminus-2 vs 87 min in DeepSeek Harness.
๋ ๋ณด๊ธฐ
Time-limited agentic benchmark on DGX Spark (44 tasks, 4 agents in parallel, auto-graded):
๐ฅ DeepSeek-V4.1-Flash (low effort, 4 Sparks): 40/44
๐ฅ GLM-5.3 (low effort, 4 Sparks): 37/44
GLM-5.3-Flash (2 Sparks): 33/44
MiniMax-M3.1-Flash (4 Sparks): 26/44 (but best vision)
Low effort edged out max for both DS and GLM. Agent tasks are time-bound, so thinking less finishes more - though max wasn't far behind.
Caveat: these are practical, moderately hard tasks (TB 2.1 subset: 14 medium, 6 hard). On harder problems without a clock, higher effort may still win.
๋ ๋ณด๊ธฐ
