Register and share your invite link to earn from video plays and referrals.

Benjamin Marie
@bnjmn_marie
Independent AI researcher (LLM, NLP). My blog, The Kaitchup - AI on a Budget:
Joined June 2019
221 Following    6.9K Followers
If you retry a lot, Qwen3.8 27B can reach 92.04% on DeepSWE 1.1, ~18 pts above GPT-6 Astra’s reported ~74%. I ran 20 configs across harnesses, thinking levels, and quantizations. Coverage varied a lot: two runs both scored 31.86% Reward, yet disagreed on 42/113 tasks. I saw this mainly for quantized models. I pooled results per task: if any single candidate passed all new tests, the task passed. No patch mixing. Best-of-k: k=1 → 81.43% F2P / 37.21% reward (that's the average single-run performance) k=10 → 98.31% / 86.25% k=20 → 99.18% / 92.04% This is oracle-selected, cross-config coverage, not single-run accuracy. It shows a 27B model can produce passing solutions for most tasks. The harder problem is reliability + selecting the right candidate. With strong independent acceptance + regression tests, retries to improve the result are less unrealistic: generate → verify → retry → stop on pass. Cost is also important: imagine if one Qwen attempt costs 1/20 of a frontier attempt, 20 retries ≈ 1 frontier attempt in generation cost. But that’s only a hypothesis until actual per-attempt cost, verification, infra, and latency are measured.
Show more