Introducing SWE-Together: a multi-turn benchmark built from real userโagent coding sessions.
Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the final code. But real coding help is a conversation โ users clarify goals, add constraints, and correct course along the way.
SWE-Together turns real coding work into a reproducible, verifiable benchmark: 109 repo-level tasks curated from 11,260 recorded sessions, replayed with a reactive LLM user simulator that preserves the original userโs intent.
We evaluate agents as collaborators, not just patch generators: final pass rate and how many user interventions were needed to get there.
In this evaluation snapshot, claude-opus-4.8 currently leads among the 7 agents we tested โ achieving the highest pass rate while requiring the fewest user interventions.
๐ Paper:
๐ป Code:
๐ Website: