๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Yifan Wu
@yifannnwu
ๅดๅฅ•ๅ‡ก; AI Research Scientist @Meta | Ph.D. @penn @picslupenn @GRASPlab.
๊ฐ€์ž… November 2016
490 ํŒ”๋กœ์ž‰ ์ค‘    1.4K ํŒฌ
Introducing SWE-Together: a multi-turn benchmark built from real userโ€“agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the final code. But real coding help is a conversation โ€” users clarify goals, add constraints, and correct course along the way. SWE-Together turns real coding work into a reproducible, verifiable benchmark: 109 repo-level tasks curated from 11,260 recorded sessions, replayed with a reactive LLM user simulator that preserves the original userโ€™s intent. We evaluate agents as collaborators, not just patch generators: final pass rate and how many user interventions were needed to get there. In this evaluation snapshot, claude-opus-4.8 currently leads among the 7 agents we tested โ€” achieving the highest pass rate while requiring the fewest user interventions. ๐Ÿ“„ Paper: ๐Ÿ’ป Code: ๐ŸŒ Website:
๋” ๋ณด๊ธฐ