janak(@janaksunil):Introducing Real-SWE, a coding benchmark built from private company codebases. We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases. These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc. The results are quite surprising.

2026.09.10 17:00

Introducing Real-SWE, a coding benchmark built from private company codebases. We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases. These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc. The results are quite surprising.