janak
@janaksunil
cofounder specific labs (yc f25), real-swe
1.1K Following 4.8K Followers
we benchmarked particle-1 on
turns out you can achieve the same performance with 43% fewer tokens
congrats @Suhail
Introducing Real-SWE, a coding benchmark built from private company codebases.
We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases.
These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc.
The results are quite surprising.
Show more
