Nobody at Rox writes evals anymore.
Devin now QAs our agent every week, on real traffic.
It reads real conversations with our agent, rebuilds each failure in a full copy of Rox, and turns it into an eval. We just review the PR.
The eval suite already covers 72% of real usage happening in the product, vs. 43–57% for the evals we used to write by hand.
And it’s growing week over week with Devin.
Built with
@cognition and
@braintrust: