Register and share your invite link to earn from video plays and referrals.

Ishan Mukherjee
@ishanmkh
dad + founder. now: cofounder/ceo @rox_ai past: 4 newcos + 3 bigcos after @mit
Joined April 2009
586 Following    1.8K Followers
Every AI team says they have evals, but most are just guesses about what users will do. Using @DevinAI and @braintrust, the team built an eval suite that grows automatically with the product. Proud of @gopalgoel, @XuBriGuy, @AksharSarvesh for building this. This is the kind of problem we hire for.
Show more
Nobody at Rox writes evals anymore. Devin now QAs our agent every week, on real traffic. It reads real conversations with our agent, rebuilds each failure in a full copy of Rox, and turns it into an eval. We just review the PR. The eval suite already covers 72% of real usage happening in the product, vs. 43–57% for the evals we used to write by hand. And it’s growing week over week with Devin. Built with @cognition and @braintrust:
Show more