Every AI agent vendor says they're the best. For weeks, we've been wondering about how to determine who actually is. Fin used to have a chart on their website showing they are better than Decagon and Sierra. Is that true? It's impossible to validate their claim. Our goal was to build an objective measurement for our internal teams to know where we're at relative to competitors. Where we're strong and where we need to learn. So we built a benchmark with AI. 12 agents, 160+ live Shopify and ecom stores, the same typed questions for everyone, a blind LLM grader that never sees the vendor name (we even asked Fable+Astra to tell us if our grader was objective enough to make sure it was!). At first we thought the ranking would be the interesting part. Who's #1#, who isn't. Then we looked at the data and the interesting part was somewhere else: - "Automation rate" is misleading. An agent can carry a whole conversation, claiming it solved the problem, but actually didn't. Handling a ticket isn't the same as helping the customer. - The most common failure more is a wall or a loop. Asking for an order number three times, forcing a login, quietly passing everything to a human. So now, let's talk about the results 🥁 No one wins on all three but Gorgias is #1# overall - CX: 🥇 @gorgiasio & Yuma, 🥈 Decagon, 🥉 Sierra - Sales:🥇 Envive AI, 🥈 Gorgias, 🥉 Fin - Overall: Gorgias ranks #1# across all categories We're running ˜100+ new conversations a day in this benchmark, so results are dynamic. You can explore the full results here: Results on our G2 page here: And if you'd like to do the same thing for your company, we open sourced this benchmark here: Huge shoutout to @maxpruv - who spent hours and quite a few tokens getting this live!