Introducing: Odysseys2 - hardest long horizon benchmark for web agents based on real user tasks ๐๏ธ
We collaborated with
@JangLawrenceK @rsalakhu from Carnegie Mellon University
> SOTA agent fully completes only 47/100
> 5 agents complete none
> 8-12 real rubrics per task, 60 min on the live web
Web agents are becoming better than humans at knowledge work.