Register and share your invite link to earn from video plays and referrals.

Isaac Flath
@isaac_flath
Documents in AI Products
Joined August 2020
11 Following    3.1K Followers
Everyone tends to be #1# on their own benchmark, but it's not a sign of trickery or financial bias. It's a sign that problem specific evals are what's important. Here's why. Whatever you think is most important is what your solutions will probably do well at, because that's what you focus on (duh). Whatever you think is most important is what your benchmark should measure (also duh). So it's only natural that your solutions will do best. They're both designed to focus on your assumptions of what's most important. But it also shows that even small differences in the primary goal changes benchmark rankings for models. Which should tell you that you really need to make your own dataset with your own tests based on what's important for your product.
Show more