matt
@matt_feroz
Joined September 2015
608 Following 828 Followers
It's crazy how early we are to evals.
There are millions of different permutations that can be tested and the systems we have in place need to be improved.
Working on something to help though, stay tuned
Show more
I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry.
@xeophon pointed me to this excellent site which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪
For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality.
I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.
Show more
