matt(@matt_feroz):It's crazy how early we are to evals. There are millions of different permutations that can be tested and the systems we have in place need to be improved. Working on something to help though, stay tuned

2026.10.26 14:41

It's crazy how early we are to evals. There are millions of different permutations that can be tested and the systems we have in place need to be improved. Working on something to help though, stay tuned

Hamel Husain@HamelHusain

2026.10.25 19:09

I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry. @xeophon pointed me to this excellent site which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪 For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality. I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.