注册并分享邀请链接,可获得视频播放与邀请奖励。

matt
@matt_feroz
founding dev advocate @merge_api | ai content creator | oss @t3codes @dhh
加入 September 2015
608 正在关注    828 粉丝
It's crazy how early we are to evals. There are millions of different permutations that can be tested and the systems we have in place need to be improved. Working on something to help though, stay tuned
显示更多
I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry. @xeophon pointed me to this excellent site which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪 For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality. I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.
显示更多