註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

matt
@matt_feroz
founding dev advocate @merge_api | ai content creator | oss @t3codes @dhh
加入 September 2015
608 正在關注    828 粉絲
It's crazy how early we are to evals. There are millions of different permutations that can be tested and the systems we have in place need to be improved. Working on something to help though, stay tuned
顯示更多
I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry. @xeophon pointed me to this excellent site which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪 For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality. I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.
顯示更多