註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Viv
@Vtrivedy10
applied research @LangChain Labs, prev @awscloud, phd cv @templeuniv
加入 February 2013
1.9K 正在關注    16.9K 粉絲
when running your “Eval Factory” -> runs should encourage finding tasks that separate models by intelligence never perfect bc models are spiky but traces where 6.1 Terra fails & Sol passes are super valuable resources to understand intelligence & then build the next round of evals on you quickly get to understand what stuff is easy vs hard for each new model generation i have a big eval graveyard of tasks that were interesting but not hard enough for Astra/Sol/Opus/Argon but they were challenging ( <40% pass @ k) for Luna/terra/sonnet i hoard those in a repo instead of throwing away all that compute 😭 -> often useful for 1. building curriculums for small/medium open mocks on a particular domain (like visual reasoning) 2. having an artifact that shows how model strength is separated across tasks and why it’s fine to hoard evals, useful data to have!
顯示更多