when running your “Eval Factory” -> runs should encourage finding tasks that separate models by intelligence never perfect bc models are spiky but traces where 6.1 Terra fails & Sol passes are super valuable resources to understand intelligence & then build the next round of evals on you quickly get to understand what stuff is easy vs hard for each new model generation i have a big eval graveyard of tasks that were interesting but not hard enough for Astra/Sol/Opus/Argon but they were challenging ( <40% pass @ k) for Luna/terra/sonnet i hoard those in a repo instead of throwing away all that compute 😭 -> often useful for 1. building curriculums for small/medium open mocks on a particular domain (like visual reasoning) 2. having an artifact that shows how model strength is separated across tasks and why it’s fine to hoard evals, useful data to have!