Sarah Catanzaro
@sarahcat21
“All methods are sacred if they are internally necessary” (GP @amplifypartners, prev @canvasvc; Head of Data @Mattermark; @palantirtech; @c4ads)
1.6K Following 16.4K Followers
I think we’ll likely see a dramatic decrease in the number and potentially the value of public benchmarks; so many people are realizing that your benchmark IS your secret sauce AND it’s hard to come up with “general” frontier tasks vs what’s frontier for your specific use cases
Show more
“the cold start is smaller than people think” - many people complain about designing a comprehensive set of tasks & verifiers. But you don’t NEED this to get started. You need a few good ones & the means to analyze your outcomes/traces so you can update & append; evolve your evals.
Show more
Eval developers are the next analytics engineers; people who will set standards that ultimately govern how models are trained and deployed. I expect to see many more companies hire for this role.
