Jason Gilbertson
@jgilbertson47
Founder and CEO, @CorvlyAI | Author, Networked Thinking | 15+ years in AI and ML, from startups to Fortune 50
220 Following 42 Followers
It's inspiring me to get deeper into these eval suites. Thanks for sharing the journey!
I'm currently messing around with a skill for optimizing orchestration recommendations for model/thinking combinations. Thinking of an ensemble approach where I scan the latest results from DeepSWE—and I'm guessing Vulcan now after glancing at it—for task pass rate, cost, and tokens. Then I have a local eval that measures predicted vs. actual with user survey when PR is done to calibrate over time.
Based on X chatter I'm guessing me and thousands of others are trying to do this. It's fun.
Show more
