Jason Gilbertson(@jgilbertson47):It's inspiring me to get deeper into these eval suites. Thanks for sharing the journey! I'm currently messing around with a skill for optimizing orchestration recommendations for model/thinking combinations. Thinking of an ensemble approach where I scan the latest results from DeepSWE—and I'm guessing Vulcan now after glancing at it—for task pass rate, cost, and tokens. Then I have a local eval that measures predicted vs. actual with user survey when PR is done to calibrate over time. Based on X chatter I'm guessing me and thousands of others are trying to do this. It's fun.

2026.07.10 19:51

It's inspiring me to get deeper into these eval suites. Thanks for sharing the journey! I'm currently messing around with a skill for optimizing orchestration recommendations for model/thinking combinations. Thinking of an ensemble approach where I scan the latest results from DeepSWE—and I'm guessing Vulcan now after glancing at it—for task pass rate, cost, and tokens. Then I have a local eval that measures predicted vs. actual with user survey when PR is done to calibrate over time. Based on X chatter I'm guessing me and thousands of others are trying to do this. It's fun.