Register and share your invite link to earn from video plays and referrals.

VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
Joined March 2020
73 Following    2.8K Followers
🎉 VulcanBench v0.9.0 is out, and the focus this release is fewer dependencies. The harness dropped 7 direct dependencies and the 16 packages they needed. This means a plain install is now 15 packages, down from 37. A reduction of over 50%. I think we are moving into a time where you want to rely on as few packages as possible. Every package is code you did not write, running on your machine. I think all of us, as software developers, need to think about this, not just for ourselves but for our users. It has been an honor to see more and more people using VulcanBench to run their own benchmarks. I want to keep the list of things they have to trust as short as I can. Now live on GitHub: Live long and benchmark 🖖
Show more