VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
加入 March 2020
71 正在关注 2.7K 粉丝
For anyone who's interested, here's a bit more about the new eval suite I'm building for models like Jev.
It has been a really interesting experience to build an eval suite for System One Models like Jev.
My first two attempts didn't work out. But I learned a lot in the process, and since Jev is so fast, and cheap, it's a lot easier to run quick tests than with an LLM.
I'm on attempt number three with this eval suite, and thought I'd share more about it.
The eval suite is called @VulcanBench Verdict, and here's the high-level on how it works:
显示更多
