Epoch's slack is an amazing place to be, with @scaling01, @fleetingbits, @justjoshinyou13, and many other fantastic researchers discussing their takes and posting charts. Too much cool stuff going on there to keep track of all of it.
We audited 15 benchmarks and labeled 9 flawed:
- In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues.
- In HLE, 46% of the 48 questions we randomly sampled were broken.
- In DeepSWE 1.1 we found a bug that can break grading for every task.
I continue to be surprised about how big of a deal Mythos (and co) have been to cybersecurity. Here's critical vulns found at Oracle over the past few years: