Alex Shaw
@alexgshaw
Hacking on @terminalbench and @harborframework. Founding MTS @LaudeInstitute. Formerly Google. BYU alum.
加入 October 2021
771 正在关注 2.8K 粉丝
Btw we are the ones who open the GitHub issues bc we continuously maintain our benchmarks
We audited 15 benchmarks and labeled 9 flawed:
- In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues.
- In HLE, 46% of the 48 questions we randomly sampled were broken.
- In DeepSWE 1.1 we found a bug that can break grading for every task.
显示更多
