Alex Shaw
@alexgshaw
Hacking on @terminalbench and @harborframework. Founding MTS @LaudeInstitute. Formerly Google. BYU alum.
가입 October 2021
771 팔로잉 중 2.8K 팬
Btw we are the ones who open the GitHub issues bc we continuously maintain our benchmarks
We audited 15 benchmarks and labeled 9 flawed:
- In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues.
- In HLE, 46% of the 48 questions we randomly sampled were broken.
- In DeepSWE 1.1 we found a bug that can break grading for every task.
더 보기
