Went to look at the benchmark results for Grok 4.6 x Cursor, and I'm almost there, but burned through $400 in credits fixing the cheating issues.
So...need to kindly ask for some more credits from the SpaceXAI team. Already very generous of them, really want to finish this benchmark, likely needs less than $100 to go.
Also good example of how expensive benchmarking harnesses might be. Everyone keeps saying to me, benchmark in this harness, and that harness, and yup, I'd love to, but I can't spend $10,000/mo of my own money on benchmarks.
Thanks to companies like SpaceXAI/Cursor for supporting independent benchmarks like mine.
Now I wonder if OpenAI and Anthropic might support me benchmarking their harnesses? ๐