One thing that has been on my mind a lot lately as I continue to build out eval suites, is the time given to each task in the suite, and how that scales with difficulty. I think TerminalBench does a pretty good job with this, here's a look at how they do it (image below). As you can see from the table below, most of the tasks in their eval suite, need to be completed by the model in two hours or less. But there are some tasks, that are more difficult, that are given 4, 5, and even one task with 8 hours. Thinking about following a similar pattern for my next eval suite, but a little scared at what my costs would be on an all effort sweep at 8 hours, so might not go that high. For anyone else building eval suites, curious how you think about agent time to completion budget?