I just wanted to do a deeper dive in how we underestimated Grok 4.7. I wanted to take a task-by-task look of the cost results given the interest in this piece. I wanted to focus on the most common effort for work, medium. I also am including the full suite of Composer 2.5, Grok 4.5, Grok 4.6, and 4.7. from my Cursor family set. I ran the Grok 4.5 and 4.6 full suites a few weeks ago but hadn't had a chance to do a full post. Note that with all models, caching can be inconsistent cost wise, especially during rollouts. One first realization is the clear improvement from Grok 4.5 to 4.6. For many tasks, you also see a clear improvement to Grok 4.7. At the task level, performance, in 7 of the 23 Grok 4.7 outright tied Astra and 13 out of 23 Grok 4.7 tied/beat Opus 5.5. However, for about 1/3 of the cases Grok was inconsistent in costing significantly more and not in trend with each other or other families. Opus and Sol have their own quirks, but we do see less consistency with Grok. This likely explains why there are many strong views of Grok as depending on your tasks, Grok 4.6 and 4.7 can be either on par with frontier or significantly less efficient. Overall, Sol 6.1 is still the cost-efficient leader. However, for the upper end of challenging but common problems, Grok is a decent choice. What will be really interesting is to see the routine tests. I also want to say that Composer 2.5 continues to get its honorable mention for outperformance. A big thank you to @morganlinton and @VulcanBench for developing their tool and running so many benchmarks.

So I went task by task for the recommended medium level and max for comparison. Sol and Astra clearly look alike. Similar trend with increased effort with more cost and more time. Sol 6.1 looks like a cheaper slower Astra. Opus 5.5 results are wild for 2/3 of the tasks it is cost competitive. However it doesn't have any effort trend and some tasks are all over the place in cost. Still, all are major improvements over predecessors. Kudos to @morganlinton and @VulcanBench

I rely on AI heavily to maintain my OSINT analysis enterprise. For defense readers, you can skip to the first reply to see warfare and Ukraine implications. For those interested in how Composer 2.5 holds its own against the Frontier, keep reading. We don't always need Frontier models when many older models get the job done. Transforming equations and data collection to code that runs on VMs 24/7 takes time and effort Just as I benchmark my oil and interception models, I also rely on benchmarks for my AI tools. As I do this on my own budget, costs matter. Many benchmarks use hypothetical problems that don't relate to normal work. @morganlinton has made a benchmark @VulcanBench that does a great job of testing models against real world stressing problems. Recently he compared Fable 5.1, Astra, and Opus 5.5. I decided to compare it to Composer 2.5, an older but very well developed and economical mode by @cursor_ai . It held is own and was able to accomplish many of the tasks and be cost competitive. This is why. I want to start by saying Morgan made some amazing tests and his prompts are incredibly well written. Looking at the results for each of the 23 models shows that for shorter tasks, Composer does a great job and is either the cheapest or 2nd cheapest option for about half. For the other half it is more expensive. This is simply a case where it cannot support tasks of that length. Too long and too many steps for the context window. This is where frontier models shine and why many benchmarks undersell their capabilities versus tailored coding models like many of the Chinese ones. However, many of these frontier models are great at breaking tasks into smaller chunks so strong, cheap, but older models like Composer 2.5 can handle. I have used Grok 4.6/4.7, Fable 5/5.1, Sol 5.6, and Opus 5.5 to break down tasks. When paired with Composer, it is like getting those frontier models at a very steep discount. For cost conscious researcher like myself, most of what we need are Lunas and Sonnets and Composers led by frontier models. @grok please keep your brother-in-law, Composer 2.5, in the family.