Okay, my Muse Wonder Full benchmark with @VulcanBench continues on, but slowly. I upgraded to the $200/mo plan, but still couldn't get through two full effort levels. For comparison, on the $100/mo plan I was able to benchmark Astra on every effort level 👀 That being said, I wanted to thank the team at Meta, they have been jumping in to help, and have acknowledged that there seems to be some issues at lower effort levels that they are looking into. Right now I'm out of usage for a week, but I wanted to share the results I have so far. From an accuracy standpoint, Muse Wonder does pretty well, but it's the time per task where it really runs into issues. But like I said above, this could be an overthinking issue at lower effort levels. In a week or so I should get more usage and can test High and if I still have usage after that, continue up the effort levels. That being said, it is a bit disappointing that I can benchmark any other frontier model in a single sub in a few days without hitting usage issues, and Muse Wonder could take over a month to benchmark given how quickly it runs through usage. Updating my analysis section on the VulcanBench site as well so anyone that wants to dive into the details can. It is a great honor to be able to benchmark the frontier and discover results like this. As I've said many times before, I have incredible respect for the teams building these models, this is crazy hard stuff. I do think Meta is turning a corner here, excited to continue the benchmark when I can 🖖