Okay well I finished my Devin SWE-2 results with @VulcanBench today, so figured I would share them now since I'm already spinning up benchmarks of all the new stuff both OpenAI and Anthropic released today. It's a busy time to be a benchmarker. And an expensive time to be an independent benchmarker 😅 Overall Devin SWE-2 is a pretty impressive model for the cost, which is $0 right now so pretty hard to beat. It came in noticeably more accurate than Muse Spark 1.3 and only a few points behind Astra and Fable. So while I might not use it for my hardest of hard tasks, for normal routine tasks I think it's very likely that Devin SWE-2 is more than powerful enough. Still hard to beat Astra when it comes to speed, it is noticeably faster than Fable, Muse, and Devin. Both Devin and Muse take longer than Fable, they can think a lot on my Frontier v4 suite, which is designed to give them quite a challenge. I will be running all of these on my new Routine v1 suite which will be pretty interesting as I don't think Muse or Devin are really designed to throw your hardest tasks at, so it will be interesting to see how they do on more routine tasks that Astra and Fable, and now Opus 5.5, are all likely overkill for. More to come, as always, live long and benchmark 🖖

Very interesting, just got an update from my agent running a judging sweep. I think the Cursor name could be going away 👀 (which just to share my opinion, I'm fine with, makes sense to have it under one brand, new era imo)

Okay, finally finished my Muse Spark 1.3 benchmark. This ended up being the longest-running benchmark I've ever done at @VulcanBench and it definitely looks like Muse has some serious issues at lower effort levels. I was able to run Astra through my v4 eval suite across every effort level in ~12 hours, Muse Spark had single tasks at low effort levels that hit the 10 hour mark, on just a single task. It took a total of two weeks for me to run the entire benchmark. Overall my assessment is that Muse is a promising model, but they'll have to figure out why it has issues at lower effort levels, looking at the traces, it's just thinking, thinking, and thinking some more, while models like Astra and Fable just got it done. Astra continues to be the most token efficient of these three, and I continue to feel very good about Astra Light, both token efficient and accurate. Using Muse Spark 1.3 at lower effort levels is a complete non-starter imo because you'll be waiting 10 hours for something Astra light can do in 10 minutes. Model cards below, adding to the site soon:

I think a lot about model routing, heck I even write a Substack just it, so yeah, I'm kinda obsessed. At a high level, I don't think most of the model routers out there make much sense for anyone but enterprise co's already paying api pricing to use services like Anthropic and OpenAI. And I think this is honestly, the target market for most of the companies releasing model routers today. So it makes sense, and yes, will definitely save enterprises money. If you're spending $500,000/year on a Claude Sub. And everyone is just using Opus High for everything, yup, putting some thought into model selection, even at API+ pricing, which is what most routers charge, is still a savings. But if you're using a subscription, at $100 or $200/mo, switching to a model router will dramatically increase what you pay, because you're not paying API pricing, you're paying a massively discounted price. This doesn't mean the model routing co's are doing anything wrong, they're just not solving a problem for you, they're solving a problem that big enterprise co's have. I've seen so many posts, and talked to so many people at big co's that say, it's all too overwhelming for them and their teams, so they tend to just see everyone using one model, usually the latest frontier model, and at High effort for everytihng. What will be interesting to see is over time will these companies just decide to outsource this decision logic to third party model routers, or will they use benchmark data, both public, and internal evals, to just give their teams some model routing decisions, i.e. use these three models, at these effort levels, etc. I think the biggest, unexplored territory right now is effort level. This is why I started @VulcanBench because I saw just about every benchmark just running evals with models at Max effort, and I knew for myself and my team, we used Medium effort more than any other effort level. But a model that does great at Max, might not be great at Medium, and another model might be better. And it could be completely different for Python code than Rust code, and medium-sized codebases vs. large. These days, I wake up every morning so excited to be doing research in this area. There is so much to discover. And that's my Monday brain dump, big week ahead, TGIM 🖖

Okay, bought more parts to connect my new 3090 to my existing gaming pc. As I said, I’ll share every step of the journey. Yesterday I shared the power supply I ordered. Just ordered these two today.

Interesting analysis, hearing non-stop reports at this point of ppl saying their Codex usage is going down way faster than before. I am not experiencing this myself, but I also use most models on Low and Medium effort, so I might not be a great example.

One thing I’ve learned about myself is that for my work environment, I need to look out at either trees or mountains with trees. Two total no-go’s for me are working with a view of a city/buildings, or working inside just looking at other people working. Coffee shops are my nightmare work environment for many reasons, this being one. There are a few different places I work from in our house, here’s one, and the view. My soul needs the trees 🧘‍♂️ 🌲

I am adding Jev to @AlbatrossCoding, and it could be a really interesting way to make my harness a lot more token efficient. Instead of just adding Jev as a model provider, I'm going to have essentially what I guess I'll call "Jev Mode" that allows you to leverage what a model like this is actually good at, i.e. making decisions. Right now, we waste a lot of time with agents having them perform lots of tool calls. I think Jev could essentially short circuit this in a way. The logic I'm adding will add a step where your prompt goes to Jev first. If it's a question with predefined answers, Jev can just answer it. Otherwise, it goes to an LLM, which can answer or use tools. I can benchmark this against a traditional harness to see what kind of performance gains it gets, my thought it is could be a lot faster and more token efficient for some tasks. Will share more once it's ready, but I'm also very open to changing my design here so please poke holes in it if you see a way I could improve. Hoping to make Albatross one of the first harnesses with Jev Inside™ 🪽 Diagram below of the flow I'm building out:

Just in case anyone thought I was abandoning Diablo for WoW, not to worry, still giving Diablo the love it deserves. But I have played more WoW today than Diablo 👀 And yes, I’m re-specing a Spiritborn rn with a Quill Volley Build. Shared in the second image for anyone interested.

So here’s the power supply I’m thinking of getting for my 3090, anyone from the local ai world want to help give this a thumbs up or thumbs down? Seems like more than enough power and solid reviews.