Look at the next tweet you’re about to write. Is it bringing more positivity to the world or more negativity? I can tell you this, you will never regret helping to spread positivity. Life is short, if you’re spending your time complaining or making other people feel bad, stop wasting your precious time. Here’s to all the amazing and wonderful things we have to be grateful for. To those accounts out there spreading negativity, it’s never too late to change. I love you all ❤️

I keep hearing more good things about Antigravity. To be honest, I had totally counted it out. Now, Google seems to be making major steps forward with their models, and now their harness. Only way to know, for me, is to benchmark with @VulcanBench and compare 🖖 I know Google is making some changes/updates to Antigravity this week, so I'll wait for those to hit, and then start comparing harnesses. My plan to start is to compare Pi and Antigravity. Pi is such a solid harness, I feel like it's the best reference. Now to just figure out how to get access to Google's models without going broke 😅 But before I do anything, curious if ppl agree that Pi is a good reference harness if I wanted to compare Antigravity to one harness?

BeautifulRust just crossed 1,600 stars 🦀 🤌 Seeing a lot of great PRs, this week going to add it as an official plugin for ChatGPT, then going to try the same with Muse. I think it could be the first Rust-focused coding plugin for Muse too which would be pretty neat. Thanks to everyone for your support, very humbling.

I think we are reaching an incredibly interesting point in history where the vast majority of people are actually living in the past. We often say here on X, that we're all living in a bubble. But I'd argue some of us, like Jacob, are actually not living in a bubble at all. You can choose to live in the future. Awesome to see VCs like A16z recognize this. I see a lot of VCs getting heat lately, but it's important to remember, there are lots of great VCs out there that get it, and make it possible for great entrepreneurs to pursue huge dreams.

This morning I'm working on a new eval suite for @VulcanBench, Routine v2. I can't stop thinking about a gap that exists in benchmarking today. And I don't think I'm effectively filling that gap yet. I think what me, and many other benchmarks find ourselves doing is trying to challenge the frontier. To be clear, I think we should all keep doing this. I know for me and many others, building eval suites that truly challenge the ever-changing frontier is one of the most interesting things you can do right now. It is very likely, the hardest thing I've ever done in my life, and I'm really loving how much harder it gets each month. In many ways I find something so satisfying about seeing my eval suites get saturated, it pushes me to find new ways to experiment and learn about models. At the same time, I think the challenge with most benchmarks today, including my own, is that we tend to focus too much on trying to give models super hard tasks. Like I just said, a new set of models comes out, an eval suite saturates, and it's off to try to build a harder one. But what is still missing is real signal around what models we all need for just normal daily coding tasks. This was originally my goal with VulcanBench, then I got too interested in challenging the frontier and my core eval suite, now Frontier v4 became my focus. Now I'm working on v5 because I'm addicted to challenging the frontier and really can't stop at this point...but I also want to get back to my original mission. So I'm working on a new kind of benchmark, something I think is uniquely different from anything else out there. The goal is not to challenge the frontier, because most of the daily coding work engineers are doing, isn't really that challenging to the frontier, it's normal daily work. And I think many people are very lost now trying to figure out what model and effort level they need for regular daily work. While model routers exist, they are in a very nascent stage today. I don't think there's any model router that has the sophistication to know, ah, this is a medium-sized Python repo, doing mostly Pandas stuff, let's use this model and effort level. I just don't think current model routers would know the difference between this, and say, a medium-sized Python repo, mostly doing say supervised ML with SciKit Learn. Instead, model routers today mostly just pick an orchestrator, executor, and code review model, that will be cheaper than someone using one model for all three. Sure, it's better than just using Opus 5.5 High in Claude Code for everything, but it's not actually optimized in any way to the codebase size, programming language, etc. I really want to be able to help people understand. Okay, what model do I want to use for planning, execution, and code review for a repo of a specific size, with a specific programming language, and for certain types of tasks leveraging specific libraries. This also means rethinking how to score benchmarks, because every frontier model should be able to pretty much ace most routine tasks. So instead we need to look at things like accuracy, cost, time, and code quality, all together vs. just looking at something like accuracy and cost-per-task on a pareto chart. All of what I have shared above, will become the Routine v2 eval suite, along with a new benchmark/scoring system designed to provide signal on routine tasks, broken down by language and repo size. My goal is that I can essentially create a system where you can select your codebase size and programming language, and rank the difficulty of the tasks you're doing, and get a model and effort level out of it. For example, you might select: Repo size: Medium Programming language: Python Difficulty: Easy and it might recommend Opus 5.5 Medium for planning, GPT-6.1 Terra Medium for execution, and DeepSeek V4.1 Flash High for code review. This would be very different than if you used most of the model routers out right now, they might never use GPT-6.1 Terra Medium. They might always use the same model at the execution layer, and yes, it's 30% or 40% cheaper than if you just used Opus 5.5 High for planning, execution, and code review, but it's not actually the optimal model combination. As usual, I like getting things out of my head, and into the world, which I've done a lot on X this year and through it, I've received a ton of great feedback. So if you read this far, first, common, it's a Sunday morning, what are you doing reading my insanely long tweet? And second, what do you think? I'm always looking for feedback. Comment and let your voice be heard. (just please, no ai slop comments, real people, with real thoughts only, all accounts just posting ai comments like, "the real win is" will be blocked)

Super interesting. A number of people commented on my GPT-6.1 Terra benchmark this morning and suggested comparing it to Sonnet. Soooo, thought I'd whip something up. Here's a comparison of GPT 6.1 Terra to Claude Sonnet 5.5 across all effort levels. And the data is pretty clear here, Terra is a lot cheaper than Sonnet, but not quite as accurate until you get to Max. Traces for all 115 Terra runs, with every tool call: Leaderboard:

One of the coolest things about running Bold Metrics is that I use our product all the time. I'm a skinny dude, and if I follow size charts or size recommenders that just ask for my height and weight, I get something that doesn't fit the way I like. Personally, I actually like my tops to be slightly snug in the chest, has been my fit preference for a loooong time now. Never gets old using our custom AI models to help me find the right size the first time with some of my favorite brands. Pretty surreal feeling, and this Black Friday/Cyber Monday is going to be a historic one for us 🚀

Okay my GPT-6.1 Terra benchmark with @VulcanBench is now finished 🖖 And before I talk about the key insights from this one, let me also remind everyone that VulcanBench is a coding benchmark. 100% of the tasks in my eval suite are coding tasks. Having now used Terra all week I can tell you, it's a stellar model for computer use, I'm crazy impressed. But VulcanBench isn't about seeing how good a model is at computer use, it's all about seeing how well a model completes more complex coding tasks. So now for analysis of GPT-6.1 Terra when it comes to coding. I typically say a score between 80 - 90 on Frontier v4 is the sweet spot for routine daily coding tasks. For GPT-6.1 Terra, this means that for coding, you'll want to use Extra High or Max only. Low, Medium, and High don't really make the cut. Looking at cost-per-task, it's a very inexpensive model to use, but when you look at it side-by-side with GPT-6.1 Sol, I'd probably just use Sol. For me, after looking at this data, I still plan to use GPT-6.1 Terra as my go-to for computer use, but I probably won't use it for coding. As usual, as a completely independent benchmark, everything I do at VulcanBench is available for you to review, inspect, and even run yourself if you want to. And the traces now include tool calls so you can really examine exactly how every model I benchmark thinks! Benchmark report: Model page: Public traces on GitHub: Live long and benchmark 🖖