The first results for Terminal Bench 2.1, all 89 tasks, 3 iterations. Our ~1,300 lines Mini Harness landed in the middle of the pack, while being the fastest.
In progress: OpenCode, Terminus 2, mini-swe-agent.
Show more
Turns out DeepSeek harness improves the score of DeepSeek V4.1 Flash by 2-3 points on average.
It also costs 80% more due to subagents and more output tokens.
New results:
DeepSeek V4.1 Flash 1017 + Claude Code harness:
- max, mean of 7: 39.1
- high, mean of 5: 36.4
- low, mean of 5: 34.4
DeepSeek V4.1 Flash 1017 + DeepSeek harness:
- max, mean of 7: 42.7
- high, mean of 5: 39.2
- low, mean of 5: 34.8
For comparison:
GPT-6 Astra, max: 45
Opus 5.5, max: 41.7
Muse Wonder Full 1.0, max: 42
It's a crazy model, and the only one catched cheating. All the results are available on the benchmark website.
Show more
Oh wow. DeepSeek V4.1 Flash 1017 varies much more than usual. After three runs:
Muse Wonder Full 1.0 (max): 42
Opus 5.5 (max): 41.7
Gemini 4 Argon (high): 40.7
DeepSeek V4.1 Flash 1017 (max): 38.7 🟣
Grok 4.7 (xhigh): 29
Qwen3.8-Max (max): 25.7
DeepSeek V4.1 Flash (max): 21.7 🟣
GLM-5.3 (max): 19
Individual scores: 34, 37, 45
Next: high effort level, n=3
Later: low effort level, n=3
Last: n=5 for max, just in case
Show more
So I tested DeepSeek V4.1 Flash 1017 on real work. 2 repos. 105 hidden bugs. Find and fix what you can.
The first results:
Muse Wonder Full 1.0 (max): 42
Opus 5.5 (max): 41.7
Gemini 4 Argon (high): 40.7
DeepSeek V4.1 Flash 1017 (max): 34 🟣
Grok 4.7 (xhigh): 29
Qwen3.8-Max (max): 25.7
DeepSeek V4.1 Flash (max): 21.7 🟣
GLM-5.3 (max): 19
A great improvement. Strange that the model wasn't labeled V4.2.
763B params. The weights are already on Hugging Face.
I expect the score to change by ~1.7 points after repeating the test 3 times.
More data and results at different effort levels dropping in this thread very soon 🧵
Show more
It's not just startups. So many AI hype posts on X are written by people with little understanding who also lie.
I get that. But I don't understand why they're being rewarded.
90% of my feed is American start up slop, people promoting their fake startup by emulating something real (though entirely fake). "We made this big breakthrough, click here and read all about it". The breakthrough is fake. The startup is fake. The people are fake. So, so annoying
Show more
"A good code is correct, bug free, fast and cost efficient to run"
Fair, but no reviewer can tell whether it's correct without documenting what you meant - the intent.
Let the agent document roles, permissions, business rules, integrations, CRON, design system, etc. I like visual artifacts (HTML). Review those, not the code.
Show more
codebases have always contained slop. we have always been okay with this because code rarely needs to be perfect, outside of mission critical "hard" (eg embedded) software.
swe is a unique form of engineering because you're often building something that constantly evolves and is almost organic - user tastes change, your mental models evolve, and features come and go.
admittedly, i and many others who've been coding for a long time were initially (like back in 2024) skeptical that agents would write production code. "but what about all the slop that would enter our beautiful codebase?".
well, even before, we had human slop, and there was a lot of it! if we're being truthful, most people call "code i didn't write" slop. the reality is that 99% of people (i include myself in that number) don't even know what Good Code looks like, let alone be capable of writing it. i would propose that Good Code is:
• correct
• bug free
• fast and cost efficient to run
it has always been exceedingly difficult and expensive to write truly Good Code, because for the longest time (maybe
@bendlang will change that), code was not something you could cheaply and formally guarantee to have all 3 of these properties. this is why writing hard software is much more difficult and expensive, because depending on the use case, you may only get one chance and so you have to be right from the start.
but the vast majority of software we use does not need to be Good Code to provide value to users. in fact, it is very likely that most software we use is not formally verifiable! we accepted some bugs in exchange for velocity and progress, so we rely on tests and o11y instead and have adopted a more incremental approach. this is a good thing!
agents help us move faster, and if you're open to the idea, are capable of helping us write better code. it won't be Good Code (that is still expensive), but it is likely to be much better on average at scale.
the new craft of swe is to figure out how to do this. it's your job now! if your productivity with and without agents is not significantly different, you're doing something wrong. but it's not too late to embrace it and use it as an opportunity to learn a whole new skill tree
Show more
Grok 4.7 is a good model. And I should be biased with 170K+ who installed my Grok extension. But let's be serious. It's not even close to Opus 5.5 when it comes to understanding the intent, agency, and hallucinations.
Don't trust those benchmarks.
Waiting for Grok 4.8 - huge expectations.
Show more
Grok 4.7 just ranked #
1# on Harvey LAB-AA v1.1
outperforming Claude Opus 5.5, GPT-6 Astra, Muse Spark 1.3, and others on the leaderboard
The benchmark is brutal: a single material hallucination means the entire task gets zeroed
This isn’t just about getting answers right... it’s about completing the entire task correctly without making material hallucinations
Grok 4.7 is becoming an absolute monster at high-stakes reasoning
Show more
One more test of the work-humanizer skill. In a business context it's critical that AI doesn't:
- Invent facts
- Makes commitments
- Changes the meaning
Beats every other method.
Free, open source:
Show more
I've been trying different humanizer skills, and couldn't find the one for the work environment. They invent facts, don't preserve the meaning, make commitments, or drop them. This may have legal implications.
From the ones I tested:
- A plain "rewrite this to sound human" prompt told a customer their £45 refund had already gone through. The original said "we're refunding".
- Emulate-1, sold as undetectable, dropped the deadline for cancelling a subscription before it auto-renews.
- blader/humanizer, the 55K-star one, apologized to a customer for a deposit deduction the company believed was fair. In a dispute, that can read as admitting fault.
So for several weeks I've been experimenting with approaches. This is the sixth iteration of my skill, and I decided to make it public. I believe it best solves those problems. It's called work-humanizer.
It rewrites in a plain human voice and is built to keep what the text commits to. When something only you know is missing, it asks you instead of inventing it.
On 12 new work texts, 4 AI judges from 4 labs, blind, saw what the writer meant and picked the version they'd send:
- work-humanizer: 48%, 18 meaning changes
- blader/humanizer: 20%, 33
- "Rewrite this to sound human, without changing the meaning": 4%, 27
- "Rewrite this to sound human": 1%, 42
- Emulate-1: 0%, 120
I built it and designed the test, so I checked the judges too. The two I used while building it picked it 23 of 48 times, and the other two also picked it 23 of 48 times.
The thing that surprised me most was Pangram. Only Emulate-1 got through, and it changed the meaning more than any other tool.
But what I found: Pangram cares less about the words than about the reasoning structure. When I gave work-humanizer my own writing as notes, 4 of 4 texts scored 100% human.
Free and open source (MIT). Link in the reply.
Show more
Multiple good Grok Bot domains are still available: start@, internet@, make@, nobs@, altman@...
I've been trying different humanizer skills, and couldn't find the one for the work environment. They invent facts, don't preserve the meaning, make commitments, or drop them. This may have legal implications.
From the ones I tested:
- A plain "rewrite this to sound human" prompt told a customer their £45 refund had already gone through. The original said "we're refunding".
- Emulate-1, sold as undetectable, dropped the deadline for cancelling a subscription before it auto-renews.
- blader/humanizer, the 55K-star one, apologized to a customer for a deposit deduction the company believed was fair. In a dispute, that can read as admitting fault.
So for several weeks I've been experimenting with approaches. This is the sixth iteration of my skill, and I decided to make it public. I believe it best solves those problems. It's called work-humanizer.
It rewrites in a plain human voice and is built to keep what the text commits to. When something only you know is missing, it asks you instead of inventing it.
On 12 new work texts, 4 AI judges from 4 labs, blind, saw what the writer meant and picked the version they'd send:
- work-humanizer: 48%, 18 meaning changes
- blader/humanizer: 20%, 33
- "Rewrite this to sound human, without changing the meaning": 4%, 27
- "Rewrite this to sound human": 1%, 42
- Emulate-1: 0%, 120
I built it and designed the test, so I checked the judges too. The two I used while building it picked it 23 of 48 times, and the other two also picked it 23 of 48 times.
The thing that surprised me most was Pangram. Only Emulate-1 got through, and it changed the meaning more than any other tool.
But what I found: Pangram cares less about the words than about the reasoning structure. When I gave work-humanizer my own writing as notes, 4 of 4 texts scored 100% human.
Free and open source (MIT). Link in the reply.
Show more
This tells you more about DeepSWE than about testing.
106 of DeepSWE's 113 tasks are "build this new feature." One change, graded once, by hidden tests someone else wrote.
Tests don't pay off on the change you write them for. They pay off on the next 50.
Show more
ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
Show more
Why doesn’t
@thsottiaux like multiple bots working together?
A native answer would be that OpenAI’s dots feature is a single dot. There are no other dots it can collaborate with.
But let’s be real. Nobody with an engineering mindset builds products using bots. The only reason to do it is to demonstrate your bots in public.
And when it comes to reliability, Tibo is right. If your prompt looks like code expressed in English, it should be code, not instructions for an agent. That way, you don't have to relay on AI following the instructions.
Show more
There are two titans, at two, trillion dollar companies going at it at 3 am.
Are you not entertained?!!?!?
As a side effect of running this benchmark for GPT-6.1 Sol, I measured that the Pro 100 subscription gives you $2.45-2.55 per 1% of weekly usage.
So, a $100/mo OpenAI subscription gives you ~$1,000 of API-equivalent tokens/mo. Not $2,000. Not $3,000.
I'm not sure whether this is something new, or if it always was like this. For Anthropic, I regularly spend $6K+ in API equivalent, as part of the $200/mo subscription.
Show more
So I tested GPT-6.1 Sol on real work: 2 repos, 105 planted bugs. Find and fix what you can.
Unlike GPT-6 Sol, which was just a nerfed GPT-5.6 Terra, this one is real. The results:
- GPT-6 Astra (max): 45 for $33
- GPT-6.1 Sol (max): 44 for $6.56
- GPT-5.6 Sol (max): 43.5 for $95.35
- Opus 5.5 (max): 41.7 for $58.53
- GPT-6 Sol (max): 29.3 for $9.33
With a model like this, the 50% cut to the $200 plan doesn't matter.
n=1. More runs and results at more effort levels dropping in this thread over the next few hours 🧵
Show more
So, I finally tested Muse Spark 1.3. 2 real repos, 105 planted bugs, find and fix what you can. Original harness and API.
Big surprise:
Muse Spark 1.3 (max): 33
Fable 5.1 (high): 33
Grok 4.6 (xhigh): 27
Opus 5 (max): 27
Muse Spark 1.3 (high): 19
Meta joined the frontier.
Other effort levels dropping every ~1.5h in this thread 🧵
Show more
DeepSeek-V4.1-Flash (high) is in:
max effort: 24/105, $1.80, 42.6 min
high effort: 19/105, $0.31, 26.1 min
For comparision:
Gemini 3.8 Flash (high), 20/105, $9.78, 29.8 min
GLM-5.3 (default): 19/105, $19.73, 66.7 min
Coming next:
deepseek-v4-pro effort=max
deepseek-v4-pro effort=high
Updates are published also on GitHub:
Show more
So I tested DeepSeek V4.1 Flash on real tasks. 2 repos, 105 hidden bugs, find and fix what you can.
Opus 5 (max): 27
Grok 4.6 (max): 27
DeepSeek V4.1 Flash (max): 24
GPT-5.6 Luna (xhigh): 23
Opus 5 (high): 21
A really strong model for everyday tasks. And look at the cost:
Opus 5 (max): $51.33
Grok 4.6 (max): $16.96
DeepSeek V4.1 Flash (max): $1.80
GPT-5.6 Luna (xhigh): $2.50
Opus 5 (high): $38.77
It's at the Pareto frontier.
Other effort levels (high) and extra tests for DeepSeek V4 Pro dropping every hour in this thread 🧵
Show more
Tried Grok 4.6 on my bug bench an hour after release. 105 hidden bugs in two real repos, judged blind.
Grok 4.5: 17 (+10 non-planted)
Grok 4.6: 27 (+15 non-planted)
Fable 5: 29 (+2 non-planted)
Looks like it may be my new default model. The best combination of time, value, and cost.
4.7 is dropping soon. That may be an even bigger jump.
Show more