Ofir Press
@OfirPress
10.3K Following 20.8K Followers
GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly more 'out of domain' than DeepSWE questions?
Show more
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
Show more
The best coding-agent harnesses solve half the tasks per attempt, but only a third every time.
@jeffweisbein asked on my launch post how often each harness solves the same task on all three attempts. That matters when an agent runs without anyone watching.
The Harness Leaderboard now shows it. It compares nine coding-agent harnesses on the same open models.
Averaged over three open models, mini-swe-agent (@KLieret) solves 51.4% of tasks per attempt, 68.6% at least once in three tries, and 33.0% every time.
Five harnesses solve 32.0% to 33.3% every time, too close to call. @openclaw solves 15.3%.
On Flint, the weakest model, Hermes Agent (@NousResearch) solves 28.6% at least once but only 10.0% every time.
Leaderboard:
Show more
We suddenly became twice as interested in the "appendix"... 🙃 #ICLR2027#
this type of personal, realtime modding is so cool. i think soon it'll make it to non-game programs too.
In 2004 Greasemonkey let you write js mods to any website. At Palm in 2008 i had a demo to let you mod any webOS app but couldn't ship it. This idea of modifying apps at runtime has always been stuck in my head.
Now you can just speak and alter the digital world around you.
Show more
mini-swe-agent is 130 lines of code and it keeps on winning in harness comparisons.
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard:
Harness Lens:
Show more
i was cheering people on from the sidelines of the nyc marathon and whenever i saw someone going fast i asked them to slow down, telling them that it's a marathon, not a sprint
Today we introduce Foresight-SWE 🦉: a benchmark that tests whether coding agents can catch their own bugs.
tl;dr: They don't!!
Across 11 models, coding agents solve developer-specified issues 60.7% of the time on average, but only 14.8% when they must additionally avoid bugs developers later fixed.
Show more
Congrats on the ProgramBench result!!!
👋Hy4 is here, new with vision.
Firmly among the leading open models.
More productive everywhere: finance, frontend, game, officework, research…
Built responsibly, with cyber/biosecurity in mind.
Try it out and let us know!
🔗：
Show more
Fun project led by @jowenpetty ! StateBench is benchmark for long-context state tracking abilities that we're already using internally to develop new Olmo models
Using an LM on a task it wasn't trained on:
🎬Opus 5.5 can make @OpenAI's new promo films entirely in code.
No video diffusion model, just three.js, GLSL, and an SDF heroine.
😉Got a harder case next time?
This is all being generated via code + Blender. I imagine that in 1-2 years we'll have Pixar level visual-generation abilities all via Python. Really cool!
Can coding generate 4D, like a dancing rabbit? 🐰
Opus 5.5 vs. Gemini 4 Argon with 3DCodeVerse, using Blender code. Opus is closer to the reference; Argon went plush toy. Which one wins?🤔
Show more
super timely work, and a really great explanation for why models might feel "benchmaxxed"!
SWE evals definitely feel extremely Markovian, which lends really well to compaction (since your codebase is essentially agent memory)
real-world agent use is clearly multi-turn and partially-observable (uncovering requirements once you see your own spec assumptions challenged and overturned)
Show more
In SwarmChat, you can see the communication methods of different model families, i.e. OpenAI vs Anthropic. Anthropic seems like a much more chill place to work as an AI agent 😀
Show more
"Slack" is a powerful way for agents to communicate: DMs, channels, threads, and files - generalizes many past setups (e.g. message tool, boards).
We built an open-source API-based Slack for agents and ran large-scale experiments on ProgramBench with pretty interesting results!
Show more
If an agent truly understands a video, can it reconstruct it?
Introducing BVB: benchmarking agentic video understanding via programmatic reconstruction in Blender.
288 real videos. 51 agent configurations.
Paper, demos & leaderboard:
Code:
Paper:
Show more
Great to see more teams starting to explore the agent swarm domain using ProgramBench!
Announcing new research on scaling multi-agent teams for long horizon software engineering tasks. 🤖🧩📈
Work led by @keenansamway, @mrdanieldsouza, Omar Darwiche Domingues, @mziizm, @yichern_tan, and @nouhadziri
Read the paper:
Check out the blog post:
Show more
so exciting to see our work SWE-fficiency used and some really incredible results on designing agents for codebase optimization!
one step closer to self-maintaining codebases :)
More than half of the men surveyed in a new Harvard report claimed to recognize AI leaders who were, in fact, totally fictional.
