Ofir Press
@OfirPress
10K Following 20.4K Followers
If an agent truly understands a video, can it reconstruct it?
Introducing BVB: benchmarking agentic video understanding via programmatic reconstruction in Blender.
288 real videos. 51 agent configurations.
Paper, demos & leaderboard:
Code:
Paper:
Show more
Great to see more teams starting to explore the agent swarm domain using ProgramBench!
Announcing new research on scaling multi-agent teams for long horizon software engineering tasks. 🤖🧩📈
Work led by @keenansamway, @mrdanieldsouza, Omar Darwiche Domingues, @mziizm, @yichern_tan, and @nouhadziri
Read the paper:
Check out the blog post:
Show more
so exciting to see our work SWE-fficiency used and some really incredible results on designing agents for codebase optimization!
one step closer to self-maintaining codebases :)
More than half of the men surveyed in a new Harvard report claimed to recognize AI leaders who were, in fact, totally fictional.
My brother developed a method that allows LMs to explain why one chess move is better than another. The LM does this by writing code that analyzes the board, the LM knows how to write this code since it gets access to 200k board states from real games along with engine analysis.
Show more
LMs can play chess, but they can't really explain their moves well (chess engines can't at all). Our new BinProp scaffold tackles this, and nearly triples LMs’ score on our chess-explanation benchmark. It works on other games too!
Try it on your own games on our site-🧵👇
Show more
my brother just sent me this lol
When we were at human-level AI we could take human tasks and translate them into benchmarks. In many domains, we can't do that anymore, those tasks are too easy.
Here are my guidelines on how to build benchmarks in this new era:
Show more
To push the AI frontier and make AI more useful for humans, we need to understand and then formalize the gap between ‘what agents can do’ and ‘what humans need’ via benchmark building. This is becoming harder than ever, but there’s definitely still a gap there so there’s work for us to do.
This is an amazing, stressful, confusing and inspiring time but not an unprecedented time. Horses can carry more weight than you or I, for longer distances, electric grids and radio networks transmit energy or information faster than any human could, and vaccines protect us in a way no human ever could. We’ll figure this out too.
Show more
1. Use tasks that would take teams of humans years or decades to solve
2. Find real-world tasks that real-humans had to actually solve, don't just make stuff up or have humans make stuff up for you
3. Find tasks that can be scalably crawled from the web
Show more
1. Use tasks that would take teams of humans years or decades to solve
2. Find real-world tasks that real-humans had to actually solve, don't just make stuff up or have humans make stuff up for you
3. Find tasks that can be scalably crawled from the web
Show more
When we were at human-level AI we could take human tasks and translate them into benchmarks. In many domains, we can't do that anymore, those tasks are too easy.
Here are my guidelines on how to build benchmarks in this new era:
Show more
why does astra provide such incredibly long explanations when i ask it to explain something? the value of an explanation is *lower* the longer it is.
maybe we should make a ShortQuickAndToThePoint-Explanation-bench
Show more
congrats @finkd on the super strong programbench results!
Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.
We built an all-synthetic simulation of a hospital that can be shared publicly without any issues. Our data quality is so high, physicians cannot tell the difference between real/synthetic patients. The data is fully verified -- perfect for RL.
Show more
mini-swe-agent beats native harnesses on the new SWE-Serve benchmark
🚀 New benchmark alert: SWE-Serve (
Can AI agents develop inference engine and make it serve real models?
Our team at @nvidia built 53 tasks from real @sgl_project engineering work to find out. The chart shows why live-serving tests matter. 🧵
Show more
One of the last programmers still programming. Congrats Tim!!
Releasing our runtime dynamic compression framework that achieves 1.5-2.0 bit compression at high quality. This is integrated into the bitsandbytes2 library, which starts as a private beta today.
Paper:
Private beta signup:
Show more
awesome to see our work featured again in another @AnthropicAI release, and some incredible multi-agent results in the Opus 5.5 model card evaluating how a swarm of 100 agents coordinates to solve ProgramBench!
love how agents are becoming bigger computational threads: wondering if we'll need agentic system concurrency primitives as multi-agent really starts scaling
Show more
thank u for the kind words <3
Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family.
It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.
Cool multi-agent experiments in the 5.5 system card!
They're getting a solve rate of close to 100% on their subset of 166 ProgramBench instances (out of 200), so I think they're not using the tougher programs (FFmpeg, the PHP compiler, ...) since we're seeing a very low solve rate on those.
Show more
Multi-agent runs on ProgramBench in Opus 5.5 system card
Multi-agent systems get to the same outcomes as a single-agent, but quicker.
Worth noting that just 166/200 tasks were used. I still strongly encourage using all 200! The long tail of hard tasks is really difficult.
Show more
another win for mini-swe-agent vs codex & claude code. mini-swe-agent is consistently the best harness for evals and post-training. great work by @KLieret
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：
Show more
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：
Show more
kinda weird that we're already more than a few hours into this week and no new millennium problems have been solved.
