Hallucinations are hard to measure because reality is messy. So we built our own world. Introducing HalluWorld 🌎, a controlled benchmark for LLM hallucinations. Now accepted to #NeurIPS2026# Evaluations & Datasets! 🧵

🙌 All things data at @PatronusAI and @gracegongGG's panel session on evals tonight! Everything from SpeedrunBench (how fast an agent can finish a game) to benchmarks for world models and even evals for agent actions in spreadsheets with embedded formulas and charts (!!). 📊📈

We're excited to share that our paper on prompt infilling for dLMs has been accepted to the "Non-Autoregressive Language Models for Fast & Flexible Text Generation" workshop at COLM 2026 🎉 Masked diffusion language models like LLaDA and Dream have bidirectional architectures, yet they can't infer prompts from desired outputs out of the box. We traced this to a training gap: during SFT, prompts are simply never masked. The fix is straightforward. Mask both prompts and responses during SFT (full-sequence masking). That single change unlocks the ability to infer task-adapted prompts from few-shot examples. Prompt infilling in dLMs just requires the right training approach. We're calling on the community to release full-sequence masked SFT checkpoints alongside standard ones. Congrats to @akkikiki and Keisuke Sakaguchi! Paper: Fine-tuned models:

We're hosting a researcher night next Wednesday at our office in SF. Fast-paced spotlight talks on post-training and evaluation. ⚡️ The talks: @mariyaivasileva on Beliefload: an evaluation of how LLMs formulate and revise hypotheses in light of new evidence, and how they diagnose and repair a broken environment without overwriting the parts that already work. @Nick_saban20 on GlobeBench: a benchmark for whether language models can faithfully simulate the environments we train agents in. @AnmolGulati06 on Beyond Rows to Reasoning: an agentic framework for reasoning over and editing enterprise spreadsheets with millions of cells, cross-sheet dependencies, and embedded charts. @akkikiki on SpeedrunBench: a benchmark that asks not whether an agent can finish a game, but how fast. Drinks, sushi, merch incoming.

I’m joining @runwayml to work on AI video generation and world models. when I first used runway in 2022, it felt magical. even then it felt like a glimpse of a very different creative process. and since then, AI video has gone from something experimental to something that feels like a real medium. since college, I spent a lot of time thinking about human-computer interaction and how you give people agency over something like design. I think video is one of the richest creative formats we have, but also one of the hardest to work with. fast forward to 2026 and it feels like exactly the right moment to build interfaces that make powerful generative systems feel expressive, controllable, and useful. getting to hear that vision directly from @c_valenzuelab, @agermanidis, @kamilsindi, and others made the decision to join them easy. here's to the next chapter! 🎉

True story: our original company name when we were in stealth was Zeno AI, not Patronus AI. Back in 2023, Rebecca and I started the company on the idea that AI evaluation was like Zeno’s Paradox. Every time models reached the bar we had set, the frontier of what we needed to evaluate moved further ahead. Fast forward to 2026, I still think Zeno's Paradox is the most accurate way to describe AI progress. Every finish line we draw for AI moves the moment it reaches it. In the original Zeno's Paradox, every time Achilles reaches where the tortoise stood, the tortoise has moved a little further on, so he appears to chase forever. Mathematically the paradox does resolve. The shrinking gaps sum to a finite length, and Achilles does pass. In the AI version of the paradox, the idea is that humans are the tortoise, and AI is Achilles. We start far ahead, carrying millions of years of evolutionary and cultural learning. And over the last few years, Achilles has closed most of the visible distance. But the reason the AI version is more interesting is that the tortoise keeps speeding up. Each time AI closes a gap, it hands the people defining the frontier a better tool for defining the next one. Mathematicians use the newest models to probe harder open problems. Researchers use them to help design the very evals the next generation of models will struggle against. Everyone is anxious about the rate of progress in AI right now, but ultimately I think we’re going to land in a pretty optimistic place: AI will enable humans to reach farther than ever. @danshipper has a nice GIF of this tortoise/Achilles setup, see below.

Everyone is freaking out about the rate of progress in AI right now, but ultimately I think we’re going to land in a pretty optimistic place: AI will enable humans to reach farther than ever. Several very talented mathematicians were inevitably involved in the current Millennium Problem solutions, so AI became an extension of their own skills. I think this is like Zeno’s Paradox at play - each time AI closes a gap, it hands the people defining the frontier a better tool for defining the next one.

Meet Abdelrahman Madkour, from our research team. 👋 Madkour was a previously a Research Scientist at Meta, where he worked on coding benchmarks for coding agents. Before that, he completed his Ph.D. in CS at Northeastern University, where he focused on procedural content generation in games. Fun fact: Madkour plays the Arabic oud 🪕

I don’t often write up my thoughts, but here’s a new blog post summarizing some of my recent musings on RSI and verification: Recursive Self-Inflation. A lot of it is informed by my own research experience and by grappling with the fact that the strongest recent RSI results depend, and quite reasonably so, on fast, automatic feedback: we generate candidates, evaluate them, retain the ones that work, and keep iterating. The problem, however, is that every practical verifier has holes in it, and sustained optimization will eventually find and exploit them. That matters a bit differently in a recursive loop than in a static eval. In a static eval, a verifier mistake just gives you the wrong score. In an RSI loop, an accepted mistake becomes the seed for the next round, so errors can compound over time while the curve still looks like the model is progressing. I recently went through evidence of verifier failure across coding and science benchmarks, reward hacking, model judges, and formal proofs, and tried to characterize what I think credible RSI results would need to report, and what they need to rule out. As always, feedback welcome!

Today, we’re releasing SpeedrunBench: the first benchmark that measures how fast agents can beat video games. We asked frontier models not to simply complete video games, but to speedrun them - across 10 titles like Mario Kart and Pokemon. On simpler games, agents get close to the human world record. However, on more complicated games like Pokemon Blue, the best models Kimi-K3 and Opus 5 are ~4x off the world record. Benchmark, paper, and demo below. @PatronusAI

Meet Zhe Li, from our research team. 👋 Previously, Zhe lead post-training at Inflection AI (now Microsoft). Before that, Zhe was a key contributor to FaceID and Apple Vision Pro at Apple. Zhe did his PhD in computer science at the University of Iowa, where he worked on neural network optimization. Fun fact: Zhe is a naturally curious person and will happily go down a rabbit hole researching the most random things.

Tbh I’m not surprised OAI is pulling out of Cursor - Elon admitted under oath in April that xAI was partially distilling OAI data. And now OAI traces are directly flowing to SpaceX after the Cursor acquisition - Rumors are that Astra is a major step up from previous models, so cutting Cursor distribution probably won’t hurt OAI that much. Developers will use Astra anyway even if higher friction initially - Vertical integration, i.e. you want our models, you use our platform. I mean OAI doesn’t want to depend on NVIDIA for chips, so why would they depend on Cursor for developer distribution in the long run? Ecosystem strategy is how you win like Apple Btw I expect Chinese labs to eventually pull out of Hugging Face for similar reasons, but I’ll save that for another day

Lots of people have been asking me about the trend of acquiring enterprise data. As the founder of a data research lab, everyday I get pitched by companies trying to sell data. This includes non AI companies looking to monetize user data, data resellers and defunct startups. These are not “tasks” or “datasets”. They are messy artifacts that require a skilled researcher to manufacture into a useful end-product. Realistic data provides great raw material. It does not prevent reward hacks, unfair graders or the many other defects impacting “quality”. What the industry is lacking is not data brokers. It’s research expertise.

Today, we're open sourcing the first computer-use dataset for design. We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks. FigmaTrace dataset, paper, and blog below :) @PatronusAI

We just hired the creator of Fruit Ninja and Jetpack Joyride. Excited to share that Duncan Curtis is joining us as SVP of Product and Operations at @PatronusAI. Previously, Duncan was Head of Product at Zoox, SVP at Sama, and Product Lead at Google Play. He’s also the one to blame if you lost time playing Fruit Ninja back in the day… Designing LLM post-training data is oddly similar to designing games. A great game has simple rules, but creates a huge space of possible strategies. It has to be challenging without being impossible, and the challenge has to actually reveal skill. You can tell the difference between someone who has memorized a trick and someone who has actually mastered the system. Great RL tasks work the same way. You’re designing a simulation world with rules, tools, and feedback, and you’re testing whether agents can complete tasks through good decision making and NOT reward hacking. As models get better, I think agent task design will increasingly look like game design... the same principles that make games compelling can make RL envs powerful.