"we chased after benchmarks, when none of the benchmarks measure whether humans actually enjoy working with the model" We actually DO have benchmarks for the user experience of coding agents! Check out SWE-Together by @yifannnwu et al. It replays real user-agent interactions and measures not just pass rate but user effort: the number of corrective feedback turns needed to keep the agent on track.

We're publishing over 365,000 open and agentic RL Environments for SWE, terminal, and search agents The open research ecosystem has produced many great datasets for the three main agentic domains - software engineering, terminal use, and web research - but every one of them ships with its own harness, its own image conventions, its own grading scripts, and its own failure modes. We integrated them all. 23 tasksets behind one API, one sandbox lifecycle, one command. 365,000+ tasks in total, ~198,000 software engineering tasks across 20+ languages ~28,600 terminal tasks ~137,600 search tasks Ready for evals and RL training on Prime Intellect infrastructure, with validated and cleaned dataset re-uploads where the originals needed fixing.

Thanks for trying the task. This is really cool to see! It highlights two related questions: how independently an agent can complete a task, and how human input can improve the result. To evaluate autonomy fairly, we anchor the human simulator to the original user’s actions. Your result also raises a broader question: what makes AI genuinely helpful? Task score may miss differences in simplicity and elegance. Beyond autonomy, we need to understand how well AI uses human input to produce better outcomes, and how to define and measure effective human-AI collaboration.

Thanks @guohao_li for sharing—and for building SETA! It gave us a practical RL environment for training an open-weight memory policy with SFT + GRPO on Terminal Bench tasks. Also grateful to the open-source projects that made this possible: @harborframework for the agent harness and @rllm_project for the training infrastructure.

Really excited to see @humansand training models from their long-horizon interactions with people to make AI more human-aligned. We strongly share this vision, which is why we built SWE-Together, a benchmark reconstructed from 11K+ real user–agent coding sessions that evaluates agents not just on whether they finish the task, but on how well they stay aligned with user intent. If interaction with people is the training signal, interaction quality should be the eval. Would love to see models trained with this recipe tested on SWE-Together 🚀 📄 💻

Excited to share what we’ve been building at Meta Superintelligence Labs! Today we’re launching Muse Spark 1.1, our strongest model yet for complex agentic workflows — delivering massive gains in agents, computer use, coding, multimodal reasoning, and multi-agent orchestration. This is also our first API release! We’d love to hear your feedback. And we’re just getting started, super proud of the team behind it, and the larger models are training right now. 🚀

This matches exactly what we saw building SWE-Together. For real-world coding tasks, a fixed test suite rarely captures the whole truth. Generated tests can be too strict or low-coverage, too tied to the reference patch, or too shallow about whether the feature actually works. And a failing run can come from environment noise rather than a true model capability gap. So we added an agentic judge: we freeze a weighted behavioral rubric from the task spec, user intent, and oracle trajectory, then run it in a fresh task sandbox to inspect the patch, workspace, and tests against those goals. Separating model capability from eval noise takes careful design and audits. The next hard problem is maintenance: benchmarks seem to saturate every few months. We need better ways to refresh tasks reliably and automatically. How should the field approach this? See more at

This is a great step toward a cognitive theory of LLMs: maybe they don’t just compute tokens, but form a kind of internal workspace. But cognition is not just representation. In brains, what matters is also control: what gets routed, what gets stored, what gets suppressed, and what gets acted on. That is where LLM analogies to brain memory still feel thin. Relatedly, our work and others have found modularity in RNNs/LLMs that parallels modular organization in the brain. The open question is no longer whether these parallels exist, but what mechanisms actually make them useful. @Jack_W_Lindsey @GuangyuRobert @pengrui_han [1] Gu et. al., Sci. Adv. 2024, Representation induces modularization [2] Yang et al. Nat. Neuro. 2019, Task representation causes compositionality [3] preprint 2025, Multi-task load induces modularization. [4] preprint 2026, modularization in LLM representation.

For years the most popular coding benchmarks rank models on whether they can finish a fully-specified task on their own. Our new benchmark, SWE-Together, instead turns that one-shot test into an interactive session, and scores the agent both on how well it solves the problem and on how much steering it takes to get there. To measure that, we collect 11,260 recorded sessions, filter for those with genuine multi-turn feedback, real agent-authored edits, and verifiable outcomes, and rebuild the survivors into reproducible tasks, where each task is reconstructed in a sandbox with the repo pinned at its original commit and the user's first message as turn one. A reactive LLM user simulator then replays each session. It stays anchored to the original user's intent but speaks only when the agent's own trajectory calls for it (e.g., a clarification, a correction, a new requirement) instead of firing on a fixed schedule, so the corrections are something the agent draws out rather than a script we impose. SWE-Together measures two things: 1. Final correctness — whether the final repo (after the user interventions) does what the user actually asked. 2. User Correction — how much the user had to steer to get there. We also track Intent Coverage, a check that the simulator put the same underlying requests to every agent, so differences in correction reflect the agents and not an inconsistent simulator. As the existing one-shot scores saturate, how little an agent makes you intervene — and how well it ends up where you actually meant — can be the new signal for which model is worth using, and that's what we built SWE-Together to measure. Benchmark page:

Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the final code. But real coding help is a conversation — users clarify goals, add constraints, and correct course along the way. SWE-Together turns real coding work into a reproducible, verifiable benchmark: 109 repo-level tasks curated from 11,260 recorded sessions, replayed with a reactive LLM user simulator that preserves the original user’s intent. We evaluate agents as collaborators, not just patch generators: final pass rate and how many user interventions were needed to get there. In this evaluation snapshot, claude-opus-4.8 currently leads among the 7 agents we tested — achieving the highest pass rate while requiring the fewest user interventions. 📄 Paper: 💻 Code: 🌐 Website:

OPD for multi-turn scenario ( Teacher determines whether intervention is needed and how confident it is in the supervision, and these are used as weighting factors.

I really enjoyed @willccbb's theory-grounded post on the post-training landscape and where OPD fits: A simple framing I liked: -SFT gives dense imitation, but lacks on-policy correction. -RL gives sparse but verifier-grounded correction. -OPD gives dense on-policy correction when the teacher signal is reliable and compatible. Our work pushes this one step further: can we get the benefits of OPD without requiring same-family, token-level, logit-based teachers? We show the answer is yes. By replacing token-logit matching with chunk-level semantic verification, we enable dense on-policy distillation from black-box teachers, with stabilizers that help prevent collapse. More details in our paper: