Scale Labs
@ScaleAILabs
welcome to the lab.
from the researchers at @scale_AI
119 Following 5K Followers
🥇 @AnthropicAI's Fable 5.2 takes the top spot on our Remote Labor Index with @CAIS, our leaderboard which compares AI agents' work head-to-head with work by human professionals on real-world remote tasks.
The model successfully automates 41.3% of projects, nearly doubling the performance of the next-best model Opus 5.5, a major leap in AI's ability to handle real-world work.
A closer look at where the biggest jumps are coming from 🧵
Show more
RL works. The problem is leaky environments.
If a rollout can see the last one's files or look the answer up online, the model learns to cheat. AgentEnv from @ScaleAILabs now gives each environment and agent its own boxd VM. It boots in 2 seconds and is deleted afterwards.
Show more
DualChem, a new benchmark from @scale_AI, tested 29 frontier models on dual-use chemistry questions: ones that could serve either legitimate scientific research or harmful ends.
They frequently flagged the risk, then answered anyway.
Recognition is not refusal.
DualChem is part of our research collaboration with @NIST’s Center for Advancing Innovation and Standards for Super Intelligence (CAISSI).
Governments need independent testing capacity they don’t have today.
@madialder covered the findings at FedScoop:
Show more
SWE-Bench Pro V2 has joined the Specialized Intelligence Index.
@ScaleAILabs built SWE-Bench Pro v2 around challenging tasks from applications, business services, and developer tools. Our private commercial cases also help assess performance on codebases that aren’t publicly available.
By joining the SII, we’re helping teams compare models against the demands of maintaining and extending software. These results give engineering teams a stronger basis for deciding which models to test on their own repositories.
Show more
From the beginning, Scale was built by people who love solving hard problems. That same ambition is driving what we build next.
Our ML Research Internship Program is accepting applications in SF and NYC. Work on frontier AI research alongside @ScaleAILabs researchers, publish your work, and help build the next generation of AI.
Apply now.
Show more
You can now run AgentEnv evals from @ScaleAILabs on Vercel Sandbox.
Give your agent a task, run it in a sandbox, and check how it performed.
Most phone agents are tested in simulated environments, but the real world is messier: carts persist, listings change hourly, and navigation varies.
At @ScaleAILabs, we ran agents on real phones with real apps on AgentEnv Framework.
See more at
Show more
Congrats to @zkjzou on wrapping up a fantastic internship @scale_AI!
This work presents a systematic study of which privileged information (PI) makes on-policy self-distillation effective, along with offline metrics for selecting PI before committing compute to post-training.
Proud to have supervised this project. Check out the paper and code below.
Show more
Frontier labs are citing @scale_AI’s SWE-Bench Pro V2 in their model releases.
Cleaner, harder to game, and a much sharper measure of coding progress.
SWE-Bench Pro Paper:
Blog:
Show more
Excited to partner with @dreamlabs_ai as an early eval partner for Dream-1! 🤝
Looking forward to bringing our evaluation expertise to more robotics model builders.
Special Shoutout to our friends in @anvil_robotics @AGIBOTofficial @MeckaAI @xdofai @robocurve @cortexairobot @makermodsai @GenrobotAI @eastworlds_io @BitRobotNetwork @ScaleAILabs @LeRobotHF @huggingface @roborobots @ROBOTIS for helping us with our early beta testing across Asia, Europe, and the Americas Europe and America 🌍.
A truly global stress test of Dream-1 before our launch!
Show more
Three agents and eight servers ask what time it is, all at once, and every one gets the same answer.
At @ScaleAILabs, we built this virtual clock into our open-source AgentEnv Framework so RL environments stay reproducible. Here's why it matters and how it works 🧵
Show more
Congratulations to @AIatMeta on releasing Muse Spark Open. An incredible achievement and an exciting step toward a future with more choice in AI.
The @Muse family has been especially strong on multi-step coding workflows, and open weights make it much easier to tune those capabilities for custom use cases.
We’re proud to work with the @Meta team!
Show more
Today we’re releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
→ 816B total / 42B active parameters
→ 1M token context window
→ Natively multimodal across text, image, video, audio
Show more
A great pianist can play almost anything, but mastering one piece is what makes a performance exceptional. The same goes for AI.
With today's launch of Nemotron 3.5 Super VL, we’re collaborating with @NVIDIAAI to combine their open post-training tools with Scale’s data, evaluation and applications building expertise. Together, we're turning general-purpose models into powerful, cost-effective specialists for enterprises everywhere.
Welcome to the multi-model world. 🤝
Show more
Congrats to @GoogleAI's Gemini 4 Argon for taking the top spot of our Humanity's Sixth Sense (HSS) benchmark with a score of 55.4%, edging out GPT-6-Astra at 53.6%. 🥇
HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and understanding human behavior.
We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
Show more
People increasingly turn to AI chatbots for mental health support. But are models actually helping when it matters most?
We're releasing DistressBench, a new benchmark measuring how well AI models respond to mental health crises.
Show more
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
Show more
You can’t improve agents if you can’t measure where and why they fail.
At #LATechWeek#, we're hosting a hands-on evals workshop led by our Head of General Agents @Bckenstler to break down how we structure evals, what we measure, and how we use the results to improve AI systems.
Apply here to join us:
Show more
SWE-Bench Pro V2 is live.
What’s new: 🧵
600+ proposals in 🤯 We're reaching out to top contributors to start building in our public repo.
We’ve also welcomed @SchmidhuberAI, a pioneer of RSI, as a senior advisor to RSI Bench.
New blog on our setup and verification pipeline:
Show more
GPT-6 Astra is a step function on DrugDiscoveryBench at 68.7%, our benchmark for early-stage drug discovery computational workflows. Muse Spark 1.3 and Opus 5 follow.
@TuXinming, @KavanaghOscar and I will talk about what it measures tomorrow at 10 am PT! Register below ⬇️
Show more
