๐ฌ Even the top AI agents leave about a third of scientific code repair tasks unsolved within a one-hour budget.
Title: ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments
URL:
๐งฉ Overview
AItonomy, Qwen, and PhAI-Labs built ScienceIDE, infrastructure that turns 27 scientific codebases, spanning astrophysics to ocean modeling, into executable environments AI agents can actually learn from. It ships 64 environments and 2,812 tasks.
โ๏ธ Problem it solves
Scientific code encodes decades of expertise, but fragmented toolchains and undocumented numerical conventions make it hard to turn that knowledge into reliable learning experience.
๐ Methodology
"Scientific checks" comparing numerical tolerances and conserved quantities validate task correctness, across 7 task categories like repair and implementation. For RL, a "truncation masking" trick prevents unfairly penalizing trajectories that hit the budget cutoff.
๐ Results
Under a one-hour budget, Fable 5.1 leads the agent leaderboard at 67.1%, with Astra at 63.1%. Models trained on ScienceIDE experience (PhAI-IDE) gain up to +33.33 points on scientific code repair, and RL training lifts reward on LAPS tasks 2.4x.
#
AIAgents# #
AI4Science#