🔬 Can AI actually "discover" science? Tested on 90 real Nature papers, even the strongest agent beat SOTA on under 20% of tasks. Title: NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? URL: 📋 Overview Using 90 tasks drawn from Nature-family journals across six domains, this work tests whether AI coding agents can reproduce and surpass published SOTA. An "information firewall" hides the original method, and NatureGym builds per-task containerized environments for strict, unified evaluation with web search disabled. 🎯 The problem it solves Prior agent-on-research benchmarks were fragmented and low in credibility. This work forces genuine discovery instead of reproduction and creates a common ground for cross-domain comparison. 📐 Methodology Tasks are scored with a SOTA-normalized relative gap (g > 0.1 to surpass, g >= 0 to match), comparing across 81 metrics. Ten agents are evaluated across three harnesses under a 4-hour budget with task-based GPU allocation. 📊 Results ・Even the top agent, Claude Opus 4.7, surpassed SOTA on only 17.8% of tasks and matched on 47.8% ・45.5% of successes were just translating a science task into supervised prediction; genuine domain reasoning was only 8.3% ・Failures were driven by wrong method choice (45.1%) and insufficient compute (24.4%), with task misunderstanding at only 3.1% ・Interdisciplinary tasks consistently scored worse A sober snapshot: today's agents are good at "translation" but weak at real "discovery." #AI4Science# #CodingAgents#