These new benchmarks keep exposing two things: first, open source models lack high quality data and a strong RL environment; second, OpenAI and Anthropic have invested heavily in data quality and in making tasks more difficult and diverse. I expect Gemini Carbon and future Gemini models to be at the same level.
Introducing FrontierPhysics - a benchmark evaluating agents for end-to-end physics research
Our core team from physics labs at Berkeley, Caltech, and Harvard recruited 42 physicists from 26 institutions. 28 of them published papers on Nature, Nature Physics, PRX, PRA*
1/7 🧵