🚀Excited to release PACE: A Proxy for Agentic Capability Evaluation! Evaluating LLM agents on benchmarks like SWE-Bench and GAIA is expensive, slow, and infrastructure-heavy, often costing $$$ and taking hours or days per model. ❓But do we always need to run full agentic evaluations? In PACE, we show that agentic benchmark performance can be accurately predicted from a small, carefully selected set of cheap non-agentic benchmark instances. PACE automatically selects proxy instances from existing benchmarks covering skills like instruction following, planning, tool use, reasoning, coding, retrieval, and multimodal understanding. Across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks, PACE-BENCH achieves: ✅ 3.80% MAE for absolute score prediction ✅ 0.81 Spearman correlation for model ranking ✅ ~84% pairwise preference accuracy ✅ ~100× lower cost than target benchmark sampling Beyond prediction, PACE also reveals what capabilities different agentic benchmarks actually require, e.g., planning, verification, long-context aggregation, and instruction following. We hope PACE makes agentic evaluation cheaper, faster, and more accessible for model development, model selection, and routing :) 📃 Paper: 💻 Code: I'm incredibly grateful to have worked with @lintangsutawika, @Jiarui_Liu_, @lltjuatja, @JiayiiGeng, @lrzneedresearch, @daniel_js_lee, @Aditya_Soni_8, @Vincent92965015, @xiangyue96, and @gneubig .