註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Alvin Sng
@alvinsng
MTS at @FactoryAI; previously at: Meta, Airbnb, Brex & Opendoor
加入 March 2009
618 正在關注    5.9K 粉絲
The model providers optimize for raw benchmark scores or they optimize for price per token. Neither captures what customers actually care about: the cost to efficiently complete the work that actually exists. That’s why we built Legacy: a benchmark representative of the real distribution of software work, and why we evaluate models on the cost to complete those tasks, not just their raw scores.
顯示更多
Today, our benchmark, Legacy-Bench, joins @FireworksAI_HQ’s Specialized Intelligence Index. Frontier models have blind spots for legacy code, what’s benchmarked is improved. Legacy-Bench tests how well AI models can debug, extend, and migrate historic systems, to drive the future of software.
顯示更多