注册并分享邀请链接,可获得视频播放与邀请奖励。

Alvin Sng
@alvinsng
MTS at @FactoryAI; previously at: Meta, Airbnb, Brex & Opendoor
加入 March 2009
618 正在关注    5.9K 粉丝
The model providers optimize for raw benchmark scores or they optimize for price per token. Neither captures what customers actually care about: the cost to efficiently complete the work that actually exists. That’s why we built Legacy: a benchmark representative of the real distribution of software work, and why we evaluate models on the cost to complete those tasks, not just their raw scores.
显示更多
Today, our benchmark, Legacy-Bench, joins @FireworksAI_HQ’s Specialized Intelligence Index. Frontier models have blind spots for legacy code, what’s benchmarked is improved. Legacy-Bench tests how well AI models can debug, extend, and migrate historic systems, to drive the future of software.
显示更多