Alvin Sng(@alvinsng):The model providers optimize for raw benchmark scores or they optimize for price per token. Neither captures what customers actually care about: the cost to efficiently complete the work that actually exists. That’s why we built Legacy: a benchmark representative of the real distribution of software work, and why we evaluate models on the cost to complete those tasks, not just their raw scores.

2026.09.24 20:12

The model providers optimize for raw benchmark scores or they optimize for price per token. Neither captures what customers actually care about: the cost to efficiently complete the work that actually exists. That’s why we built Legacy: a benchmark representative of the real distribution of software work, and why we evaluate models on the cost to complete those tasks, not just their raw scores.

Factory@factory

2026.09.24 19:38

Today, our benchmark, Legacy-Bench, joins @FireworksAI_HQ’s Specialized Intelligence Index. Frontier models have blind spots for legacy code, what’s benchmarked is improved. Legacy-Bench tests how well AI models can debug, extend, and migrate historic systems, to drive the future of software.