TypeSafe AI(@typesafeai):who benchmarks the benchmarkers? a good eval is: - ephemeral. it snapshots a moment in time and capabilities. - grounded. it directly measures performance on tasks and data you care about. - specific. it operates on the desired results and not some proxy of them. take notes!

2026.10.18 17:43

who benchmarks the benchmarkers? a good eval is: - ephemeral. it snapshots a moment in time and capabilities. - grounded. it directly measures performance on tasks and data you care about. - specific. it operates on the desired results and not some proxy of them. take notes!

Tim Neumann@TimNeumann84165

2026.10.15 21:44

Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑

504

Forward to community