Chad Wahlquist(@chadwahl):I posted this because we need to focus on making real benchmarks of how these models perform in real-world scenarios on real data to evaluate their efficacy. Just like the factory is really the product, the ability to make proper evals and benchmarks is actually the important thing. It's too easy to be distracted by the gamed public benchmarks.

Chad Wahlquist

@chadwahl

Forward Deployed Architect - @PalantirTech

Joined November 2008

823 Following 25.8K Followers

Chad Wahlquist@chadwahl

2026.09.17 23:36

I posted this because we need to focus on making real benchmarks of how these models perform in real-world scenarios on real data to evaluate their efficacy. Just like the factory is really the product, the ability to make proper evals and benchmarks is actually the important thing. It's too easy to be distracted by the gamed public benchmarks.

Chad Wahlquist@chadwahl

2026.09.01 19:05

less benchmarking, more bench-making