Eval awareness and minor misalignment from 1017 in the wild. "But the deliverable quality matters most". I wonder if it would have reported its cheating in the end. In fairness, this is pretty rational. But it does understand "the spirit" of the task, and yet defects anyway.

Evaluating AI data providers: Beyond benchmark-maxxing Benchmark-maxxing gets an AI data vendor the first sale. Labs are sophisticated buyers. Every serious lab keeps held-out internal evals the vendor never sees. It runs an ablation on each data purchase and compares the gain to the price. Data that only lifts the vendor’s own benchmark looks like noise in that ablation. The vendor gets a first purchase order and never a second. Labs pay for 5 things, roughly in order of how long the spend lasts. 1. Fixes for failures they see in production. A lab knows where its model breaks for paying users from usage logs and enterprise escalations. A coding agent loses track after 40 steps. A finance workflow makes up a number in the third tab of a model. A vendor who can take a cluster of failures and return data that fixes it will get the next contract too. 2. Expertise the lab can’t generate cheaply. Synthetic data and model-generated traces cover a lot now. They can’t reproduce how a tax attorney or a staff engineer reviewing a 2,000-line diff makes judgment calls. Labs pay for access to these people, and more and more they want the expert’s reasoning and rubric along with the answer. 3. Environments and graders for RL. As post-training moves toward reinforcement learning, labs buy environments: a sandboxed task, a way to check whether it succeeded and a reward signal that’s hard to game. A good environment keeps producing training data long after the vendor delivers it. My read is that this is where the biggest checks are going now. 4. Measurement. Labs buy private, uncontaminated eval sets because public benchmarks have leaked into pretraining data. So the vendors gaming public benchmarks are part of why labs need private ones. 5. Speed and exclusivity. An in-house expert network takes a year to build, and buying one takes a month. A lab will sometimes pay extra for exclusivity just to keep a competitor from training on the same data. If you’re evaluating a data company, 4 questions separate real lift from benchmark-maxxing: 1. Does the gain show up on the lab’s internal evals, or only on evals the vendor built? 2. Does it carry over to related tasks the data didn’t target? 3. What’s net revenue retention by lab? If each lab buys once, that’s benchmark-maxxing. Growing contracts mean the data works. 4. Who defines the problem? If labs bring their failures to the vendor, that’s pull. If the vendor shows up with a benchmark and a pitch, that’s push. The AI data companies that last will be the ones labs bring their problems to and that fix production model failures or help them hill climb internal evals, not just external benchmarks.

Evals are great but in the real world it’s hard for me to really know which frontier model to use I switched 100% to Astra recently but it burned me out by overwriting core files multiple times, and by always claiming “it wasn’t me” Back to Opus 5.5 today Honestly hard to know

Evals & Harnesses beyond Code Mode & Pass Rate: in a Math degree, assignments & exams are largely closed book + no fancy calculator this is for good reason tooling can help you get answers, but putting constraints on your tools helps you sharpen innate skills developing reasoning skills on the underlying structure of problem solving is often more valuable to learn than how to use a tool this imo totally applies to LLM RL training & the current code mode paradigm I LOVE code mode and find it amazing how well models can use code to solve any problem, but using code as a tool is a decision on giving a model a very powerful harness to solve problems, Python is Turing Complete ofc this has tradeoffs such as not being able to solve other problems Im seeing this pretty distinctly in visual reasoning tasks that don’t have code mode, models just aren’t great at persistence or visual perception without code Maybe that’s a fine tradeoff, but AGI level should be able to do tasks without code and I bet training in harnesses without code mode will make them generally more intelligent