I agree with @thorstenball, I think unit tests are dead in the water. The ones the models write are terrible, at best just doubling total LOC. Inverting the testing approach - heavy e2e/black box/golden master, reaching for lower levels only if necessary, works better for me

With the numbers from a day of benchmarking @typesafeai's Jev against our production judges... What Jev is good for Picking a label from a fixed list, when the list is defined by you and the evidence is in the input. That is the shape where it matched or beat the models we run today, at a fraction of the cost and latency. - Classifying sales-call transcripts into one of 32 call types: 35 of 36 correct on a synthetic set with 20 adversarial cases, versus 34 of 36 for our current small-model classifier. Zero label changes across three repeat runs. About 100 times cheaper per call. - Typing the relationship between two knowledge-graph nodes from a fixed vocabulary: passed all four of our hard quality gates on a 47-case gold corpus, one case behind production, at roughly 65 times lower cost and 15 times lower latency. - Categorizing Slack channels into four buckets: 16 of 16 on the held-out set, tied with a frontier model. - Yes/no questions about evidence that is visible in the input: in our tests, "does this item contain a visible defect" and "is this candidate a real customer signal" both worked. It is also fast and stable. Median latency was 100 to 200 milliseconds per call. Across roughly 3,700 calls we saw nine server errors and no throttling. Confident answers were identical run to run; only low-confidence answers flipped. The confidence score is the most useful thing about it. When it was wrong, its confidence was usually low. A frontier model told us it was 95 percent confident on both of its misses. What Jev is not good for Judging quality, or anything where the right answer depends on the model's own sense of what counts as good. - Judging whether an AI agent's output is good enough to publish, against a written rubric: 45 of 50 on our labeled fixtures versus 48 of 50 for a frontier model alone. On a held-out set weighted toward data-heavy outputs it dropped to 18 of 27 with nine false rejects, while production got 24 of 27 with none. Six rewrites of the question changed nothing. On 71 real production rejections it would have let 27 through. - Judging whether two decisions contradict each other: on 36 human-labeled pairs it found 2 of the 12 real contradictions. Ten of them scored a probability of 0.13 or lower. - Code review. A staged Jev-only reviewer we tested on 14 real merged PRs, with human review findings as ground truth, found none of the 13 human-identified problems on the PRs it could process and produced 24 low-severity "compatibility" and "test gap" notes across PRs that humans approved cleanly. The pattern across all of it: Jev works when a definition outside the model fixes the answer, such as a taxonomy or a checkable condition. It fails when the answer is a matter of judgment, because its prior about what is trivial, safe, or contradictory does not match ours and no phrasing moved it.

Increasingly feels like the architecture for working with LLMs is going to involve a subagent that does the work, and then a personality hire main agent whose job it is to speak to you in plain english like the little dummy you are.

Astra: "Caveats: A photon could have hit the RAM bank while calculating this answer, and so our data could be incorrect. Butterfly wings may be flapping in India. Mercury may be in retrograde, I have not verified astrological observations yet."

this is actually literally insane because it doesn’t appear to have any trade-offs. it’s just better engineering. the linked blog goes into detail about the implementation: mapping to CTIDs, various bitmaps, AVX, doing all this while they keep all the postgres constraints, etc.

If you're on the fence about SFRuby Conf they just gave me a $50 off coupon to share! Keynote from Ryan at Intercom/Fin. AFAICT Intercom is maybe the most effective software factory at scale in existence today so eager to hear what he has to share.

I think agent memory is like a kind of shitty glue over the work you _should_ be doing to build a software factory manually. Building new skills, loops, bots, etc. It _kind of_ works but ultimately falls down because agents aren't as good as you are.

For the last 3 months I've been telling all my clients to move everything to MCPs and I saw the light when @jsharkey showed me an internal "mcp proxy" they had built that was similar, and saw big co's like Ramp adopt the same "mcp of mcps" approach

Interesting to see Yegge say he never successfully built anything with Gas Town. In I mentioned not finding these ultra vibed orchestrators useful b/c reliability (w.r.t. completing tasks). Turns out the author of the most famous one had the same issue.

Watching an LLM just rip all-out on an autoresearch project is fascinating. Just thinking about all the human hours it would've taken to get this font file reduced by 5%... now available to you for the low cost of $7.82.

Prompt I use a lot: "how can I pokayoke this so this kind of error never happens again" pokayoke is using simple, low-complexity methods to make a particular kind of mistake impossible to make. classic one would be connector shape: make it impossible to insert "the wrong way".

Here's all of the models currently in the "frontier" discussion for interactive, human in the loop coding sessions. @ArtificialAnlys data. Takeaways: 1. GLM-5.3-Flash is cheap but extremely slow (high output tokens per task) 2. Sol dominates the time per task frontier 3. Gemini 3.7 flash is very similar to Sol. 4. Opus is smart and fast, but costs far more. 5. Grok is good but a step down from Opus and Gemini. 6. OSS models require ~10-20x as many output tokens per task, which means they're extremely slow. 7. For interactive HITL work, you should pick a sub at a frontier AI lab and use it. 8. Google/Spacex/OpenAI/Anthropic are close enough that you can squint and say they're the same 9. OSS models are capable and cheap or equivalent per-task but extremely, painfully slow because they require so many output tokens to succeed.

These two charts from @ArtificialAnlys explain why GPT-5.6 Sol (med) is the king of all models for interactive coding agent sessions right now. Sol(m) is 5x faster per task than Fable, 10x cheaper. It's even faster per task than Flash 3.7, despite having ~6x less tok/sec.