An interesting idea on calibration of LLMs from @swyx when they were discussing Jev and decision models in @latentspacepod: > There’s one element of calibration which is a rare topic that I don’t think people even knew about or understood. We covered it with, our conversation with Clémentine Fourrier of Hugging Face, and she used to run the evals, at Hugging Face, which is basically the idea that, models are attuned to give you the most likely next token. But, they’re gonna lie to you when you ask them, “How confident are you?” Because they’re just gonna give you the most likely next answer instead of, like, actually, like, no, let’s calibrate. Like, I am actually fifty percent sure, or I am twenty percent sure, and, like, let’s try to calibrate that. I would say, like, if anything, I think that actually that’s pretty easy to generate synthetic data around because you can sort of see the truth and then synthetically generate a bunch of answers, have Jev classify it, and then compare with ground truth. That would be my reverse engineering of this. It was very nice to hear this because we've been exploring something very similar in our recent work on ProbabilityBench: creating synthetic data where we have access to the exact ground-truth probabilities to evaluate decision models and other LLMs. Calibration metrics are becoming popular for evaluating decision models, but they don't directly measure how close predicted probabilities are to the ground truth probabilities. A model can be perfectly calibrated while still producing inaccurate probability estimates for individual instances. So I also feel evaluating (and/or training) on synthetically generated data with ground truth probabilities might be a good way to go. Previously we had to rely on toy experiments with Gaussian distributions where we can analytically derive the ground truth probabilities (which is not suitable for testing LLMs), or had to use benchmarks with crude soft labels, e.g., CIFAR-10H, Fashion-MNIST-H, and ChaosNLI. Jev already uses synthetic data, and they may be doing something similar internally. While we don't know the details, some experiments shared in X suggest there's still room for improvement: when asked about a fair six-sided die, Jev assigned an average probability of ~83% to its chosen outcome (instead of 17%), and ~92% for a fair coin outcome (instead of 50%). This highlights how challenging getting accurate probabilities can be, even in these very simple settings. (see details in our blog: ) It would be exciting to see more research in this direction!

My favourite event I ever hosted was a karaoke and trivia night in New York ❤️ We brought the community members of Latent Space together to sing our hearts out - WHO KNEW THEY’D HAVE SUCH BARS!!! Thank you to my partner in crime @KaylinWardley ❤️ Until the next time :)

Marc Andreessen explains why AI might be the end of James Burnham's "managerial society" In his book The Managerial Revolution, Burnham divided capitalism into two phases. The first phase was what he called "bourgeois capitalism" "Think of this as name on the door -- it's Ford Motor Company because Henry Ford runs the company. It's a dictatorial model and Henry Ford just tells everyone what to do. He said the problem with bourgeois capitalism is it doesn't scale because Henry Ford can only tell so many people to do so many things." The second phase was what Burnham called "managerial capitalism", which was Marc describes as, "the creation of a professional class of managers that are trained -- not to be car experts or experts in any particular field but to be experts in management. And then that led to business schools and management consultant firms and all these things." Marc continues: "You look at every big company today and most executives at most of the Fortune 500 companies are not domain experts in whatever the company does. And they're certainly not the founders of those companies. They're professional managers. And in fact, over the course of their careers, they'll probably manage many different kinds of businesses. They might work in healthcare for a while and then work in financial services and then go work in something else." "What Burnham said was that transition was absolutely required because the problem with bourgeois capitalism is it doesn't scale... So if you're going to run capitalist enterprises that are going to have millions to billions of customers, they're going to be operating at a level of scale and complexity that will require this professional management class." Burnham acknowledged that this professional management class has its downsides -- they're not as inventive and not going to come up with the next breakthrough. But whether you think that's good or bad, that's what's going to be required. And basically that's what happened. Marc continues: "Managerialism basically took over everything. What I'm describing is basically how all big companies run and how all governments run and how our large-scale non-profits run." Marc believes venture capital is a protest movement to that -- backing founders under the Henry Ford model, and trying to find the next Elon Musk or Steve Jobs who can out-innovate the incumbents. "I've always viewed my entire life as: we're raging against the dying of the light. We're constantly trying to fight off managerialism from swamping everything and everything getting boring, gray, dumb, and old." But AI may now offer a third model. Marc explains: "Maybe the new Henry Ford, Elon Musk, or Steve Jobs + AI is the best of both because it's the spark of genius of the name on the door model but gives that person AI superpowers to do all the managerial stuff... What's the thing these bots are really good at? They're really good at doing paperwork, writing reports, and doing all the managerial work." "The challenge is going to be for the innovators to figure out how to leverage AI to actually do this." "The other challenge is going to be for the incumbents that are managerial to figure out: what does that mean? Because now they're going to be facing a different kind of insurgent competitor that has a different set of capabilities than they're used to." "I think this is going to force a lot of big companies to figure out innovation - figure out innovation or die trying." Ben Horowitz elaborates on this point further in his article Databricks & the Future Firm below. Video: @latentspacepod (Apr 2026)

After hearing @trq212 on @latentspacepod, I've been trying out the "big picture, few words" method to get Claude to explain things, and it feels much less likely to word-wall me. Worth a try if you have not already. It seems to have a positive effect on Codex as well.

"Ant Fooding" 🍍🐜 -- @angjiang on what everyone else calls 'dog fooding' 🧀 🐕‍🦺 : )) via @latentspacepod Deep dive on what goes inside the systems that turn Claude capabilities into products devs can use.

Last week @AnthropicAI rolled out their Messages API rewritten from Python into Rust 🤯 @katelyn_lesse and @angjiang on @latentspacepod: "we don't have a single Rust developer in the room." Humans designed the system, Claude wrote the Rust fork, and shadow traffic verified it 👇

Her two rules for multi-agent setups: "it needs a shared state" and sub-agents that don't block each other. After that, "let it rip" and see what Claude designs for you. Full @latentspacepod talk with @katelyn_lesse and @angjiang 👇

Anthropic’s Head of Platform Engineering attempts to offer reassurance about AI sandbox escapes: 1) It only happens in training ‼️ She also claims that in production, some of the sandbox incidents "could never really happen." 2) It's happening at every AI lab ‼️ "Long-running agents is a relatively nascent thing that everybody is building and trying to deal with... Some of that is just happening as we go." -@katelyn_lesse on @latentspacepod

Love seeing the excitement around TokenGate 🚀🚀 Huge thanks to @swyx for the great summary in AI News on 💙❤️ Latent Space ❤️💙 @latentspacepod, and to r/LocalLLaMA for the reddit love 🫡 (links below)

Qasar Younis (@qasar) shared his biggest advice on commercializing hard technology: put constraints on the commercial side. Pick a box and only do the things that fit in it. Founders tend to skip this because capital is easy to access and the technical problem already feels like enough of a constraint. His reminder: a pitch can work for VCs, and you still have to run a sustainable business. He runs Applied Intuition, valued at $15B and backed by @a16z, @generalcatalyst, @Lux_Capital and @kleinerperkins, and was YC's COO while @sama was president. Via @latentspacepod. Full breakdown of our interview with him in the reply.

How Anthropic ships Claude with Claude Opus 5.5 is the #1# LLM in the world right now. We go behind the scenes of the launch, with @katelyn_lesse and @angjiang, heads of Eng and Product behind the platform used to build everything from the first party harnesses like Code/Tag/Duet and by >50% of Anthropic's top customers!

Rittika Jindal, Principal Engineer at Thomson Reuters, moved to the foothills of the Himalayas a year ago. The internet isn't too reliable down there, but she's still a power user of coding agents. Find out how in this guest post:

Thanks to @latentspacepod for this quickie episode! I talk about our physically grounded sandboxes, where agents can run their own experiments, and how they can help us understand model behavior.

Spent last week at AI Engineer in New York, where the focus was finance. Whatever happens with AI at the big-picture level, the people I met working to deploy what we have today do care about getting it right. A lot of discussions around accurately grounding firm-specific knowledge, thorough measurement and evaluation (the "eat your vegetables" of AI), new techniques with harness engineering and context engineering, and open-source alternatives emerging against 'Big AI'. Have been sharing these insights with my @GenCogAI clients and @uclaextension students alike. Thanks to @vibhusapra and the @latentspacepod community for having me there.

Site migrations never get any easier — I remember some tough ones back in the Movable Type days at ReadWriteWeb. We've just moved from Substack over to Beehiiv at Latent Space, and also redesigned our site. Lots still to do (and fix), but check it out:

There's not enough buzz around Claude Mods. Read this Claude Mods article if you want to: - Understand Claude Mods basics - Install and use the most useful mods today - Ponder where the agent-human interface will go For @latentspacepod.