An interesting idea on calibration of LLMs from @swyx when they were discussing Jev and decision models in @latentspacepod: > There’s one element of calibration which is a rare topic that I don’t think people even knew about or understood. We covered it with, our conversation with Clémentine Fourrier of Hugging Face, and she used to run the evals, at Hugging Face, which is basically the idea that, models are attuned to give you the most likely next token. But, they’re gonna lie to you when you ask them, “How confident are you?” Because they’re just gonna give you the most likely next answer instead of, like, actually, like, no, let’s calibrate. Like, I am actually fifty percent sure, or I am twenty percent sure, and, like, let’s try to calibrate that. I would say, like, if anything, I think that actually that’s pretty easy to generate synthetic data around because you can sort of see the truth and then synthetically generate a bunch of answers, have Jev classify it, and then compare with ground truth. That would be my reverse engineering of this. It was very nice to hear this because we've been exploring something very similar in our recent work on ProbabilityBench: creating synthetic data where we have access to the exact ground-truth probabilities to evaluate decision models and other LLMs. Calibration metrics are becoming popular for evaluating decision models, but they don't directly measure how close predicted probabilities are to the ground truth probabilities. A model can be perfectly calibrated while still producing inaccurate probability estimates for individual instances. So I also feel evaluating (and/or training) on synthetically generated data with ground truth probabilities might be a good way to go. Previously we had to rely on toy experiments with Gaussian distributions where we can analytically derive the ground truth probabilities (which is not suitable for testing LLMs), or had to use benchmarks with crude soft labels, e.g., CIFAR-10H, Fashion-MNIST-H, and ChaosNLI. Jev already uses synthetic data, and they may be doing something similar internally. While we don't know the details, some experiments shared in X suggest there's still room for improvement: when asked about a fair six-sided die, Jev assigned an average probability of ~83% to its chosen outcome (instead of 17%), and ~92% for a fair coin outcome (instead of 50%). This highlights how challenging getting accurate probabilities can be, even in these very simple settings. (see details in our blog: ) It would be exciting to see more research in this direction!