No it's not, you need calibration to make it useful for decision making. Next time you go to ICML/ICLR/NeurIPS, do not walk away from those calibration posters.
LLM next-token prediction is already a probabilistic classifier, so an open LLM can serve a Jev-like interface: discrete options in, fast probabilities out. Our own @ekzhang1 made it better at the job with a $5, 10-minute run on Tinker.
Show more
