When an LLM picks from fixed options, wouldn't it be handy to adjust after the fact whether it thinks first? 🤔
ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions
❓ What is a typed decision?
💡 A choice among a fixed set of options that code then acts on, such as routing a ticket or checking a rule. Reading option probabilities in one forward pass is fast, but it fails on multi-step decisions like date arithmetic.
❓ How does it solve this?
💡 With no training, one Qwen3 model plays two roles: System 1 answers without thinking, System 2 answers after thinking, both on the same cached prompt. Cutting one thought short and closing it also gives an accuracy curve for every budget.
❓ How well does it work?
💡 On Qwen3.5-27B, direct answers score 62.2% and always thinking scores 82.7%. The anytime policy reaches 81.6% using 72% of the thought tokens. At a 5% target, System 1 answers 24.5% of decisions with a realized error of 3.9%.
❓ Any caveats?
💡 The pre-registered savings criterion failed on every model. System 1 confidence is a weak signal for when to think, and calibration must be refit when the thinking budget changes.
#
LLMReasoning# #
Calibration#