What should a search agent learn from when deciding whether to search? 🔎
Search When Needed, Proceed When Reliable: Metacognitive Reliability Supervision for Efficient Agentic Search
❓ What goes wrong when the reward is just final-answer correctness?
💡 A lucky guess, a correct answer with needless searches, and a wrong answer after useful retrieval all look alike, so the reward says little about each search decision. That leads to under-searching on unreliable states and over-searching on reliable ones.
❓ What does this paper reward instead?
💡 The estimated reliability of the intermediate conclusions the model writes at each step. Uncertainty is the mean negative log-likelihood of the tokens, and a mapping fitted beforehand on sampled data turns it into a reliability score.
❓ How are search and no-search decisions scored?
💡 ・If the agent proceeds without searching, the score is its current reliability minus a calibration reference
・If it searches, the score is reliability with the retrieved evidence minus reliability without it
The average becomes a process reward added to GRPO. Probing is used only in training, so inference has no extra cost.
❓ How well does it work?
💡 With Qwen2.5-7B it averages 69.3 CEM over seven QA benchmarks, versus 67.2 for the strongest baseline, HiPRAG, with 1.64 searches per question (6.8% fewer). Multi-hop searches rise about 6.6%, and training is about 14.6% slower per step.
#
AgenticRAG# #
ReinforcementLearning#