TL;DR Advanced encoders were losing to older models on sparse retrieval for one surprising reason: a vocabulary mismatch. Fixing it with just 20k adaptation steps sets a new BEIR state of the art.
Title: Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
URL:
Key points
🔤 Raw case-sensitive vocabularies were splitting single meanings across redundant surface forms
🧩 The proposed Vocabulary Transfer method migrates vocabularies in 3 lightweight steps
📊 ModernBERT-VT hits nDCG
@10 52.4 on BEIR, a new SOTA
💥 A collapsed RoBERTa-large jumps from BEIR score 1.4 to 51.3
⚡ Uses under 0.2% of the original pretraining token budget
🧪 Near-optimal performance reached in just 500 MLM steps
🧬 Also works for domain-specific vocabularies like chemistry
It's a nice reminder that what looked like an architectural ceiling turned out to be a fixable vocabulary design problem.
#
InformationRetrieval# #
LLM#