TL;DR: The common claim that dense embeddings beat BM25 in code search depends heavily on how you evaluate. Against a fair BM25 the premium shrinks, and fusing dense with BM25 is the safer choice.
Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search
Key points
ă»đ§Ș Proposes RepoPremium, a repo-level benchmark (core: 1,265 queries over 10 Python repositories)
ă»đ Switching BM25 from chunks to whole files shrinks the Cur-Pre premium from +0.121 to +0.034
ă»đ€ With generate-then-grep, where an LLM writes identifiers first, BM25 scores 0.613 vs 0.532 for dense
ă»đ€ RRF fusion of dense and BM25 scores 0.609, above dense alone at 0.550
ă»đ§ About 30 routers were tried; none met the preregistered criterion
ă»đ€ In agent runs, swapping the search tool moves exact-match by only about 0.04
Caveats: mostly 10 Python repositories and localization-only evaluation; other languages use one repository each.
#
CodeSearch# #
RAG#