TL;DR: The common claim that dense embeddings beat BM25 in code search depends heavily on how you evaluate. Against a fair BM25 the premium shrinks, and fusing dense with BM25 is the safer choice.
Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search
Key points
ใป๐งช Proposes RepoPremium, a repo-level benchmark (core: 1,265 queries over 10 Python repositories)
ใป๐ Switching BM25 from chunks to whole files shrinks the Cur-Pre premium from +0.121 to +0.034
ใป๐ค With generate-then-grep, where an LLM writes identifiers first, BM25 scores 0.613 vs 0.532 for dense
ใป๐ค RRF fusion of dense and BM25 scores 0.609, above dense alone at 0.550
ใป๐งญ About 30 routers were tried; none met the preregistered criterion
ใป๐ค In agent runs, swapping the search tool moves exact-match by only about 0.04
Caveats: mostly 10 Python repositories and localization-only evaluation; other languages use one repository each.
#
CodeSearch# #
RAG#