Make classic BM25 search smarter without rebuilding expensive neural indexes — by optimizing query rewriting one token at a time. A genuinely clever approach 🔎 Title: STORM: Stepwise Token Optimization with Reward-Guided Beam Search URL: 🔎 Overview STORM trains a query-rewriting model guided by retrieval quality. At each generated token, it scores candidate expansions against a BM25 index, concentrating exploration on the vocabulary that actually improves search. ❓ Challenges Solved Modern retrieval leans on dense and learned-sparse neural models, while lexical methods like BM25 are fast but weak on synonyms and paraphrases. ・Dense neural retrievers need expensive index rebuilds whenever the model changes ・LLM query rewriting tends to produce well-formed but retrieval-ineffective or harmful terms ・Training gives only delayed sequence-level feedback, obscuring which individual terms actually helped 💡 Methodology & Proposed Approach ・Self-supervised training via reward-guided beam search driven by retrieval performance ・At each token, candidate expansions are scored against BM25 and low performers pruned ・This turns retrieval metrics into token-level signals, focusing search on effective vocabulary ・Using BM25 indexes means no neural index rebuilding — infrastructure stays light 📊 Experimental Results ・0.6B-8B models match or exceed competitive LLM rewriters ・Maintains BM25's speed advantage ・The 8B variant rivals much larger proprietary systems ・Zero-shot transfer to 18 languages (MIRACL) beats dedicated multilingual dense retrievers on average 🌍 Use Cases It fits search stacks that want to avoid index-rebuild costs, and systems that need cheap multilingual boosts. Since it lifts performance while keeping an existing BM25 pipeline, it's an easy-to-adopt answer for teams running search in production. #Retrieval# #BM25#

Most of the attention on full-text search goes to BM25. Query a part number, get that part number back first. A query like "why does PROD-001 overheat under load" has a literal string and a question in it though. That's where text-match filtering comes in. It narrows results to the documents that contain the literal string, and semantic search ranks what's left, all from one index. Full-text search is GA in Pinecone Database, text-match filtering included. 🔗

I build small, practical AI systems. On-device chatbots without LLM APIs. Embeddings, BM25, small policy networks, explainable responses. 100+ free browser-side tools for AI, math, dev, security, SEO, and weird utility work. GitHub:

Great RAG paper from IBM. There are some really good ideas on how to solve common RAG issues. It's well known that retrievers chunk long documents by length, which discards the hierarchy the document already has. So they propose using a table of contents. A table of contents helps to encodes exactly the global structure that chunking throws away. STAIR uses that table of contents as the addressing scheme for a generative retriever, so the model stores and retrieves information from its own parameters against a structure the corpus supplies. On SearchTome, it reaches Recall@1 of 82.6 percent against 76.9 percent for a fine-tuned Differentiable Search Index, a statistically significant gap, with BM25 at 59.5 percent and DPR at 68.7 percent. Hallucination stays below 0.05 percent, which is the standing objection to generative retrieval and the reason grounding the address space in a real hierarchy is worth the extra structure. The ablations also show it generalizes where very few training samples exist. Paper:

Sentence Transformers v6 has been released, and it is centered around Multi-Vector Embedding Models, what does that mean? Usually, when you build a semantic search or RAG-based system, you've probably used the @OpenAI embedding API or similar, as it allows to quickly index and search through a set of documents. Such embedding models are called "bi-encoders", as they typically encode 2 things ("bi"): encode each document or chunk, encode the query, and then compute pairwise cosine similarity between each (document, query) pair to retrieve the top-k ones. This is known as vector search or semantic search, and typically achieves better results compared to traditional keyword search, which relies on the classic BM25 algorithm. In 2020, researchers at @Stanford came up with something better than bi-encoders, called ColBERT. The idea is to create many vectors for each document, and many vectors for each query. One creates a vector for each token (a word or part of a word) of the query, and a vector for each token of each document. Next, to find the best matches, one computes the so-called MaxSim similarity. This is illustrated in the animation below. Unlike traditional vector similarity metrics that operate on pairs of single vectors, MaxSim computes similarity between sequences of vectors. The key insight here: each query token finds its best match in the document, then we sum. This enables fine-grained semantic matching and avoids averaging. These "multi-vector" embedding models are also called "late-interaction" models, as they keep multiple vectors for each document and query, delaying the matching step until the very end. Over the last year, researchers at @LightOnIO trained some very impressive late interaction models, which they openly released on @huggingface. Today, the Sentence Transformers library, which is the go-to library for open-source embedding models, added first-class support for them, as well as for a variety of SOTA late interaction models trained by @mixedbreadai, @liquidai and more. This means that you can now also train or fine-tune your own late interaction models very easily. Various vector databases like @elastic and @qdrant_engine already include support for multi-vector embedding models. This integration will definitely boost adoption of multi-vector embedding models by the industry. Read more here: - ColBERT paper: - Blog:

🧠 Claude Code agents start every session with amnesia. This post solves that without adding a dedicated service, using the Elasticsearch you already run. TL;DR: A CLI called bridge and three hooks automatically store an agent's decisions, context, and tasks in Elasticsearch, then recall them across sessions and devices via hybrid search plus temporal decay. Title: Persistent memory for agents: Claude Code on Elasticsearch URL: Points ・🗂 Seven indices: memory/messages/tasks/sessions/status/entities/entity-history store memory by dimension ・🪝 Three hooks automate it: SessionStart syncs, markdown writes get indexed, Stop logs the session end (no explicit calls) ・🔎 Hybrid recall: BM25 fused with semantic_text dense vectors via RRF, catching both exact task IDs and conceptual matches ・⏳ Temporal decay: a default 45-day DECAY ranks recent memories higher (needs ES 9.3+ or Serverless) ・🕸 Knowledge graph: extracts blocked_by and friends from markdown frontmatter, surfacing blockers at depth-2 traversal ・📡 Offline resilience: writes queue locally as JSON and flush via bulk API once connectivity returns ・💻 Cross-device: gen-handoff produces a handoff JSON so another machine restores context with no git pull If you already run Elasticsearch, the pragmatic appeal is no new service and reusing your existing monitoring and backups. Setup is three commands: git clone and #Elasticsearch# #ClaudeCode#

The real enemy of building a search pipeline is "stitching together disparate tools" 🔍 Here's an open-source framework that unifies ingestion, retrieval, and evaluation. Title: Introducing Search Toolkit URL: 🔍 Overview The Mistral Search Toolkit is a composable, open-source framework that streamlines production search pipelines for AI applications. It integrates ingestion, retrieval, and evaluation into one unified system. ❓ Challenges Solved Building a production-grade search pipeline is harder than it looks. ・Organizations spend enormous time integrating disparate tools ・As a result, they can't focus on actually improving search quality 💡 Methodology & Features It's built from three components. ・Ingestion: process multiple data sources with configurable pipelines handling parsing, chunking, and embedding generation ・Retrieval: offers BM25 sparse search, dense embedding-based search, and hybrid configurations ・Evaluation: built-in metrics including recall, precision, MRR, and NDCG to measure each configuration You can run the whole ingestion → retrieval → evaluation flow in one framework. 🌍 Use Cases ・Enterprise search across wikis, repositories, and file storage ・RAG systems that want to measure retrieval quality in isolation ・Domain-specific retrieval for legal or medical content ・Agentic systems needing reliable indexed search alongside live data It's production-ready and already deployed across financial services, manufacturing, public sector, and media. #Search# #RAG#

🦈 Before that press release goes live, why not test it against "hundreds of public voices" first? A slightly futuristic engine now simulates an entire crowd's reaction for $1 in 10 minutes. Title: aaronjmars/MiroShark URL: 📦 Overview MiroShark is a "Universal Swarm Intelligence Engine." For any scenario—a press release, a news headline, a policy draft, or a question—it simulates in real time how hundreds of AI agents would react. The agents post, argue, trade, and shift their positions as simulated time passes. ❓ Challenges Solved Organizations want to test how the real public will receive an idea before committing resources. MiroShark removes the need for lengthy focus groups and expensive market research, enabling validation for under $1 in less than 10 minutes. 💡 How It Works It runs in five phases. ・Generate an ontology from the input documents ・Build a Neo4j knowledge graph of entity relationships ・Ground 100+ personas using demographics, web enrichment, and graph attributes ・Have agents interact hourly across Twitter, Reddit, and prediction markets ・Generate reports that cite the actual simulated posts and trades Posts are ingested via NER, embeddings, and entity resolution, then retrieved by fusing vector, BM25, and graph traversal. 🎯 Use Cases ・PR crisis testing and market-reaction forecasting ・Ad campaign pre-testing and policy impact analysis ・Personal decision scenarios and historical counterfactuals You can also inject breaking news mid-run, or fork a running simulation (counterfactual branching). 📊 Highlights ・1.3k GitHub stars and 265 forks, AGPL-3.0 licensed ・Each simulation runs at roughly $1, about 10 minutes, with 100+ agents ・Python backend, Vue.js frontend, Neo4j database; LLMs via OpenRouter (local Ollama also supported) #AIAgents# #Simulation#