Make classic BM25 search smarter without rebuilding expensive neural indexes — by optimizing query rewriting one token at a time. A genuinely clever approach 🔎 Title: STORM: Stepwise Token Optimization with Reward-Guided Beam Search URL: 🔎 Overview STORM trains a query-rewriting model guided by retrieval quality. At each generated token, it scores candidate expansions against a BM25 index, concentrating exploration on the vocabulary that actually improves search. ❓ Challenges Solved Modern retrieval leans on dense and learned-sparse neural models, while lexical methods like BM25 are fast but weak on synonyms and paraphrases. ・Dense neural retrievers need expensive index rebuilds whenever the model changes ・LLM query rewriting tends to produce well-formed but retrieval-ineffective or harmful terms ・Training gives only delayed sequence-level feedback, obscuring which individual terms actually helped 💡 Methodology & Proposed Approach ・Self-supervised training via reward-guided beam search driven by retrieval performance ・At each token, candidate expansions are scored against BM25 and low performers pruned ・This turns retrieval metrics into token-level signals, focusing search on effective vocabulary ・Using BM25 indexes means no neural index rebuilding — infrastructure stays light 📊 Experimental Results ・0.6B-8B models match or exceed competitive LLM rewriters ・Maintains BM25's speed advantage ・The 8B variant rivals much larger proprietary systems ・Zero-shot transfer to 18 languages (MIRACL) beats dedicated multilingual dense retrievers on average 🌍 Use Cases It fits search stacks that want to avoid index-rebuild costs, and systems that need cheap multilingual boosts. Since it lifts performance while keeping an existing BM25 pipeline, it's an easy-to-adopt answer for teams running search in production. #Retrieval# #BM25#

I build small, practical AI systems. On-device chatbots without LLM APIs. Embeddings, BM25, small policy networks, explainable responses. 100+ free browser-side tools for AI, math, dev, security, SEO, and weird utility work. GitHub:

🦈 Before that press release goes live, why not test it against "hundreds of public voices" first? A slightly futuristic engine now simulates an entire crowd's reaction for $1 in 10 minutes. Title: aaronjmars/MiroShark URL: 📦 Overview MiroShark is a "Universal Swarm Intelligence Engine." For any scenario—a press release, a news headline, a policy draft, or a question—it simulates in real time how hundreds of AI agents would react. The agents post, argue, trade, and shift their positions as simulated time passes. ❓ Challenges Solved Organizations want to test how the real public will receive an idea before committing resources. MiroShark removes the need for lengthy focus groups and expensive market research, enabling validation for under $1 in less than 10 minutes. 💡 How It Works It runs in five phases. ・Generate an ontology from the input documents ・Build a Neo4j knowledge graph of entity relationships ・Ground 100+ personas using demographics, web enrichment, and graph attributes ・Have agents interact hourly across Twitter, Reddit, and prediction markets ・Generate reports that cite the actual simulated posts and trades Posts are ingested via NER, embeddings, and entity resolution, then retrieved by fusing vector, BM25, and graph traversal. 🎯 Use Cases ・PR crisis testing and market-reaction forecasting ・Ad campaign pre-testing and policy impact analysis ・Personal decision scenarios and historical counterfactuals You can also inject breaking news mid-run, or fork a running simulation (counterfactual branching). 📊 Highlights ・1.3k GitHub stars and 265 forks, AGPL-3.0 licensed ・Each simulation runs at roughly $1, about 10 minutes, with 100+ agents ・Python backend, Vue.js frontend, Neo4j database; LLMs via OpenRouter (local Ollama also supported) #AIAgents# #Simulation#