Struggling with pgvector's HNSW returning fewer results under filters, or with heavy indexes? Title: Introducing pg_pinecone: Open-Source Search for Your Postgres Data URL: 🐘 Overview An Apache 2.0 open-source Postgres extension. It adds vector search powered by Pinecone's search engine to your existing pgvector columns. It runs inside the database, so your data never leaves Postgres. ❓ Problem HNSW needs more memory and build time as data grows, and filtered searches, such as per tenant, can return fewer results than requested. 🧩 Approach Instead of a graph, it uses IVF with RaBitQ codes (about 1 bit per dimension) and searches in three steps: probe nearby clusters, estimate distances from the codes, then rerank with full vectors. Filters matching 3% of rows or fewer use exact pre-filtering; wider ones are checked inside each cluster. Beyond the native engine, which needs no API key, you can add remote, embed(), BM25 and rerank(). 📊 Results (vendor-reported) On 1M vectors with 768 dimensions: ・1.27 ms latency at 0.95 recall (HNSW halfvec: 1.71 ms) ・136 MB index size (HNSW halfvec: 2.0 GB) ・1% filter: 0.981 recall at 8.3 ms (HNSW halfvec: 0.924 at 56 ms) These come from one dataset and one setup, and large bulk loads need a REINDEX. #Postgres# #VectorSearch#

Struggling to describe your interior style to a search bar? The UK's leading home furnishings retailer, Dunelm, built a conversational ‘Shopping Agent’ to serve as your digital concierge, powered by Google Cloud tools and Gemini. See how Dunelm is redefining retail AI ↓