Struggling with pgvector's HNSW returning fewer results under filters, or with heavy indexes? Title: Introducing pg_pinecone: Open-Source Search for Your Postgres Data URL: 🐘 Overview An Apache 2.0 open-source Postgres extension. It adds vector search powered by Pinecone's search engine to your existing pgvector columns. It runs inside the database, so your data never leaves Postgres. ❓ Problem HNSW needs more memory and build time as data grows, and filtered searches, such as per tenant, can return fewer results than requested. 🧩 Approach Instead of a graph, it uses IVF with RaBitQ codes (about 1 bit per dimension) and searches in three steps: probe nearby clusters, estimate distances from the codes, then rerank with full vectors. Filters matching 3% of rows or fewer use exact pre-filtering; wider ones are checked inside each cluster. Beyond the native engine, which needs no API key, you can add remote, embed(), BM25 and rerank(). 📊 Results (vendor-reported) On 1M vectors with 768 dimensions: ・1.27 ms latency at 0.95 recall (HNSW halfvec: 1.71 ms) ・136 MB index size (HNSW halfvec: 2.0 GB) ・1% filter: 0.981 recall at 8.3 ms (HNSW halfvec: 0.924 at 56 ms) These come from one dataset and one setup, and large bulk loads need a REINDEX. #Postgres# #VectorSearch#

💾 Billion-scale vector search without keeping everything in memory. 📰 Title: HFresh: Memory-Efficient Vector Search 🔗 URL: Weaviate introduces HFresh, a new disk-based vector index built to handle billion-vector datasets on constrained memory budgets. Highlights 🧩 Partition-based architecture Instead of connecting every vector in one giant graph like HNSW, HFresh splits vectors into small regions called postings. An in-memory centroid HNSW narrows down the right region, then only the relevant postings are read from disk. 📦 Two-stage quantization Centroids use RQ8, cutting memory about 4x while preserving routing accuracy. Postings use RQ1, compressing up to 32x versus 32-bit floats to minimize disk I/O and storage cost. 🔄 Rebuild-free background maintenance Split, Merge, and Reassign, based on the LIRE protocol, keep the index fresh continuously with no full rebuilds required. On the 1M-vector DBpedia dataset, HFresh used just 239MB of heap versus 6.67GB for uncompressed HNSW, about 28x less. It has also been proven at 1 billion 256-dim vectors, making it a strong fit for memory-constrained, large-scale deployments. #VectorSearch# #Weaviate#

TL;DR: Pinecone released VQ-bench, a benchmark that treats today's zoo of vector quantization methods as combinations of a small set of shared primitives, making it possible to compare them fairly under the same conditions. Title: VQ-bench: a Composable Vector Quantization Framework URL: Points 🧩 Defines composable primitives like Center, Normalize, PCA, and RandomRotate that any quantizer can be built from 🔗 Existing methods like E-RaBitQ reduce to just 4 chained primitives: Center, Normalize, Random Rotation, Angular Cast 📊 Benchmarks 14 quantizers on 5 VIBE datasets using Reconstruction MSE, Recall@10, and encode time 🥇 PQ and OPQ consistently achieve the lowest Reconstruction MSE ⚡ EDEN encodes far faster than PQ, OPQ, and E-RaBitQ while keeping recall competitive 🛠️ Adding a new quantizer often takes just a few lines of code, and a new primitive automatically composes with every existing one Putting fragmented quantization methods on the same evaluation footing should make it much easier to pick the right one for your use case. #VectorSearch# #VectorDB#

🎨 Every studio has a "final_final_v7.png." It's often faster to remake old work than to find it, and AI helps with that from an unexpected angle. In creative work, assets pile up faster than anyone can organize them. The catch is that keyword search fails: search "blue environmental concept" and you get nothing if the file is named ENV_ALTSTYLING_DARK_V4. The words and the filenames simply don't share a vocabulary. So flip the approach. Convert images and footage into vector embeddings that represent meaning as numbers. Now "things that mean something similar, however differently phrased, sit close together in vector space," so you can search in natural language. Built on Weaviate, hybrid search combines vector proximity with metadata filters (project, date, type) to land on the right asset. Search tens of thousands of concept images for "a cold, forested biome that feels visually distinct," pull B-roll by shot characteristics, dig up an old drum texture, all without knowing the naming convention. Here, AI isn't the star that generates new content; it's the infrastructure layer that makes existing work accessible, returning to creation the hours lost to organizing and searching. Building Foundry: AI isn't replacing creativity, it's removing friction makes the case for that quiet but essential value. 🔗 #VectorSearch# #CreativeAI#

Vector search that works at 10M vectors shouldn’t need a redesign at 10B. LanceDB hit 18.05ms p50 / 21.61ms p99 across 10B vectors, while scaling indexing and letting each query tune precision. New deep dive from @Yah01_

Here are 5 distance metrics in vector search. But how do you choose the right one? • L1 (Manhattan): sum of absolute differences, exact kNN only with no HNSW support • L2 (Euclidean): straight-line distance, the safe default for most models • Cosine similarity: angle between vectors, magnitude ignored • Dot product: same ranking as cosine on normalized vectors, less compute • Max inner product: dot product without the normalization constraint Most teams default to cosine and move on. That works until your model outputs non-normalized vectors, and suddenly dot product or max inner product is the better fit. Scoring formulas and config details in the blog.

We benchmarked Lakebase Search on 1M to 4M passages to understand how vector search performs as the index grows. Here's what we learned 👇 Cache size relative to the vector index mattered more than row count. When the cache roughly matched the index, 95–98% of page reads avoided storage and warm vector search p50 latency stayed between 11 and 23 ms. As datasets grow, sizing the cache alongside the vector index helps maintain low-latency search.

had an agent pull a 21-day-old architecture decision out of vector search and rebuild an entire module around a pattern we explicitly killed. vector similarity was high, so the model treated it as canon. the newer decision used slightly different phrasing, so it ranked lower in the query. because the memory had zero lineage attached, the agent couldn't tell whether the note was five minutes old or three weeks dead. without provenance metadata (timestamps, tool IDs, model versions) similarity search is just a confident liar. storing tool IDs and timestamps per write in sqlite fixed it. stale facts get expired by tool lane before retrieval even touches them.

The hard parts of hybrid retrieval, already done. Elasticsearch Vector Database is a new serverless offering where expert-level tuning is the default: - bfloat16 storage: half the disk footprint before quantization even starts - BBQ: up to 32x vector compression, 95% less memory - Auto-calibration re-tunes quantization on every merge as your data drifts - Filtered vector search at up to 8x higher throughput than OpenSearch - Jina AI embeddings and reranking on managed GPU inference, or bring your own models You bring documents and queries. We handle the embeddings, tuning, and infrastructure. Full breakdown, including the semantic_text quickstart and what ships in vectorDB index mode:

Sentence Transformers v6 has been released, and it is centered around Multi-Vector Embedding Models, what does that mean? Usually, when you build a semantic search or RAG-based system, you've probably used the @OpenAI embedding API or similar, as it allows to quickly index and search through a set of documents. Such embedding models are called "bi-encoders", as they typically encode 2 things ("bi"): encode each document or chunk, encode the query, and then compute pairwise cosine similarity between each (document, query) pair to retrieve the top-k ones. This is known as vector search or semantic search, and typically achieves better results compared to traditional keyword search, which relies on the classic BM25 algorithm. In 2020, researchers at @Stanford came up with something better than bi-encoders, called ColBERT. The idea is to create many vectors for each document, and many vectors for each query. One creates a vector for each token (a word or part of a word) of the query, and a vector for each token of each document. Next, to find the best matches, one computes the so-called MaxSim similarity. This is illustrated in the animation below. Unlike traditional vector similarity metrics that operate on pairs of single vectors, MaxSim computes similarity between sequences of vectors. The key insight here: each query token finds its best match in the document, then we sum. This enables fine-grained semantic matching and avoids averaging. These "multi-vector" embedding models are also called "late-interaction" models, as they keep multiple vectors for each document and query, delaying the matching step until the very end. Over the last year, researchers at @LightOnIO trained some very impressive late interaction models, which they openly released on @huggingface. Today, the Sentence Transformers library, which is the go-to library for open-source embedding models, added first-class support for them, as well as for a variety of SOTA late interaction models trained by @mixedbreadai, @liquidai and more. This means that you can now also train or fine-tune your own late interaction models very easily. Various vector databases like @elastic and @qdrant_engine already include support for multi-vector embedding models. This integration will definitely boost adoption of multi-vector embedding models by the industry. Read more here: - ColBERT paper: - Blog: