Register and share your invite link to earn from video plays and referrals.

Akshay 🚀
@akshay_pachaar
Simplifying LLMs, AI Agents, RAG, and Machine Learning for you! â€ĸ Co-founder @dailydoseofds_â€ĸ BITS Pilani â€ĸ 3 Patents â€ĸ ex-AI Engineer @ LightningAI
Joined July 2012
512 Following    292.6K Followers
Meta just upgraded k-means after 70 years. (runs 10-45x faster on GPUs) the new variant, e-means, gets to the loss k-means reaches after 25 epochs in about 2, and it runs 10-45x faster on GPUs. classic k-means (Lloyd's algorithm) assigns every point to its nearest centroid, then moves each centroid to the mean of its points. one full pass over the data gives you exactly one update. mini-batch k-means updates from small batches instead. it moves fast early on but settles at a worse loss, which is why Lloyd stayed the default for decades. e-means keeps the mini-batches and fixes the reasons they fall short. it answers three questions: "how far should a centroid move after each batch?" each centroid keeps a running average of the points assigned to it. early batches move it a lot so it finds its region quickly, and later batches move it less so it settles. "what happens to a cluster nobody joins?" some centroids drift into empty regions and stop receiving points. every few batches, e-means splits a crowded cluster to give those centroids useful work again, and this step mattered more than any other in the paper's tests. "which batch size and step size should i pick?" earlier mini-batch methods needed a hyperparameter sweep for every new dataset. e-means sets both values directly from the number of points, clusters, and epochs, using scaling laws fitted over 75k training runs. each epoch costs about the same as a k-means epoch, and e-means simply needs far fewer of them. the chart in the image below shows its loss dropping under both k-means baselines within the first couple of epochs. at scale, it built 16 million clusters from 10 billion embeddings in under a day of GPU time. this matters well beyond clustering, since k-means builds vector database indexes, product quantization codebooks, and data curation pipelines. so before you throw another 25 epochs of Lloyd at your embeddings, consider e-means. it already ships with Faiss, and the paper is attached below if you want to read more. 🔗
Show more