TL;DR: e-means optimizes k-means with EMA and matches Lloyd's 25-epoch loss in about 1.4 to 2.6 epochs. No hyperparameter sweep needed.
e-means: outperforming k-means (in one-tenth the epochs)
Key points
・⚡ Reaches Lloyd's 25-epoch loss in roughly 1/10 of the epochs (as reported)
・🔁 EMA-updated counts and sums per cluster; centroid read out as S/N
・🌊 Cosine-annealed step size
・🩹 Dead clusters revived by splitting every 5 batches (the biggest effect)
・📐 Decay and batch size set by scaling laws, no sweep
・🚀 DINO10B (10B embeddings) split into 16M clusters in about 2.5 epochs
・🖥️ 9.5-45.5x wall-clock speedup over Flash-KMeans
The scaling laws were mostly fitted on DEEP, so accuracy can drop in extreme settings.
Clustering large embedding sets just got much cheaper.
#
MachineLearning# #
kmeans#