TL;DR: e-means optimizes k-means with EMA and matches Lloyd's 25-epoch loss in about 1.4 to 2.6 epochs. No hyperparameter sweep needed.
e-means: outperforming k-means (in one-tenth the epochs)
Key points
ใปโก Reaches Lloyd's 25-epoch loss in roughly 1/10 of the epochs (as reported)
ใป๐ EMA-updated counts and sums per cluster; centroid read out as S/N
ใป๐ Cosine-annealed step size
ใป๐ฉน Dead clusters revived by splitting every 5 batches (the biggest effect)
ใป๐ Decay and batch size set by scaling laws, no sweep
ใป๐ DINO10B (10B embeddings) split into 16M clusters in about 2.5 epochs
ใป๐ฅ๏ธ 9.5-45.5x wall-clock speedup over Flash-KMeans
The scaling laws were mostly fitted on DEEP, so accuracy can drop in extreme settings.
Clustering large embedding sets just got much cheaper.
#
MachineLearning# #
kmeans#