Recent research has highlighted the promise of scaling memory embeddings in LLM training. While Engram and STEM index memory by token identity or local n-grams, can we design more flexible memory routing that captures how each token’s meaning changes with context? This paper introduces Mixture-of-Memory Embeddings (MoME), which uses a learned router to sparsely select among multiple memory slots for each token. The architecture outperforms strong memory baselines across three model families. Interestingly, both qualitative and quantitative analyses show that the learned routers’ activations correlate with the context-dependent senses of polysemous words. The authors also train a sub-billion-parameter model that achieves competitive CORE-22 performance against similarly sized base models, including Qwen3-0.6B and Llama 3.2-1B. Code and pretrained models are publicly available!