@Translation just released Lara Mini base, a small open-weights model trained on English and 44 other languages, covering all 24 official EU languages! ๐ช๐บ
"Lara Mini Base: Language Diversity as a Resource, Not a Trade-off"
This paper shows that in small models, many languages don't have to compete for capacity: multilingual training can help rather than dilute.
It covers a 7.2B-total, 0.6B-active MoE trained from scratch on 6T tokens, the highest average score among open base models with up to 2B active parameters (ahead of models trained on 4 to 6x as many tokens), 128k context, generation at least 1.8x faster on one GPU than any higher-scoring model, and much more!
Read this excellent paper from the Translated team: