Institute of Foundation Models(@IFM_AI):Hector Liu, IFM's Head of Technology and US Lab Director, walked MBZUAI through how K2 Horizon was built. A rare, candid close-up of LLM training. From the room: - Two new attention designs: softplus gated attention is a more expressive form of gated attention; Mixture-of-Value Attention adds capacity at no extra KV cache cost. - Nearly half the pre-training data was synthetic. An interesting generation method: Socratic dialogues, a teacher model questioning a student model toward the answer. - A scaling law guided learning-rate choices across model sizes. Four models trained on the same data learned at the same pace, with normalized loss curves lining up almost perfectly. More in the upcoming paper.

2026.10.09 05:02

Hector Liu, IFM's Head of Technology and US Lab Director, walked MBZUAI through how K2 Horizon was built. A rare, candid close-up of LLM training. From the room: - Two new attention designs: softplus gated attention is a more expressive form of gated attention; Mixture-of-Value Attention adds capacity at no extra KV cache cost. - Nearly half the pre-training data was synthetic. An interesting generation method: Socratic dialogues, a teacher model questioning a student model toward the answer. - A scaling law guided learning-rate choices across model sizes. Four models trained on the same data learned at the same pace, with normalized loss curves lining up almost perfectly. More in the upcoming paper.