Larry Dial(@classiclarryd):Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions.

Larry Dial

@classiclarryd

Technical Staff at Open Athena, working on Marin

参加 May 2024

48 フォロー中 2.2K ファン

Larry Dial@classiclarryd

2026.06.03 16:26

Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions.