Mayank Mishra(@MayankMish98):We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵

Mayank Mishra

@MayankMish98

Doing philosophy with @tri_dao, @profjoeyg, @istoica05

参加 June 2017

696 フォロー中 2K ファン

Mayank Mishra@MayankMish98

2026.09.22 21:05

We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵