Mayank Mishra(@MayankMish98):We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵

Mayank Mishra

@MayankMish98

Doing philosophy with @tri_dao, @profjoeyg, @istoica05

가입 June 2017

696 팔로잉 중 2K 팬

Mayank Mishra@MayankMish98

2026.09.22 21:05

We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵