Alex Krentsel
@AlexKrentsel
Autonomous Systems PhD @Berkeley_EECS, Sr Researcher @GoogleResearch.
systems guy working on harness-layer self-improvement
参加 March 2022
251 フォロー中 4.2K ファン
super cool work from Mayank. excellent blog post on how to reach very impressive performance with orders of magnitude less compute
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs.
No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase.
Meet Rigel 🧵
もっと見る
