Scaling past a trillion parameters requires inference engineers to solve 3 issues:
- Width: Routing between hundreds of sparse experts
- Depth: Mixing global and local linear attention
- Length: Processing 1M-token sequences
I asked Ali and Bryce to draw it up. They delivered.