Silicon Atlas(@Silicon_Atlas):Spec sheets sell FLOPS. Decode spends bytes. Single-user BF16 decode can sit at roughly 1 FLOP per byte. On an H100 SXM, that puts its theoretical ceiling near 0.3% of peak math throughput.* Batching moves the point right: more tokens share each weight read. FP8 has a twist. Halving weight bytes moves the point right, but native FP8 math doubles the compute roof too. The ridge moves with you. Decode gets faster, yet the batch needed to reach the roof stays about the same. Attention limits the gains. Each user brings a KV cache. Longer contexts mean more bytes to read, so sharing weights buys less. Smaller caches and speculative decoding can help. *Teaching estimate for weight-dominated, batch-1 decode, not measured GPU utilization.

2026.10.06 05:21

Spec sheets sell FLOPS. Decode spends bytes. Single-user BF16 decode can sit at roughly 1 FLOP per byte. On an H100 SXM, that puts its theoretical ceiling near 0.3% of peak math throughput.* Batching moves the point right: more tokens share each weight read. FP8 has a twist. Halving weight bytes moves the point right, but native FP8 math doubles the compute roof too. The ridge moves with you. Decode gets faster, yet the batch needed to reach the roof stays about the same. Attention limits the gains. Each user brings a KV cache. Longer contexts mean more bytes to read, so sharing weights buys less. Smaller caches and speculative decoding can help. *Teaching estimate for weight-dominated, batch-1 decode, not measured GPU utilization.