Beni(@beniduboss):Didn’t see anyone mention this so guess I will My team and I looked into H-Nets pretty seriously when we first started exploring LLM efficiency It’s genuinely super interesting, and what they’ve done is very impressive, but I’m still skeptical of the inference economics at scale Dynamic chunking means you deal with unpredictable compute per request, which hence means harder batching and more complicated GPU scheduling/memory management But there are ways to handle this by using separate queues and grouping requests by expected workloads + custom kernels, but i don’t think it’s sufficient to beat the competitions A hybrid architecture might be interesting though! Because you can reduce FLOPs on paper and still end up with worse throughput, higher latency and more expensive inference in prod At the end of the day what really matters is how many requests you can serve per GPU at the same quality and latency But again That really doesn’t take anything from what @_albertgu and his team have achieved, we gave up on H-nets due to the above and a few other reasons but regardless, it still has very significant use cases even if it struggles to scale Gg

2026.10.28 22:35

Didn’t see anyone mention this so guess I will My team and I looked into H-Nets pretty seriously when we first started exploring LLM efficiency It’s genuinely super interesting, and what they’ve done is very impressive, but I’m still skeptical of the inference economics at scale Dynamic chunking means you deal with unpredictable compute per request, which hence means harder batching and more complicated GPU scheduling/memory management But there are ways to handle this by using separate queues and grouping requests by expected workloads + custom kernels, but i don’t think it’s sufficient to beat the competitions A hybrid architecture might be interesting though! Because you can reduce FLOPs on paper and still end up with worse throughput, higher latency and more expensive inference in prod At the end of the day what really matters is how many requests you can serve per GPU at the same quality and latency But again That really doesn’t take anything from what @_albertgu and his team have achieved, we gave up on H-nets due to the above and a few other reasons but regardless, it still has very significant use cases even if it struggles to scale Gg

Albert Gu@_albertgu

2026.10.28 17:53

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.