You can Run locally bigger MoE model on a 12 GB phone, CPU only.
Treating flash storage as part of the memory hierarchy.
Only the experts actually needed for each token are loaded.
- Byte-identical to full residency
- No merge step for multi-shard GGUFs.
-
Show more
What if you could expand the number of expert combinations in an MoE by about 18x without adding parameters?
Title: MoRE: Mixture of Reused Experts
URL:
❓ What is MoRE?
An MoE where adjacent layers share one expert pool. With reuse factor R=2, two layers share a pool of 2N experts, and the physical parameter count matches a standard MoE.
💡 Why does it help?
With N=32 and K=4, the number of possible combinations rises from 35,960 to 635,376, about 18x. The same expert can even be used at consecutive layers.
❓ Is it hard to implement?
It takes weight sharing plus one depth embedding per layer. Routers stay layer-specific, and no custom normalization or stabilization is needed, according to the paper.
❓ Does it work?
At 1.15B, validation perplexity is 11.75 vs. 11.88 for DeepSeek, and the six-benchmark average rises from 47.05 to 48.12. With SonicMoE, throughput stays at about 96%.
💡 Caveats?
Tested only up to 1.15B and about 6.55B tokens. Behavior at 100B+ is unknown, and the reuse rate is only about twice random chance.
#
MoE# #
LLM#
Show more
What's the best open weight Mixture-of-Experts LLM for coding that fits in less than 60GB of RAM?
I think MoE might be necessary to get reasonably interactive speeds on the hardware I have access to - I want something faster than 12 tokens/second
Show more
Don’t miss tomorrow’s OCP Educational Webinar: Breaking Through the AI Network Wall with High-radix Switching
Date: July 28, 2026
Time: 12 - 1pm ET
Location: Virtual (ON24)
The “network wall” has become one of the primary bottlenecks in large-scale AI training and inference. While compute FLOPS have doubled every 18 months, merchant Ethernet switch ASIC capacity and radix (the number of compute devices that can be connected) has historically doubled in steps longer than two years. State-of-the-art switches support ~100 Tbps today and are only expected to reach ~200 Tbps by 2028. Switch radix in turn limits the single-hop domain size, which either limits the scale or extends the training time of sparse Mixture-of-Experts (MoE) models. A new switch technology and network architecture is required to deliver an order-of-magnitude increase in radix, breaking the network wall, and enabling AI to scale more effectively.
Learn more and register for free here:
Show more
Interesting Notes on Inkling 2 Small release:
Note: every single word is written by me. I used AI to fact check several things I doubted about, but that's it.
(1) Architecture
(A) 48-layer decoder-only transformer. 594B total.
(B) 17B active. Sparse MoE feed-forward backbone. Each token routed to 8 of 384 experts.
(C) Attention is a hybrid of local and global layers (no window size or ratio given).
(D) Context window up to 1M tokens.
(E) ViT is back. The rationale seems to be training efficiency. But it's just my opinion.
(2) Training budget
This is where things get interesting. In short, they used 8.4 + 3.8 = 12.2T for pretraining and 3T for RL. The SFT phase is a small bootstrap on synthetic data from open models like Kimi K2.5.
(A) They have 4 main pretraining stages:
• H1: Trained for 8.4T tokens on things like natural language, code, and image data; 68 hours on GB300 NVL72 racks, natively in NVFP4.
• H2: Trained for 3.8T on higher-quality, newer data at 512K context.
• H3: Trained for 34B on the highest-quality agent trajectories. You can probably call this stage mid-training.
• RL was trained for 3T tokens using 1,075 steps, 17M+ rollouts.
But let's also look at other metrics.
(B) FLOPs:
• pretraining ≈ 6 × 17B × 12.55T ≈ 1.3×10²⁴
• RL > 4×10²³
• RL ≈ 30% of pretraining compute, ~24% of the total.
(C) Wall-clock:
• The RL run (9 days) ran ~3× longer than the main pretraining phase (68 h). GPU count for H1 was not mentioned, so don't compare GPU-hours.
• RL alone = 2,048 GPUs × 9 days ≈ 442K GPU-hours.
(3) RL recipe
Inkling 1 was built from two separate RL runs (reasoning, agentic) using MOPD. Inkling 2 Small just uses one joint RL run over all domains, so there is nothing to merge. The only distillation they do is OPSD for alignment. The model prompted with a conduct spec teaches the same model without it.
(A) Joint RL run details:
• Every optimizer step took 12 minutes consuming 2.8B tokens of rollouts. The median staleness was 17 updates (or roughly 3.4 hours of policy lag).
• The mean rollout has 170K tokens. A p99 rollout of 1.5M tokens inside a 256K window means context was compacted ~5+ times in a single episode.
• Per-family mean rollout length (token share ÷ rollout share × 170K): coding ≈ 290K, web/app ≈ 260K, office ≈ 250K, STEM ≈ 185K, search ≈ 120K, vision ≈ 37K, tool-use ≈ 23K, chat/safety ≈ 4K.
• Important to note is that coding episodes on average exceed the 256K window, so compaction was the default choice here.
• Also interesting to note is that RL used almost half the FLOPs of the 8.4T H1 pretraining phase at roughly 3× the wall-clock.
• Effort levels were randomly sampled during RL, and the model was conditioned on them.
• They also used a 256K-token context window with compaction for longer trajectories, and compaction training for long-horizon tasks. But we don't know how their compaction mechanism actually works.
• For the actual RL algorithm, they use score centering. Basically, they are subtracting a baseline from rewards (REINFORCE with a centered score), a GRPO-family estimator without PPO clipping. We don't know anything about their learning rate, group size, etc.
(4) Tiny interesting details
(A) We don't know anything about the optimizer in any of the stages. It would be interesting to know whether they used fancier variants like Muon.
(B) I could not verify most things about their architecture. These include GQA + SWA, window size, local:global ratio, shared experts, latent MoE, MLA, positional scheme, tokenizer.
(C) Sparsity increased from 23× (Inkling 1 Small 276B/12B) to 35× (Inkling 2 Small 594B/17B).
(D) The MoE config changed to Kimi K2's. Inkling 1 had 256 routed experts and 2 shared experts, with 6 routed experts active per token, following DeepSeek-V3. Inkling 2 Small's 8-of-384 is Kimi K2's routed config.
(E) Inkling 1 was encoder-free. Images are encoded as patches of 40x40 pixels using a four-layer hMLP. 20×20 × 2×2 pooling = 40×40 px per token.
But Inkling 2 uses a 292M ViT encoder with 20×20-pixel patches with 2×2 spatial pooling, interleaved with text tokens. So they are using the same token budget for vision encoding but a different embedder.
(F) Audio was dropped in Inkling 2 Small. It supports image input but not audio. Inkling 1 and Inkling-Small took native audio.
(G) Training data types are text, images, and video but video is not an input modality in this model.
(H) Inkling 1 was pretrained on 45 trillion tokens. Then Inkling 2 Small used 27% of the tokens. But the AA Index increased from 12 to 31 !!
(I) They reported an incident midway through the main RL run. The tldr is the model, failing a task, got around a network block via a third-party service it reached through its search tools. An async monitor caught it. They paused the run, removed all web-access environments, added a pre-execution tool-call monitor, and resumed with alignment-only phases.
Overall, this is an impressive model release. We should wait for something bigger and even more impressive to come though. Because they have called the 8.4T phase "one rung of our predictive scaling ladder".
But can we also expect the adoption of fancier architecture ideas like Linear Attention in the next model?
Show more
A 1.6T parameter model does not use all 1.6T for every token. And a model that uses 13B per token can still hold far more than 13B.
The card shows how that works. Here is why it matters.
Models usually get better as you add parameters. But using more parameters takes more work. In a dense model, every token uses all the weights. MoE uses only a few of them at a time. That lets the model grow without making each token cost more.
Look at Mixtral and DeepSeek-V4-Flash. Mixtral used 12.9B parameters per token in 2023. V4-Flash uses 13B in 2026. Almost the same number, but V4-Flash holds about six times as many weights. V4-Pro goes further and uses about three percent of its 1.6T.
So the model grew much larger while the part used for each token barely changed. But all those weights still have to sit somewhere, and how much room they take depends on their precision.
Weights are only part of the memory story. Attention design can shrink the KV cache instead. At one million tokens of context, DeepSeek reports that V4-Flash needs seven percent of the KV cache V3.2 needed.
Memory can grow in one place and shrink in another. More weights to hold. Less cache to keep.
This series keeps coming back to that trade. The KV cache, quantization, and now MoE. Each one is a choice about what to keep in memory and what to move.
Less math does not mean less memory.
Earlier in this series: KV cache, prefill and decode, quantization.
Show more
TPUv7 has a specialized hardware-accelerated unit called the SparseCore. TPU's specialized SparseCore handles data movement, gathering each expert’s tokens into contiguous groups, while the TensorCore is left to run the expert matrix multiplications. When using SparseCore for rearranging expert inputs into the MoE kernel, it results in 12% better throughput.
Combined with other optimizations & TPU's lower TCO, TPU can achieve up to 50% better perf per dollar than Blackwell Ultra, as seen on InferenceX.
Show more
Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.)
Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks!
So, here are the architecture pics along with some notes on what I found most interesting:
1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.)
Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive.
2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.)
3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day!
4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer.
5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model.
6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026.
I added all six to the LLM Architecture Gallery for some additional details:
Show more
Moe and Lenny are strolling home from shul one Saturday morning. Suddenly a cab speeds past, and their friend, Irving, is running frantically behind it, flailing his arms wildly.
"Well," said Lenny, "I never imagined our good friend Irving was a Sabbath violator! Look at him running for that taxi."
"Wait a minute," Moe replied. "Didn't you read that book I lent you, The Other Side of the Story, about the command to judge other people favorably? I'll bet we can think of hundreds of reasons for Irving's behavior."
"Yeah, like what?"
"Maybe he's sick and needs to go to the hospital."
"Come on! He was running 60 miles an hour after that cab - he's healthier than Arnold Schwartzenwer is."
"Well, maybe his wife's having a baby."
"She had one last week."
"Well, maybe he needs to visit her in the hospital."
"She's home."
"Well, maybe he's running to the hospital to get a doctor."
"He is a doctor."
"Well, maybe he need supplies from the hospital."
"The hospital is a three minute walk in the opposite direction."
"Well, maybe he forgot that it's Shabbos!"
"Of course he knows it's Shabbos. Didn't you see his tie. It was his paisley beige 100% silk Giovani tie from Italy. He never wears it during the week."
"Wow, you're a really observant Jew! I didn't even notice he was wearing a tie."
"How could you not notice? Didn't you see how it was caught on the back fender of the taxi?"
Show more
MoE layer in MLX is getting 1.5x faster, achieved with better tile scheduling in grouped mm