NVIDIA released a new RL framework:
Your entire setup is one Python file:
- write a class with a step() function
- return a reward from it
- the framework handles the rest, up to 1T-class MoE training
In a typical RL framework, the environment and reward logic are added inside the trainer's codebase.
Adding a new task requires writing framework code, registering it, and wiring up config for it.
The reward usually comes from a separately trained reward model, which needs its own GPUs and placement next to the actor and critic.
NVIDIA's Molt framework drops both these requirements.
Every run takes an agent_path flag pointing to one Python module, and the trainer code never changes.
The module gives you two ways to plug in an agent.
> The first is an Env class where you write a step() function that receives the model's output and returns a reward, and the framework handles everything else, i.e., calling the model, tokenizing, and looping turn after turn until the episode ends.
> The second is a ChatAgent class, for agents that already exist as their own loop. Here, you write the loop yourself, calling the model through a standard OpenAI or Anthropic client, and the framework spins up a local server so those calls hit its vLLM engines instead of an external API.
Either way, the trainer sees the same thing, i.e., a trajectory of tokens and a reward at the end.
That reward can come from a string-match grader, code executed in a sandboxed subprocess, or an LLM-as-judge call.
So there is no reward model to train or place, and no environment code living inside the framework.
Under the hood, Molt runs a single trainable actor (plus an optional critic for PPO) on FSDP2, vLLM engines for rollout, and Ray queues between them, so generation, training, and weight sync overlap instead of running in lockstep.
Scaling just requires setting some flags on the same command.
The framework also splits the model across GPUs for you, so the script that trains an 8B model on one node can also train DeepSeek-V3-scale MoE models on a cluster by raising the parallelism flags.
Here's the repo:
Since the framework is new, I am still experimenting with it and will share more about my learnings soon.
To dive deeper into how top AI labs are building RL Agents in 2026, I wrote a full article about it.
It covers the evolution from RLHF to GRPO with verifiable rewards and using an LLM-as-judge reward function. This idea extends RL training to non-verifiable agentic tasks without writing custom reward functions.
Read it below.
Show more
Chat with video using EmbeddingGemma 2 (100% local):
EmbeddingGemma 2 by
@GoogleDeepMind maps text, images, video, and audio into a shared vector space. This makes it possible to search video frames with natural language.
Devs have already built native Rust implementations for local inference.
Unsloth also provides a 176 MB, 4-bit GGUF checkpoint for text embeddings, although that version doesn't include the full multimodal pipeline.
I tested the video-search use case by building a chat interface that finds matching frames and jumps to their timestamps.
On my Apple M3 Pro, using the CPU:
- Indexing took 29s for a 60s video
- Search took 25 ms on average across 9 queries
- The saved frame vectors occupied 180 KiB.
The video is indexed once. Each new search embeds the query and compares it with the saved frame vectors. Processing stays entirely on the laptop.
The video below depicts the entire flow in action.
That said, model quantization reduces the memory needed for the weights. The retrieval index has its own storage cost, which grows as you add videos.
To dive deeper into reducing that cost, I wrote a full article on binary quantization.
It explains how storing vectors with one bit per dimension reduces their raw storage by 32x compared with float32, and includes a working retrieval implementation.
Read it below.
Show more
8 AI model architectures, visually explained:
LLMs get so much attention that it's easy to overlook the rest of the field. Different inputs, tasks, and deployment constraints have led to very different model designs.
Here are eight specialized AI model architectures:
1. LLM (Large language models)
Text is split into tokens, mapped to embeddings, and processed through transformer layers. The model generates its response by predicting successive tokens.
↳ GPT, Claude, Gemini, Llama.
2. LCM (Large concept models)
Moves prediction to the sentence level. Sentences become SONAR embeddings, and the model predicts the next sentence representation. Meta explored both autoregressive and diffusion-based versions.
↳ Meta's LCM research introduced this approach.
3. LAM (Large action models)
Connects a user's request to actions in an environment. These systems interpret intent, plan steps, and use interfaces or tools, often checking results before continuing.
↳ Rabbit R1, Microsoft UFO, Claude Computer Use.
4. MoE (Mixture of experts)
A router assigns token representations to selected expert networks inside the model. Activating a subset of experts adds model capacity without running every expert for every token.
↳ Mixtral, DeepSeek, Switch Transformer.
5. VLM (Vision-language models)
Combines visual representations with language processing. A vision encoder extracts image features, which are connected to a language model so it can answer questions about visual content.
↳ GPT-4V, Gemini, LLaVA.
6. SLM (Small language models)
Uses fewer parameters to reduce memory and compute requirements. These models can run locally, with quantization often reducing their deployment footprint further.
↳ Phi-3, Gemma, Mistral 7B, Llama 3.2 1B.
7. MLM (Masked language models)
Learns by predicting hidden tokens using context on both sides. This bidirectional training produces representations useful for retrieval, classification, and other language-understanding tasks.
↳ BERT, RoBERTa, DeBERTa.
8. SAM (Segment Anything Model)
An image encoder processes the image, while a prompt encoder handles points, boxes, or masks. A mask decoder combines them to identify the requested regions.
↳ Meta's SAM supports interactive segmentation and image annotation.
These aren't eight mutually exclusive categories. A small model can process images, and a language model can use MoE layers. Some labels describe architecture, while others describe training or intended use.
There's another approach I kept outside the image because it alters how a model handles context rather than introducing a separate model family.
Recursive language models, from MIT, put the input into a Python environment that a root model can inspect through code.
The model can search or slice that input, then make recursive model calls on selected portions instead of loading everything into its own context window.
The reported experiments handled inputs beyond 10 million tokens without retraining, with gains over the base model even on some tasks whose inputs already fit its context window.
I recently wrote a full breakdown of how this works, including why accessing a large input and reasoning reliably over it are different problems.
Read it below.
Show more
8 AI model architectures, visually explained:
LLMs get so much attention that it's easy to overlook the rest of the field. Different inputs, tasks, and deployment constraints have led to very different model designs.
Here are eight specialized AI model architectures:
1. LLM (Large language models)
Text is split into tokens, mapped to embeddings, and processed through transformer layers. The model generates its response by predicting successive tokens.
↳ GPT, Claude, Gemini, Llama.
2. LCM (Large concept models)
Moves prediction to the sentence level. Sentences become SONAR embeddings, and the model predicts the next sentence representation. Meta explored both autoregressive and diffusion-based versions.
↳ Meta's LCM research introduced this approach.
3. LAM (Large action models)
Connects a user's request to actions in an environment. These systems interpret intent, plan steps, and use interfaces or tools, often checking results before continuing.
↳ Rabbit R1, Microsoft UFO, Claude Computer Use.
4. MoE (Mixture of experts)
A router assigns token representations to selected expert networks inside the model. Activating a subset of experts adds model capacity without running every expert for every token.
↳ Mixtral, DeepSeek, Switch Transformer.
5. VLM (Vision-language models)
Combines visual representations with language processing. A vision encoder extracts image features, which are connected to a language model so it can answer questions about visual content.
↳ GPT-4V, Gemini, LLaVA.
6. SLM (Small language models)
Uses fewer parameters to reduce memory and compute requirements. These models can run locally, with quantization often reducing their deployment footprint further.
↳ Phi-3, Gemma, Mistral 7B, Llama 3.2 1B.
7. MLM (Masked language models)
Learns by predicting hidden tokens using context on both sides. This bidirectional training produces representations useful for retrieval, classification, and other language-understanding tasks.
↳ BERT, RoBERTa, DeBERTa.
8. SAM (Segment Anything Model)
An image encoder processes the image, while a prompt encoder handles points, boxes, or masks. A mask decoder combines them to identify the requested regions.
↳ Meta's SAM supports interactive segmentation and image annotation.
These aren't eight mutually exclusive categories. A small model can process images, and a language model can use MoE layers. Some labels describe architecture, while others describe training or intended use.
There's another approach I kept outside the image because it alters how a model handles context rather than introducing a separate model family.
Recursive language models, from MIT, put the input into a Python environment that a root model can inspect through code.
The model can search or slice that input, then make recursive model calls on selected portions instead of loading everything into its own context window.
The reported experiments handled inputs beyond 10 million tokens without retraining, with gains over the base model even on some tasks whose inputs already fit its context window.
I recently wrote a full breakdown of how this works, including why accessing a large input and reasoning reliably over it are different problems.
Read it below.
Show more
7 MoE routing strategies in production, clearly explained:
(bookmark this)
Sending every token to its highest-scoring experts can make an MoE model slower.
The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused.
With fixed capacity, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings.
MoE routing has to select useful experts while distributing work efficiently.
The seven strategies in this visual handle that tradeoff differently.
> Token-choice routing
Each token selects its top-k experts using router scores. The expert count per token stays fixed, but expert loads can vary. Load-balancing mechanisms help prevent traffic from concentrating on a few experts.
> Expert-choice routing
Each expert selects its highest-scoring tokens within a fixed capacity. Expert workloads stay balanced, while individual tokens can receive different amounts of expert computation.
> Dynamic routing
The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive, although router confidence doesn't directly measure token difficulty.
> Balanced assignment
A global matching procedure assigns tokens under expert load constraints. Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment adds overhead.
> Hash-based routing
A deterministic hash maps tokens to experts without a learned gate. This removes router learning, but expert selection has less flexibility to adapt to context.
> Soft MoE routing
Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens, changing what an expert processes.
> Group-limited routing
The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes.
These choices affect how serving engines batch expert computation and distribute work across GPUs.
Evenly distributing tokens doesn't guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time.
To dive deeper into MoE engineering, I covered everything in an article recently, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations.
It follows what happens after the router selects experts and explains the tradeoffs that determine latency and throughput.
Read it below.
Show more
7 MoE routing strategies in production, clearly explained:
(bookmark this)
Sending every token to its highest-scoring experts can make an MoE model slower.
The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused.
With fixed capacity, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings.
MoE routing has to select useful experts while distributing work efficiently.
The seven strategies in this visual handle that tradeoff differently.
> Token-choice routing
Each token selects its top-k experts using router scores. The expert count per token stays fixed, but expert loads can vary. Load-balancing mechanisms help prevent traffic from concentrating on a few experts.
> Expert-choice routing
Each expert selects its highest-scoring tokens within a fixed capacity. Expert workloads stay balanced, while individual tokens can receive different amounts of expert computation.
> Dynamic routing
The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive, although router confidence doesn't directly measure token difficulty.
> Balanced assignment
A global matching procedure assigns tokens under expert load constraints. Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment adds overhead.
> Hash-based routing
A deterministic hash maps tokens to experts without a learned gate. This removes router learning, but expert selection has less flexibility to adapt to context.
> Soft MoE routing
Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens, changing what an expert processes.
> Group-limited routing
The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes.
These choices affect how serving engines batch expert computation and distribute work across GPUs.
Evenly distributing tokens doesn't guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time.
To dive deeper into MoE engineering, I covered everything in an article recently, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations.
It follows what happens after the router selects experts and explains the tradeoffs that determine latency and throughput.
Read it below.
Show more
Another insane Grok
@bot feature!
Several apps exist because the workflows they support cannot be represented in a standard document.
They need tables, charts, and controls that respond as someone interacts with the data.
Pages in Grok Bot lets the bot build these directly into a document.
You describe what you need in chat. The bot creates the Page and saves it to its Library, where you can reopen it, share it, and ask for changes.
The same document can contain the explanation, the data, and the interface for exploring it.
For simpler trackers and dashboards, this starts to cover what would otherwise require a separate app.
I tested this myself by building a Notion-like Content OS with a pipeline board, KPI cards, and charts that update when I drag a card to Posted.
The recording shows the complete setup in action.
If you want to get started with Grok Bot, I wrote a full masterclass covering the shared computer, memory, skills, routines, and macOS setup.
Read it below.
Show more
Harness Engineering 2.0 is here!
We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too.
And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side.
You provide one task and then run it across several model x harness configs that you want to test.
Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns:
- total cost
- execution time
- whether the task passed
- files and artifacts produced
If you want to try it yourself, it is implemented within the open-source HarnessRouter project.
Here's the repo:
(don't forget to star it ⭐ )
HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:
- Codex
- Hermes
- Claude code
- DeepSeek Harness
- System One, powered by Jev
- And 25 more agent harnesses
In Harness Arena, the cheapest run does not automatically win.
A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass.
Here's the result from one Harness Arena test:
- Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits
- Claude Code + Opus 4.8: 3/5 successful runs using 223 credits
That is around 474x difference in credits, while the more expensive configuration also failed two runs.
You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task.
This helps answer practical questions such as:
- Should this task run on Codex or Claude?
- Can a cheaper config still pass every check?
- Which setup is best to handle this workload?
Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures.
So you are no longer choosing an agent stack based only on a model leaderboard.
Instead, you can benchmark the complete runtime on the task your product actually needs to solve.
To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves.
It also covers the local setup, a working request, and how sessions and files move through the system.
Read it below.
Show more
Harness Engineering 2.0 is here!
We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too.
And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side.
You provide one task and then run it across several model x harness configs that you want to test.
Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns:
- total cost
- execution time
- whether the task passed
- files and artifacts produced
If you want to try it yourself, it is implemented within the open-source HarnessRouter project.
Here's the repo:
(don't forget to star it ⭐ )
HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:
- Codex
- Hermes
- Claude code
- DeepSeek Harness
- System One, powered by Jev
- And 25 more agent harnesses
In Harness Arena, the cheapest run does not automatically win.
A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass.
Here's the result from one Harness Arena test:
- Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits
- Claude Code + Opus 4.8: 3/5 successful runs using 223 credits
That is around 474x difference in credits, while the more expensive configuration also failed two runs.
You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task.
This helps answer practical questions such as:
- Should this task run on Codex or Claude?
- Can a cheaper config still pass every check?
- Which setup is best to handle this workload?
Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures.
So you are no longer choosing an agent stack based only on a model leaderboard.
Instead, you can benchmark the complete runtime on the task your product actually needs to solve.
To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves.
It also covers the local setup, a working request, and how sessions and files move through the system.
Read it below.
Show more
A new technique just made attention 23x faster!
At 10M tokens, sparse attention slows down not because of attention itself but rather because of deciding which keys to attend to.
Let's understand why and how a new technique solves it without scanning every key.
1) Full attention at long context:
In full attention, every query is compared against every earlier key. This leads to O(T^2) complexity for prefill, and at 10M tokens it dominates the forward pass.
2) Sparse attention:
Sparse attention lets each query read only a small set of keys instead of all of them, so the attention becomes cheap.
But something still has to pick those keys, and at long context, this step becomes the bottleneck.
3) Why picking keys is still quadratic:
DeepSeek's DSA uses a lightning indexer that scores every earlier key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of nearby keys.
Each of them scans the whole context for every query, so the indexer stays O(T^2), and at 10M tokens, it takes 98% of the layer's time.
Block methods have another issue. Blocks group keys by position, so one relevant key within filler can get averaged away in the block summary.
4) How LSA picks keys:
Latent Sparse Attention (LSA) is a new approach that groups keys by meaning instead of position.
- Each key is projected into a 64-dim index space, and every 4 consecutive keys are pooled into one.
- When a 512-key block fills up, it's clustered into buckets of ~8 similar keys, with one centroid per bucket. Finished blocks are never rebuilt.
- A query scores only the centroids, keeps the top 1,024 buckets, and expands them into 4,096 keys, plus 4 sink tokens and the 512 most recent keys.
- Exact FlashAttention then runs over that list with an unchanged kernel.
If you've built RAG, this is like cluster-based vector search running inside attention. The kernel design even cites FAISS.
At 10M tokens, LSA's prefill is 72.4x faster than full attention on Qwen-3.8-27B and 22.7x faster than GLM-5.3-Flash's own sparse attention.
The time to first token improves:
- 9.2x on Qwen
- 2.2x on GLM
Below ~512k tokens, it's roughly break-even.
Here's the paper:
That said, LSA attempts to find the keys more cheaply, not storing them. The KV cache still grows linearly with context, and at 10M tokens, managing that memory becomes the next engineering problem.
I wrote an article on how 12 ways serving systems handle it, covering paging, offloading, quantizing, and more.
Read it below.
Show more
A new technique just made attention 23x faster!
At 10M tokens, sparse attention slows down not because of attention itself but rather because of deciding which keys to attend to.
Let's understand why and how a new technique solves it without scanning every key.
1) Full attention at long context:
In full attention, every query is compared against every earlier key. This leads to O(T^2) complexity for prefill, and at 10M tokens it dominates the forward pass.
2) Sparse attention:
Sparse attention lets each query read only a small set of keys instead of all of them, so the attention becomes cheap.
But something still has to pick those keys, and at long context, this step becomes the bottleneck.
3) Why picking keys is still quadratic:
DeepSeek's DSA uses a lightning indexer that scores every earlier key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of nearby keys.
Each of them scans the whole context for every query, so the indexer stays O(T^2), and at 10M tokens, it takes 98% of the layer's time.
Block methods have another issue. Blocks group keys by position, so one relevant key within filler can get averaged away in the block summary.
4) How LSA picks keys:
Latent Sparse Attention (LSA) is a new approach that groups keys by meaning instead of position.
- Each key is projected into a 64-dim index space, and every 4 consecutive keys are pooled into one.
- When a 512-key block fills up, it's clustered into buckets of ~8 similar keys, with one centroid per bucket. Finished blocks are never rebuilt.
- A query scores only the centroids, keeps the top 1,024 buckets, and expands them into 4,096 keys, plus 4 sink tokens and the 512 most recent keys.
- Exact FlashAttention then runs over that list with an unchanged kernel.
If you've built RAG, this is like cluster-based vector search running inside attention. The kernel design even cites FAISS.
At 10M tokens, LSA's prefill is 72.4x faster than full attention on Qwen-3.8-27B and 22.7x faster than GLM-5.3-Flash's own sparse attention.
The time to first token improves:
- 9.2x on Qwen
- 2.2x on GLM
Below ~512k tokens, it's roughly break-even.
Here's the paper:
That said, LSA attempts to find the keys more cheaply, not storing them. The KV cache still grows linearly with context, and at 10M tokens, managing that memory becomes the next engineering problem.
I wrote an article on how 12 ways serving systems handle it, covering paging, offloading, quantizing, and more.
Read it below.
Show more
Another big breakthrough after Jev!
Jev answers a fixed question over a fixed input, so a live chat requires re-reading the full transcript on every update.
Devs just built Sage2, which pairs an extraction model and a decision model on the same live stream.
Usually, in a live conversation, the details and the decisions keep changing.
New details show up gradually, and several questions stay open the whole time.
- The extraction stage turns each new message into structured fields in 150 ms, each tied to the exact source span it came from. Those fields then become decision options.
- The decision model scores them as text streams in, and returns typed answers with probabilities, so the full loop from raw text to action runs in about 200 ms.
This gives a fully reactive automation over a live chat, with no re-reading of the transcript.
Sage2 is built by
@levantolabs, the team that shipped the first decision model, before Jev.
The video below depicts the model live in action on a broadband support chat, and I worked with the team to put this together.
More details below.
Show more
Software can now understand human language and act on it, in real time.
Introducing Sage2
- Extract: natural language in, structured data out. 166x cheaper than Opus 5.5, beats
@Cloudflare's Clairic
- Decide: the first decision model with streaming. 3x faster, because it reads the context as it grows.
Sage2 is a new primitive for developers. Get 20M tokens free 👇
Also available on
@near_ai and, for agents, via
@mpp with
@Stripe cards and
@Tempo stables.
Show more
Claude Code’s architecture, explained visually:
(bookmark this)
Claude Code is a lot more than a CLI that invokes the Claude models.
The actual system has six layers, and the model is just one node inside the loop.
The diagram below explains every component:
1) Input layer handles session management, permission gating, and YAML-based trust tiers before anything reaches the model.
2) Knowledge layer holds the skill registry, context compressor, task graph, and cross-session memory store. This is where harness intelligence lives outside the weights.
The context compressor is a 5-layer cascade that kicks in when the context window hits roughly 95% capacity. It doesn’t summarize your conversation the way ChatGPT does. Instead, it runs structured extraction on file paths, code snippets, and error histories while pruning redundant tool outputs. The goal is to keep the context usable, not just smaller.
3) Execution layer runs tool dispatch through a typed registry with one handler per tool, like bash, read, write, grep, glob, and revert.
The streaming runtime handles parallel execution, and the prompt cache reuses stable prefixes at roughly 10% of the original cost.
4) Integration layer connects the MCP runtime to external servers (filesystem, git, custom). Tools register inward, and memory writes outward to a markdown file (agent_memory.md) that persists across sessions.
5) Multi-agent layer is the most underappreciated piece, and it works very differently from what most people assume.
Claude Code supports two levels of parallelism:
- Subagents are lightweight workers that run inside your session. They get their own context window, do a focused task (search the codebase, explore a file tree), and return results to the parent. They can’t talk to each other, and they can’t spawn their own subagents. It’s a strict parent-child hierarchy.
- Agent teams go further. One session acts as a team lead, and it spawns independent teammates, each running as a full Claude Code instance with its own context window. The team lead breaks a task into subtasks, assigns them, and monitors progress.
The coordination happens through two mechanisms → a shared task list (JSON files on disk) and a mailbox system for peer-to-peer messaging.
Each teammate gets git worktree isolation. It’s a separate working directory with its own branch, sharing the same repository history.
This means agents can write to overlapping parts of the codebase without file conflicts. When they finish, worktrees with no changes are cleaned up automatically. Worktrees with changes persist for human review before merging.
6) Observability layer wraps everything. An event bus with lifecycle hooks logs all tool calls and messages, creating a complete audit trail of the agent’s actions and decisions.
Background executors run daemon threads non-blocking, so observability never stalls the main loop.
Finally, the master agent loop sits at the center of all six layers. It assembles context, calls the model, receives a tool request, executes it, feeds the result back in, and repeats. Every iteration is one turn.
Within a turn, the model might request a tool call. That request flows through the permission system, gets executed, and the output feeds back into the loop as the next input.
The loop itself is single-threaded on purpose. All the intelligence lives in the layers around it, not in the loop logic. Anthropic calls it a dumb loop because the model reasons, and the harness mediates.
This is the architecture behind Claude Code.
If you want to dive deeper into Claude Sugagents and agent teams specifically, I recently wrote an article about them.
It explains the difference between Claude sub-agents (isolated, fire-and-forget workers) and agent teams (persistent, peer-communicating instances with a shared task list), and when to use each.
Read it below.
Show more
A tricky LLM interview question:
You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory.
You add KV cache compression and evict 90% of the cached tokens.
VRAM usage doesn't move, and the GPU still runs out of memory.
Why?
(answer below)
Evicting 90% of the KV cache can free close to zero bytes on a paged server.
During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes.
For reasoning models with long chains of thought, this cache is the main memory cost.
A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens.
Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss.
That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory.
vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each.
The allocator reclaims a block only when all 16 slots are empty.
Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block.
Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token.
So the allocator reclaims close to nothing.
You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41.
The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it.
An in-order layout doesn't need that metadata.
So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server.
The second problem is the eviction signal.
Most eviction methods rank tokens by their attention scores.
FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from.
To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides.
NVIDIA's TriAttention handles both problems.
It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters.
Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order.
On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory.
For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted.
NVIDIA's write-up:
I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching.
Read it below.
Show more
A tricky LLM interview question:
You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory.
You add KV cache compression and evict 90% of the cached tokens.
VRAM usage doesn't move, and the GPU still runs out of memory.
Why?
(answer below)
Evicting 90% of the KV cache can free close to zero bytes on a paged server.
During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes.
For reasoning models with long chains of thought, this cache is the main memory cost.
A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens.
Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss.
That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory.
vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each.
The allocator reclaims a block only when all 16 slots are empty.
Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block.
Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token.
So the allocator reclaims close to nothing.
You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41.
The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it.
An in-order layout doesn't need that metadata.
So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server.
The second problem is the eviction signal.
Most eviction methods rank tokens by their attention scores.
FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from.
To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides.
NVIDIA's TriAttention handles both problems.
It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters.
Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order.
On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory.
For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted.
NVIDIA's write-up:
I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching.
Read it below.
Show more
This is the strangest property of LLM inference:
Your LLM's output depends on requests that aren't yours.
The same prompt can produce different logits depending on how many other requests share its batch on the GPU.
Here's what happens inside the GPU.
Most of an LLM's forward pass is spent accumulating sums.
Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into a single output, so the order of those additions directly affects the result.
The catch is that floating-point addition isn't associative.
Every addition rounds the result to the nearest representable value, so (a + b) + c and a + (b + c) can produce different low-order bits.
fp16 has an 11-bit significand, so between 1024 and 2048 the spacing between representable values is 1.0.
If you compute 1024 + 0.4 + 0.4 left to right, each addition rounds back to 1024. If the two small values are added first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds to 1025.
The order of accumulation depends on batch size, because kernels choose their parallelization strategy to maximize GPU utilization.
A GPU like the H100 has 132 compute units, and a small batch doesn't give each of them enough work.
So the kernel splits each long sum into chunks, adds the chunks on different units at the same time, and then adds those partial results together.
So a different batch size gives a different partitioning, a different accumulation order, and slightly different logits.
Usually the difference is negligible, except when two tokens are nearly tied.
In a batch of 8:
- "Queens" might score 12.3417
- "New" might score 12.3415
In a batch of 32, the rounding might shift them to 12.3414 and 12.3416.
If temperature is 0, argmax will pick "Queens" in the first case and "New" in the second. Every later token is conditioned on a different prefix, so the whole completion diverges.
So the nondeterminism isn't coming from sampling, but rather from the fact that your request's numerics depend on concurrent traffic you don't control.
This is pretty evident at Temperature 0, but the nondeterminism comes from the forward pass, not the sampler.
You can fix this by making every reduction batch invariant, meaning each row is reduced in the same fixed order regardless of how many rows share the batch.
This costs throughput, because the kernel can no longer select the optimal split for each batch size. It uses one fixed strategy even when that leaves compute units idle.
The major inference engines now expose this as a flag:
→ vLLM has VLLM_BATCH_INVARIANT=1, still in beta.
→ SGLang has --enable-deterministic-inference, at a 25-45% slowdown.
→ DeepSeek's DeepInfer has DEEPINFER_BATCH_INVARIANT=1.
If you run evals, RL rollouts, or anything that needs reproducible outputs, enable it and treat batch composition as part of your experimental setup.
And in production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step.
I wrote an article that explains how it works and why every major inference engine uses it.
Read it below.
Show more
RAG vs. Vectorless RAG, clearly explained!
In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.
The problem is that similarity and relevance answer different questions.
For instance, five annual reports from one company can likely follow the same template every year, and only the numbers can change.
So a FY2022 question can lie just as close to an FY2025 chunk, and the LLM might answer from the wrong year.
Cross-references introduce another problem. If a chunk mentions "see Appendix G" but looks nothing like the table in Appendix G, that table will never make it to the top-k.
Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning instead.
The preprocessing step parses each document's layout into a ToC tree index. Every node holds a section title, its page range, and a short summary, and the whole tree fits in the LLM's context.
At query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.
For example, for a question about debt, it might pick the "Financial Summary" section and also "Appendix G", because its summary says the debt details are there.
The system then pulls those full sections and checks whether the evidence is enough.
↳ If it is, the LLM answers from the full sections, with page citations.
↳ If not, it goes back to the tree and opens more nodes.
The LLM also sees the chat history. So if you ask about assets and then follow up with "what about liabilities?", it goes back to the same report.
The trade-off is that every query now runs LLM calls over the tree.
Top-k vector search will be cheaper in that same run (although not guaranteed to be as accurate), and the tree only helps when documents have real structure.
So the two suit different data:
↳ Use a tree for long, structured documents like filings, contracts, and manuals.
↳ Stick with a vector DB for lots of short, unrelated text.
The diagram below depicts both flows.
If you want to dive deeper, I have also written a full breakdown of MIT's Recursive Language Models.
It covers why LLMs get worse as context grows and how RLMs let the model split a long document and work through it piece by piece.
Read it below.
Show more
RAG vs. Vectorless RAG, clearly explained!
In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.
The problem is that similarity and relevance answer different questions.
For instance, five annual reports from one company can likely follow the same template every year, and only the numbers can change.
So a FY2022 question can lie just as close to an FY2025 chunk, and the LLM might answer from the wrong year.
Cross-references introduce another problem. If a chunk mentions "see Appendix G" but looks nothing like the table in Appendix G, that table will never make it to the top-k.
Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning instead.
The preprocessing step parses each document's layout into a ToC tree index. Every node holds a section title, its page range, and a short summary, and the whole tree fits in the LLM's context.
At query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.
For example, for a question about debt, it might pick the "Financial Summary" section and also "Appendix G", because its summary says the debt details are there.
The system then pulls those full sections and checks whether the evidence is enough.
↳ If it is, the LLM answers from the full sections, with page citations.
↳ If not, it goes back to the tree and opens more nodes.
The LLM also sees the chat history. So if you ask about assets and then follow up with "what about liabilities?", it goes back to the same report.
The trade-off is that every query now runs LLM calls over the tree.
Top-k vector search will be cheaper in that same run (although not guaranteed to be as accurate), and the tree only helps when documents have real structure.
So the two suit different data:
↳ Use a tree for long, structured documents like filings, contracts, and manuals.
↳ Stick with a vector DB for lots of short, unrelated text.
The diagram below depicts both flows.
If you want to dive deeper, I have also written a full breakdown of MIT's Recursive Language Models.
It covers why LLMs get worse as context grows and how RLMs let the model split a long document and work through it piece by piece.
Read it below.
Show more
Prefix-aware LLM routing, clearly explained!
(trick to get first LLM token 170x sooner)
Production prompts start with the same system prompt, tool definitions and earlier turns, and only the last part of each request is new.
So the GPU that served the previous turn still holds the KV cache for that shared prefix, but a cache-blind router may send the next request to a different GPU, which then has to recompute the whole prefix before it can write a single token.
A prefix-aware router avoids this by tracking which GPU holds which part of each prompt.
The visual below shows how it works:
↳ When a request arrives with prefix blocks A to D and a new suffix E, the router hashes the prompt block by block, so each hash covers everything before it.
↳ It looks those hashes up in a block-to-worker index, which stores locations and never the KV tensors themselves.
↳ Worker A holds 2 matching blocks, Worker C holds 3 and Worker B holds all 4, so Worker B gets the request.
↳ Worker B reuses the cached KV for A to D, computes KV only for E, and starts decoding.
↳ Once E is cached, a cache event updates the router's index, so the next turn lands on Worker B too.
If no worker has a match, the request gets a full prefill on whichever worker the router picks.
In a popular benchmark on 8 vLLM pods across 16 H100s, prefix-aware routing brought p90 time to first token down to 0.54s, while random routing took 92.5s on the same pods.
Real routers also weigh load alongside cache matches, so the GPU with the warmest cache doesn't end up with the longest queue.
All of this depends on the prefix cache itself working, and a small change early in the prompt can make every block after it miss.
I covered how prefix caching works, and how prompt changes cause cache miss, in a detailed article.
Read it below.
Show more
Prefix-aware LLM routing, clearly explained!
(trick to get first LLM token 170x sooner)
Production prompts start with the same system prompt, tool definitions and earlier turns, and only the last part of each request is new.
So the GPU that served the previous turn still holds the KV cache for that shared prefix, but a cache-blind router may send the next request to a different GPU, which then has to recompute the whole prefix before it can write a single token.
A prefix-aware router avoids this by tracking which GPU holds which part of each prompt.
The visual below shows how it works:
↳ When a request arrives with prefix blocks A to D and a new suffix E, the router hashes the prompt block by block, so each hash covers everything before it.
↳ It looks those hashes up in a block-to-worker index, which stores locations and never the KV tensors themselves.
↳ Worker A holds 2 matching blocks, Worker C holds 3 and Worker B holds all 4, so Worker B gets the request.
↳ Worker B reuses the cached KV for A to D, computes KV only for E, and starts decoding.
↳ Once E is cached, a cache event updates the router's index, so the next turn lands on Worker B too.
If no worker has a match, the request gets a full prefill on whichever worker the router picks.
In a popular benchmark on 8 vLLM pods across 16 H100s, prefix-aware routing brought p90 time to first token down to 0.54s, while random routing took 92.5s on the same pods.
Real routers also weigh load alongside cache matches, so the GPU with the warmest cache doesn't end up with the longest queue.
All of this depends on the prefix cache itself working, and a small change early in the prompt can make every block after it miss.
I covered how prefix caching works, and how prompt changes cause cache miss, in a detailed article.
Read it below.
Show more