One of the underrated pieces of our post-training work with @baseten is exploring KV cache compaction. In our training runs over Legal Agent Bench, agents converged to similar policies, roughly: 1. Ingest client matter documents up front 2. Analyze + iterate on intermediate work product 3. Draft final work product 4. Revise For heavier matters (real diligence datarooms can easily be 10M+ tokens), step (1) becomes the bottleneck. Natural language compaction is the logical starting point for long context problems, and we’ve shown a few times that NL compaction improves performance, but at 10M+ tokens even NL summaries get unwieldy. NL compaction is also lossy – the model is forced to compress a rich internal state into prose. Over long-horizon trajectories, subtle degradations of compacted context can compound. In legal, if you lose a key clause, date, or defined term the impact on work product is catastrophic. KV cache compaction is an interesting alternative. STILL, a method from Baseten Research, compresses the KV cache directly. For each layer, learned latent queries cross-attend over the full cache and write a much smaller set of synthetic keys and values in a single forward pass. The compressed state stays in a representation the model can attend to directly, rather than being forced through a text bottleneck. This has two cool properties 1. Because compression lives in the model's latent space, superposition lets each slot store significantly more information than a discrete token can. That gives the agent much denser working context than a memo or summary. 2. The compactor can be trained in-domain, so it can learn what kinds of details legal tasks actually need to preserve. Having the right memory primitives is increasingly part of the post-training equation for legal work, and knowledge work more generally. Much more to come here and s/o @oneill_c, @mudithj, and team for the collaboration and innovative work here.