We think we’ve found a better path to continual post-training: merging models in input space.
🔹 A 256-token soft prompt (~0.01% of params) matches full-parameter RL
🔹 Plug-and-play composition: prompts trained separately just stack, with no joint training, and the combo beats the experts. No more MOPD!
🔹 Better OOD generalization, almost no forgetting
🔹 ~0 inference overhead, even with a different prompt per request. Built for personalization.
Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token.
Can we scale up this single stream computation?
Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows).
RNN 🤝 Transformers 🧵
“I want this country to realize that we stand on the edge of oblivion. I want every man, woman and child to understand how close we are to chaos. I want everyone to remember why they need us!" -V for Vendetta
Labeling video data with Sol-6.2 costs about 2x more than Sol-6.1 and Carbon right now. Combining Sol + Astra gives us about 90% score vs 89% for Sol 6.2 alone on synthetic benchmarks, while costing 5x more to do both. So Sol 6.2 is the best option right now in practice. Sol 6.1 scored 85% and merging it with Carbon scored 86%, for about the same cost as Sol-6.2, but also 2x slower and more complex.
GPT-6.2 is getting roasted on X
meanwhile, Astra and Sol just took #1# and #2# on my object detection benchmark
so I have some awkwardly positive results to share
↓ benchmarks and examples
If you ask an OpenAI model to label everything it sees in a video frame.. it will be extremely detailed. Compared to Carbon (lower right). Google tends to give just the obvious items. This is much more useful behavior for producing datasets for video models.
Anthropic investors seem to be funding a coordinated and obvious influencer campaign to discredit and downplay the latest OpenAI release. Tons of people on day 1 saying it sucks before I even have access to it in codex.
🚀 Stop using Transformers for weight-space programming
If you are not training with gradients on a GPU, why default to a Transformer?
When an LLM writes network weights directly, letting it also design the architecture achieves:
📉 ~1,800× fewer parameters at matched accuracy on ARC (AI2 Reasoning Challenge), compared to the best Transformer the same LLM wrote.
⚡ Up to 17× faster GPU inference with a memory architecture built for these tables, compared to a Transformer of the same size.
🧵 Paper & code at the end 👇
My theory is that the US is entering a war posture right now and “they” are intentionally breaking things one after the other to find where we are weak in the event of a real attack.
I'm adding value by periodically saying things to the swarm like "We're juggling a lot of agents - consider ways to improve the effectiveness of the swarm by adding new agents or passing some targetted suggestions" or "Add an agent working on checking for bugs"
Scaling matrix optimizers to wide networks is expensive. Our new research post introduces Echelon, a rank-anchored formulation of Muon that orthogonalizes only the top subspace in O(mnr) time while rescaling the tail: up to 10.5× faster optimizer steps, with Muon's loss at 1.3B.
Blog:
We pair the algorithm with the first Lean-checked convergence proof for spectral steepest descent.
Lean proofs:
1/ From 7B to 30B parameters, zeroing fewer than 0.05% of frozen weights with GRPO matches full fine-tuning on math, and on Qwen2.5-7B it beats the base model even at pass@128 (80.08% vs 74.01%) without writing a single new weight. 🧵
Fruit-fly connectome optimized inference for DeepSeek-V4.1-Flash on 2× NVIDIA GB300 DGX Stations! It impressively hits 56k tok/s prefill and 201.7 tok/s c1 decode. Well done little dude.