🚀 Stop using Transformers for weight-space programming
If you are not training with gradients on a GPU, why default to a Transformer?
When an LLM writes network weights directly, letting it also design the architecture achieves:
📉 ~1,800× fewer parameters at matched accuracy on ARC (AI2 Reasoning Challenge), compared to the best Transformer the same LLM wrote.
⚡ Up to 17× faster GPU inference with a memory architecture built for these tables, compared to a Transformer of the same size.
🧵 Paper & code at the end 👇