One Layer Deeper is officially live! The motivating idea is that some models just don’t want to learn. Most optimizer benchmarks ask how quickly you can train a given model, but baked into that question is the assumption that it can be trained (well) at all. We think this matters especially for adaptive computation. Ideally, a model should be able to spend (far) more compute on harder problems and use more computation at test time than it spent on any given training instance. There are many possible ways to do this, and we don’t want to assume the answer is one specific recurrent or looped architecture. Making this all work is really an architecture-optimizer co-design problem. Rather than fixing the optimizer, loss, and training setup, One Layer Deeper asks people to design all of these together. For the competition task, we use repeated modular squaring, x^(2^T) mod N. The task is inherently serial because each squaring needs the residue produced by the previous one. When N is a semiprime, the only known way to skip ahead requires knowing its factorization. Increasing T therefore adds another genuinely dependent step without making the input or output longer. We are very excited to work on this with @marksaroufim @_arohan_ and everyone on the @CoreAutoAI team. Submissions: Blog Post: