가입 후 초대 링크를 공유하면 동영상 재생 및 초대 보상을 받을 수 있습니다.

Turing Post (Ksenia Se)
@TheTuringPost
On X – we follow important research papers and explain ideas behind it. In the newsletter, we connect the dots between AI’s past, present, and future ⬇️
가입 June 2020
6.1K 팔로잉 중    89.9K 팬
Looped Transformers are low-key fascinating. First of all, you need to understand what travels around the loop and the 3 core things you set for the architecture. A Looped Transformer runs the same Transformer block (or stack) for multiple passes, feeding each output back in as the next input. ▪️ What travels around this loop is the hidden state: - Token embeddings form the initial hidden representation. - The shared stack updates that representation via attention and feed-forward operations. - Its output goes back to the stack for another pass, using an updated hidden state. Some designs also reuse the input, add time-step embeddings, or run multiple residual streams. - After the chosen number of passes, the output head converts the selected representation into token scores. That's all to win additional computation without an additional set of block weights for every pass. Although the weights stay the same during inference, changing inputs let the same block produce different updates. During decoding it attends via the KV cache, during prefill it updates masked tokens in parallel. A model can loop its entire stack or repeat only a middle stack between ordinary input and output blocks. Note that repeating the stack doesn't require generating a new text token between passes. ▪️ So there are 3 basic things you need to identify for the specific Looped Transformer architecture: 1. The repeated blocks 2. The state they receive 3. The rule that controls their execution - a fixed number of passes, a user-selected count, or an adaptive rule If you want to nerd out on the rest, like adaptive depth, what happens to the KV-cache and hardware, 2026 breakthroughs, and when another pass is worth the cost - we put together a full guide over here:
더 보기