Yuandong Tian(@tydsh):We introduce TAPA (https://t.co/YpO34Q4Zyp) that decouples magnitude from angular contributions in position encoding. TAPA yields better OOD (long-context) performance than vanilla RoPE approach. We also provide theoretical analysis why it works. Thanks @yusidwang and the colleagues for the great work!

Yuandong Tian

@tydsh

Co-founder of @Recursive_SI. ex-Meta FAIR Director. ex-Google. Reasoning, Optimization and Understanding LLM. Novelist in spare time. PhD in @CMU_Robotics.

加入 December 2009

950 正在關注 46.4K 粉絲

Yuandong Tian@tydsh

2026.06.26 16:41

We introduce TAPA ( that decouples magnitude from angular contributions in position encoding. TAPA yields better OOD (long-context) performance than vanilla RoPE approach. We also provide theoretical analysis why it works. Thanks @yusidwang and the colleagues for the great work!