Yuandong Tian
@tydsh
Co-founder of @Recursive_SI. ex-Meta FAIR Director. ex-Google. Reasoning, Optimization and Understanding LLM. Novelist in spare time. PhD in @CMU_Robotics.
加入 December 2009
950 正在关注 46.4K 粉丝
We introduce TAPA ( that decouples magnitude from angular contributions in position encoding. TAPA yields better OOD (long-context) performance than vanilla RoPE approach. We also provide theoretical analysis why it works.
Thanks @yusidwang and the colleagues for the great work!
显示更多
We’d like to introduce our paper on long-context positional encoding, centered on a simple principle:
