Register and share your invite link to earn from video plays and referrals.

elie
@eliebakouch
research @PrimeIntellect (prev: @huggingface) anon feedback:
Joined January 2024
4.8K Following    26.7K Followers
reflection beam oss architecture details IT'S SLIDING WINDOW AGAIN, same window size and pattern as inkling (window 512 and 5:1, meta is 2k and 3:1). they use a similar global attention scaling as llama4 ( which still makes it position dependent even if they don't use RoPE they use a residual depth scaling, quite similar to muse open but "fixed", tho it's not really fixed because you have a learned norm nearby. you can actually plot both the effective residual contributions and you see that it's not super different from muse they use gated attention (with a per head bias for the full attention layer), QK Norm, Sandwich norm, similar softcapping before the LM head (like gemma and also muse), no shared experts the shape is pretty similar to other american oss open weight models (but tbf, mimo v2 flash and stepfun 3.5 and 5 also have this shape, it's not only us labs 👀) model weight:
Show more