Register and share your invite link to earn from video plays and referrals.

elie
@eliebakouch
research @PrimeIntellect (prev: @huggingface) anon feedback:
Joined January 2024
4.8K Following    26.7K Followers
huge congrats to reflection on the release, very excited that they will release the weight under apache 2.0!! i'm quite confused by what seems to be a very low MFU for pre training. if you assume ~80% goodput for 4 week (which is already low) you get around ~12% BF16 MFU arch seems to be global/SWA interleaved with a 3:1 ratio, with slightly lower parameter sparsity than other oss models (~5%). their training also seems to be quite stable and they get a better base on held out code ppl than dsv4!
Show more