注册并分享邀请链接,可获得视频播放与邀请奖励。

elie
@eliebakouch
research @PrimeIntellect (prev: @huggingface) anon feedback:
加入 January 2024
4.8K 正在关注    26.8K 粉丝
huge congrats to reflection on the release, very excited that they will release the weight under apache 2.0!! i'm quite confused by what seems to be a very low MFU for pre training. if you assume ~80% goodput for 4 week (which is already low) you get around ~12% BF16 MFU arch seems to be global/SWA interleaved with a 3:1 ratio, with slightly lower parameter sparsity than other oss models (~5%). their training also seems to be quite stable and they get a better base on held out code ppl than dsv4!
显示更多
0
5
180
5
转发到社区