huge congrats to reflection on the release, very excited that they will release the weight under apache 2.0!!
i'm quite confused by what seems to be a very low MFU for pre training. if you assume ~80% goodput for 4 week (which is already low) you get around ~12% BF16 MFU
arch seems to be global/SWA interleaved with a 3:1 ratio, with slightly lower parameter sparsity than other oss models (~5%). their training also seems to be quite stable and they get a better base on held out code ppl than dsv4!