Been working on nanoTitan for the past couple of weeks and I am excited to share the progress!
nanoTitan is a minimal distributed training codebase for MoE. Currently, DP, PP, EP as well as any of their compositions (up to 3D) have been implemented.
Links below
Show more
