jietang
@jietang
Professor @ Tsinghua, Founder of
AGI, LLM.
“The value of a man should be seen in what he gives and not in what he is able to receive.”―Einstein
Joined May 2008
457 Following 76.6K Followers
cool comment: FLOPs were intelligence; parameters were knowledge!
An excellent history of scaling laws from @jietang.
In 2020, we explored the limits of sparsity in Switch Transformers by routing each token to only 1 out of 2048 experts (in retrospect, a bold choice). The model had fewer than 3B activated parameters, but 1.6T total parameters (comparable to today's frontier models).
The 1.6T model achieved better C4 perplexities than the T5 models using far less compute, set a new SOTA on TriviaQA, but was dumb as bricks on reasoning tasks like SuperGLUE.
The lesson was that the optimal tokens-per-parameter ratio is highly task-dependent. Or as @NShazeer had already intuited: FLOPs were intelligence; parameters were knowledge!
Show more
