jietang
@jietang
Professor @ Tsinghua, Founder of
AGI, LLM.
“The value of a man should be seen in what he gives and not in what he is able to receive.”―Einstein
参加 May 2008
457 フォロー中 76.6K ファン
cool comment: FLOPs were intelligence; parameters were knowledge!
An excellent history of scaling laws from @jietang.
In 2020, we explored the limits of sparsity in Switch Transformers by routing each token to only 1 out of 2048 experts (in retrospect, a bold choice). The model had fewer than 3B activated parameters, but 1.6T total parameters (comparable to today's frontier models).
The 1.6T model achieved better C4 perplexities than the T5 models using far less compute, set a new SOTA on TriviaQA, but was dumb as bricks on reasoning tasks like SuperGLUE.
The lesson was that the optimal tokens-per-parameter ratio is highly task-dependent. Or as @NShazeer had already intuited: FLOPs were intelligence; parameters were knowledge!
もっと見る
