注册并分享邀请链接,可获得视频播放与邀请奖励。

Ronan McGovern
@RonanKMcGovern
Building the realtime voice layer for intelligence. Private alpha: voice@trelis.com
加入 July 2010
741 正在关注    605 粉丝
Among open-weight models, do the Chinese have the best pre-training? I measured how well LLMs compress fresh, unseen human text (bits-per-byte). Of Western models, only NVIDIA's Nemotron sits squarely in the frontier cluster (Thinking Machines' Inkling is just behind). You can't measure compression directly for closed models like Fable or Sol... commercial APIs block the log probs you'd need. And my attempts to estimate it by sampling were too noisy. One corollary of challenges estimating logprobs is that KL/log-prob distillation against a commercial API is very hard. Sampling can't recover the tails of the probability distribution, and hidden top-p sampling truncates them altogether. That bpb is where it is for frontier Chinese models suggests to me there is quite strong pretraining on raw - not synthetic - text. [Which is not to say there isn't also lots of synthetic data, in both Western and Chinese models.]
显示更多