Sander Land
@magikarp_tokens
Breaking all the models with weird tokens
๊ฐ์
March 2024
106 ํ๋ก์ ์ค 1.6K ํฌ
โ
Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ ๐คฏ
๐
๋ ๋ณด๊ธฐ
