Sander Land
@magikarp_tokens
Breaking all the models with weird tokens
参加 March 2024
106 フォロー中 1.6K ファン
✅ Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ 🤯
🔗
もっと見る
