Sander Land
@magikarp_tokens
Breaking all the models with weird tokens
加入 March 2024
106 正在關注 1.6K 粉絲
✅ Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ 🤯
🔗
顯示更多
