Sander Land
@magikarp_tokens
Breaking all the models with weird tokens
106 Following 1.6K Followers
✅ Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ 🤯
🔗
Show more
Got quite a bit further with figuring out Claude's tokenizer, and finally wrote it all up. Enjoy!
