Sander Land
@magikarp_tokens
Breaking all the models with weird tokens
Joined March 2024
106 Following 1.6K Followers
✅ Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ 🤯
🔗
Show more
