Aran Komatsuzaki
@arankomatsuzaki
Sharing AI research. Early work on AI (GPT-J, LAION, scaling, MoE). Ex ML PhD (GT) & Google.
351 Following 173.6K Followers
Follow-up on non-English token-inefficiency with more model-language pairs:
- Chinese is cheaper than English on major Chinese models
- Gemini and Qwen provide least non-English tax
- Anthropic has the highest tax by far; Kimi is next
- Hindi is the worst-covered language here, despite its massive speaker base
Show more
The non-English tax is real.
Sutton's Bitter Lesson, translated across languages and normalized to OpenAI English token count:
Hindi: OpenAI 1.37×, Anthropic 3.24×
Arabic: OpenAI 1.31×, Anthropic 2.86×
Chinese: OpenAI 1.15×, Anthropic 1.71×
Claude’s tokenizer charges a much higher linguistic tax.
Show more
