You Jiacheng
@YouJiacheng
加入 August 2015
2.3K 正在关注 12.2K 粉丝
cool. most of tokenization time is pre-tok (regex split). gigatoken uses specialized code for pre-tok.
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
显示更多
