You Jiacheng
@YouJiacheng
Joined August 2015
2.3K Following 12.2K Followers
cool. most of tokenization time is pre-tok (regex split). gigatoken uses specialized code for pre-tok.
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
Show more
