You Jiacheng
@YouJiacheng
参加 August 2015
2.3K フォロー中 12.2K ファン
cool. most of tokenization time is pre-tok (regex split). gigatoken uses specialized code for pre-tok.
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
もっと見る
