You Jiacheng
@YouJiacheng
가입 August 2015
2.3K 팔로잉 중 12.2K 팬
cool. most of tokenization time is pre-tok (regex split). gigatoken uses specialized code for pre-tok.
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
더 보기
