Marcel Rød
@marcelroed
PhD Student at Stanford working with Tatsu Hashimoto and Jure Leskovec. Previously MIT, Oxford, CERN
277 Following 2.8K Followers
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
Show more
