Lysandre(@LysandreJik):Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size. https://t.co/F4PTDDZbuy

Lysandre

@LysandreJik

Chief Open-Source Officer (COSO) at Hugging Face

Joined March 2019

647 Following 12.5K Followers

Lysandre@LysandreJik

2026.09.21 14:02

Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size.