Lysandre(@LysandreJik):Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size. https://t.co/F4PTDDZbuy

Lysandre

@LysandreJik

Chief Open-Source Officer (COSO) at Hugging Face

가입 March 2019

647 팔로잉 중 12.5K 팬

Lysandre@LysandreJik

2026.09.21 14:02

Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size.