Lysandre(@LysandreJik):Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size. https://t.co/F4PTDDZbuy

Lysandre

@LysandreJik

Chief Open-Source Officer (COSO) at Hugging Face

加入 March 2019

647 正在关注 12.5K 粉丝

Lysandre@LysandreJik

2026.09.21 14:02

Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size.

显示更多