Arthur Zucker(@art_zucker):Happy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minimal package size and memory usage. https://t.co/HJ7T3bAQmF We're thankful for the players of this ecosystem that have pushed us to give the best we could!

Arthur Zucker

@art_zucker

Head of transformers @huggingface 🤗

Joined October 2021

679 Following 8.4K Followers

Arthur Zucker@art_zucker

2026.09.21 13:45

Happy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minimal package size and memory usage. We're thankful for the players of this ecosystem that have pushed us to give the best we could!