Happy to officially bring you the new SOTA `tokenization` library.
We focused on all languages, multi-thread scaling, minimal package size and memory usage.
We're thankful for the players of this ecosystem that have pushed us to give the best we could!
더 보기
