Hugging Face Announces Tokenizers v1 with Significant Performance Improvements

Hugging Face has announced the release of tokenizers v1, a major update to its library designed to improve the efficiency of text tokenization in machine learning workflows. According to the announcement, the new version maintains compatibility with existing models and APIs while delivering significant performance gains.

The tokenizer v1 is designed to produce the same token IDs as the previous version (v0.23) while enhancing the overall performance. The improvements are achieved through several optimizations, including a hand-written splitter replacing a regex engine, a word cache to avoid repeated merging, and a merge loop that minimizes allocator interactions. These changes result in a 3 to 30 times faster encoding speed for text compared to v0.23 on an Apple M4 Max with one thread, scaling at 76% of linear across eight workers.

The release candidate for tokenizers v1 is now available on crates.io, and users can install it using the following command:

cargo add tokenizers --pre

For those who only need encoding functionality, the training feature can be disabled to exclude the C++ dependency:

cargo add tokenizers --pre --no-default-features --features http

The next steps for Hugging Face include expanding support for more model families and integrating the improvements into the transformers library and the broader ecosystem.

This update is part of Hugging Face's ongoing efforts to enhance the performance and usability of its tools for the machine learning community.

Source: huggingface