Ollama released version 0.40.0-rc1, which updates tokenizer semantics to match the publisher's tokenizer. This change affects the pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges.
What happened
Ollama has released version 0.40.0-rc1, which includes updates to the tokenizer semantics to match the publisher's tokenizer. The changes honor the pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Additionally, the release handles empty added tokens and empty Metaspace input consistently. The update also includes shared Go/Python reference cases using published tokenizers, pulling missing models directly and failing on errors, and focused regressions for configuration precedence, byte fallback, and parallel encoding.
In the Fathom Layer index
- Ollama — Developer Tools
What to weigh
- The update includes handling empty added tokens and empty Metaspace input consistently.
- The release includes shared Go/Python reference cases using published tokenizers.
Source: Ollama
