gh-ollama Releases v0.34.2-rc2 with Memory Optimization

gh-ollama has released version v0.34.2-rc2, which includes a memory optimization feature to release freed KV buffers during speculative decode. The update addresses an issue where the decode loop releases MLX's pool of freed buffers every 256 generated tokens, but speculative decoding often skips this boundary, leaving several GB of buffers unreleased. This results in a continuously growing memory footprint until the system runs out of memory.

With the new release, the pool is now released whenever a round crosses a multiple of 256 tokens, similar to what a single-token round already did. This optimization ensures that the runner's memory footprint remains stable during long speculative generations. For example, with qwen3.8:27b-mlx at a 98k-token context on a 128 GB machine, the runner previously grew past 90 GB and panicked the kernel. Now, it stays flat at 30 GB.

Source: gh-ollama