What is VRAM, and how much do you need for local AI?

VRAM (video RAM) is the memory soldered onto a GPU, and for local AI it is the single number that decides whether a model runs at all. A model's weights, its KV cache and a working buffer must all fit in VRAM; the moment they don't, layers spill to system RAM across the PCIe bus and generation speed collapses by an order of magnitude.

How to size it

At full precision (FP16) every billion parameters needs about 2 GB. Quantise to 4-bit and that drops to roughly 0.6-0.7 GB per billion. On top of the weights you pay for the KV cache, which grows with context length and batch size, plus 1-2 GB the driver and OS reserve.

VRAM ≈ params(B) × bytes_per_param
     + 2 × n_layers × d_model × context × batch × kv_bytes
     + ~1.5 GB overhead

What each tier runs

The trade-off: dedicated GPU VRAM has far more bandwidth than unified memory (≈1 TB/s vs ≈0.5 TB/s), so a 24 GB card generates faster than a 128 GB Mac on models that fit both — but the Mac runs models the card cannot load. Size the calculation with the hardware sizer.