How much hardware do you actually need to run AI locally?

The honest answer is a range, and the number that sets it is not compute — it is memory. A model runs on your machine only if its weights, its KV cache and a working buffer fit in one pool of fast memory. Miss that by a gigabyte and generation speed drops by an order of magnitude as layers spill to system RAM.

The one calculation that matters

Weight memory ≈ parameters (in billions) × bits-per-weight ÷ 8.

• FP16: ~2 GB per billion.
• Q8: ~1.06 GB per billion — effectively lossless.
• Q4_K_M: ~0.61 GB per billion — the community default, minor quality loss.

Add the KV cache (~0.13 GB per 1k context for an 8B model, ~0.35 GB for a 70B) and ~1.5 GB of runtime overhead. The hardware sizer does this for a specific model.

What each budget realistically runs

Memory	Hardware	Ceiling (4-bit)	Feel
8–12 GB	RTX 4060 / MacBook Air 16GB	7–8B, short context	Snappy for small models
16 GB	RTX 4080 / Mac mini 24GB	13B, or 8B at long context	Comfortable
24 GB	RTX 3090 / 4090	32–34B	The single-GPU sweet spot
48 GB	2× 3090/4090, RTX 6000	70B at 4-bit (PCIe penalty)	DIY territory
64–128 GB unified	Mac Studio	70B+ without orchestration	Slower tok/s, huge capacity

Speed vs capacity, per dollar
  RTX 4090 (24GB, ~1 TB/s)   ── fast, capped at ~34B
  Mac Studio (128GB, ~0.8 TB/s) ── runs 70B+, ~40% slower per token

Buy for the workload, not the spec sheet

• Chat, coding help, RAG over your notes: a 24 GB GPU or a 32–64 GB Mac. An 8–14B model covers most of this.
• Running a 70B locally as a daily driver: unified memory (Mac Studio 64GB+) is the cheapest path. A dual-GPU rig is faster but louder, hotter and needs a real PSU and case.
• Fine-tuning or training: a discrete NVIDIA GPU. The CUDA ecosystem still owns this; Apple Silicon is inference-only in practice.
• Always-on small-model features (transcription, classification): the NPU in a current laptop, not a GPU.

The rule

Pick the largest model you actually need, size its memory with the calculator, then buy the cheapest hardware that clears it with ~15% headroom. Do not buy for the model you might run in a year — quantisation and smaller models keep making that machine cheaper.