Running and fine-tuning open models on Windows (WSL2, DirectML, CUDA)

First, a correction the old guides got wrong: GGUF is an inference format, not a training format. You run GGUF models (llama.cpp, LM Studio, Ollama); you fine-tune in PyTorch and then convert. This guide covers both paths on Windows.

The three routes on Windows

Route	Hardware	Best for	Cost
Native (llama.cpp / Ollama)	Any GPU (Vulkan) or CPU	Just running GGUF models	Zero setup
WSL2 + CUDA	NVIDIA only	Fine-tuning, PyTorch, vLLM	Install WSL2 + NVIDIA's WSL driver
DirectML	AMD / Intel / NVIDIA	Fine-tuning on non-NVIDIA GPUs	Slower than CUDA, wider hardware

To just run models

Install Ollama or LM Studio. They ship a Vulkan/DirectML backend and pull GGUF quants directly. Nothing else to do. Pick a Q4_K_M quant sized to your VRAM.

To fine-tune with an NVIDIA GPU

1. wsl --install → Ubuntu.
2. Install the NVIDIA driver for WSL on Windows (not inside WSL); CUDA toolkit inside WSL.
3. pip install torch (CUDA build), then your trainer — Axolotl or Unsloth for LoRA/QLoRA.
4. QLoRA on a 7–8B model fits in 12–16 GB. A full fine-tune of anything large does not fit consumer hardware.
5. Merge the adapter, convert to GGUF with llama.cpp/convert_hf_to_gguf.py, quantise, run.

To fine-tune on AMD/Intel

Use torch-directml. It works and reaches a wide range of GPUs, but expect roughly half the throughput of an equivalent CUDA card and thinner library support. For serious training, an NVIDIA card pays for itself in time.

Failure modes

• Old WSL2 without GPU passthrough — update Windows and WSL; nvidia-smi must work inside Ubuntu.
• Installing CUDA inside WSL AND a full driver — you only need the toolkit inside; the driver is a Windows install.
• Trying to full-fine-tune a 13B on a 24 GB card — use QLoRA or rent a cloud GPU for a day.

The rule

Running models: Ollama/LM Studio, done. Fine-tuning: WSL2 + CUDA if you have NVIDIA, DirectML if you don't, and QLoRA either way unless you're renting real hardware.