What are MLA and DeepSeekMoE?

Multi-head Latent Attention (MLA) and DeepSeekMoE are the two architecture choices that let DeepSeek-V3 run as a 671-billion-parameter model while activating only about 37 billion parameters per token. Both are set out in the DeepSeek-V3 technical report (arXiv 2412.19437, December 2024).

MLA — shrinking the KV cache

Standard attention caches a full key and value vector for every token in context; that KV cache is the main memory limit on long context windows. MLA instead compresses each token's key/value into a small low-rank latent vector and caches that, cutting KV memory sharply for a modest quality cost.

DeepSeekMoE — routing to a fraction of the model

A Mixture-of-Experts layer holds many "expert" sub-networks and routes each token to just a few. DeepSeekMoE adds two things:

• Shared experts that always run, handling common patterns so the routed experts do not each relearn them
• Auxiliary-loss-free load balancing — rather than an extra loss term fighting the main training objective to keep experts evenly used, it nudges a per-expert bias value during training

Why it matters

Together they are much of why DeepSeek-V3 trained for roughly 2.79 million H800 GPU-hours — well below comparable frontier models — and why its weights run on hardware that could never hold a dense 671B model.