What is a Small Language Model (SLM)?
SLM stands for small language model: a language model built to run with far less memory and compute than a frontier large language model (LLM), often on a laptop, phone or single GPU. There is no official parameter cutoff that makes a model "small". The term is a vendor label, and in the examples below each vendor applies it to a different range of sizes.
What "small" means in practice
No standards body defines the term, so the useful question is how the companies that ship these models use it:
| Vendor | Models it describes as small or lightweight | How it frames them |
|---|---|---|
| Microsoft (Phi-3) | Phi-3-mini (3.8B), Phi-3-small (7B), Phi-3-medium (14B) | Microsoft's announcement calls the Phi-3 family small language models and launched Phi-3-mini first, saying Phi-3-small and Phi-3-medium would follow shortly; its technical report says phi-3-mini is "small enough to be deployed on a phone" |
| Meta (Llama 3.2) | 1B and 3B text-only models | "Lightweight" models that fit onto select edge and mobile devices |
| Google (Gemma 4) | E2B and E4B ("effective" parameter sizes) | Small sizes "built for ultra-mobile, edge, and browser deployment" |
| Alibaba (Qwen3) | 0.6B to 32B dense models; 30B-A3B mixture-of-experts | Qwen calls Qwen3-4B "a tiny model" and the 30B-A3B "the small MoE model" |
Read the table as usage, not as a definition. Microsoft applies "small language model" to a 14-billion-parameter model; Qwen calls a model with 30 billion total parameters, 3 billion of them activated, a "smaller MoE model". Google notes that its E-series models need more memory to load than their effective parameter count suggests, because of per-layer embedding tables. Parameter count alone does not tell you whether a model will fit your hardware.
How it works
An SLM is the same kind of model as an LLM, a transformer that predicts the next token, with fewer parameters. Vendors use a few techniques to keep capability while shrinking size:
- Knowledge distillation. A large "teacher" model trains a small "student". Hinton, Vinyals and Dean described compressing the knowledge of an ensemble of large models into a single model that is much easier to deploy. Meta says it used distillation and pruning for the Llama 3.2 1B and 3B models, incorporating logits from the Llama 3.1 8B and 70B models into pre-training.
- Pruning. Removing parts of a trained network and recovering as much of its knowledge as possible, which Meta describes as its route to the smaller Llama 3.2 models.
- Curated training data. Microsoft's technical report says phi-3-mini was trained on heavily filtered publicly available web data and synthetic data.
- Quantization. Storing weights at lower precision after training. Google's Gemma 4 documentation lists approximate memory to load Gemma 4 E4B at 17.9 GB in 16-bit, 8.9 GB in 8-bit and 4.5 GB in 4-bit. The GPTQ paper reports 3- or 4-bit quantization with negligible accuracy loss, but that headline result is for models with around 175 billion parameters. The same paper finds that larger models generally appear easier to quantize, so a small model can lose more quality at the same bit width. Quantization is not specific to SLMs; it can be what lets a small model fit a device's memory (VRAM on a GPU), at a quality cost worth measuring.
Why SLMs exist
Each benefit below is a vendor's claim, attributed to the vendor that makes it:
- Running on the device. Microsoft says small models suit "resource constrained environments including on-device and offline inference scenarios." Meta says its 1B and 3B models let developers build on-device applications "where data never leaves the device." See on-device AI and NPU.
- Latency. Meta says local processing can make prompts and responses feel instantaneous; Microsoft says Phi-3's lower computational needs give "much better latency."
- Cost. Microsoft lists cost-constrained use cases, "particularly those with simpler tasks", and says the smaller size makes fine-tuning easier and more affordable.
- Privacy. Meta says running locally avoids sending data such as messages and calendar information to the cloud.
None of these vendors publishes a general cost or latency ratio between SLMs and LLMs, and this page does not offer one: the gap depends on the hardware, the runtime, the quantization level and the task.
SLM vs LLM: the trade-offs
| Small language model | Large language model | |
|---|---|---|
| Where it runs | Laptop, phone, edge device or a single GPU, depending on size and quantization | Usually a data centre, reached through an API |
| Data path | Can stay on the device when run locally | Leaves the device unless self-hosted |
| Stored knowledge | Less. Microsoft says Phi-3 models "do not perform as well on factual knowledge benchmarks" because the smaller size leaves less capacity to retain facts | More room to retain facts |
| Fine-tuning | Cheaper and easier, per Microsoft | More compute to adapt (inference from size; not quantified in these sources) |
| Vendor benchmark claims | Microsoft says phi-3-mini rivals Mixtral 8x7B and GPT-3.5 on its benchmarks and internal testing; Qwen says Qwen3-4B can rival Qwen2.5-72B-Instruct | Vendors position their largest models for the hardest tasks |
Benchmark claims in the table come from the companies that train the models, measured with their own pipelines. They are not independent measurements, and this index does not test models.
When to use one, and when not to
Consider an SLM when the task is narrow and well defined, data should not leave the device, the product must work offline, or a per-request API bill is a constraint. Microsoft's own list of good fits (on-device, latency-bound, cost-constrained simpler tasks) is a reasonable starting checklist.
Avoid relying on one when the answer depends on broad factual recall, which Microsoft identifies as a weakness, or when you cannot measure quality on your own inputs. A small model that scores well on a vendor's benchmark can still fail on your data. That caution is inference from how the benchmarks are produced, not a measured result, so evaluate on a sample of your real workload before committing.
In the index
- Memory estimates for small models: Phi-3 Mini, Llama 3.2 1B, Llama 3.2 3B and Qwen 2.5 7B in the hardware calculator.
- Running models locally: Ollama, the local LLM hardware optimization guide and is an NPU laptop worth it?.
- Related terms: quantization, VRAM, on-device AI, NPU.
Vendor statements on this page were checked against their pages on 29 September 2026.
Sources
- Microsoft, "Introducing Phi-3: Redefining what's possible with SLMs" (Phi-3 called small language models; Phi-3-mini 3.8B available at launch, Phi-3-small 7B and Phi-3-medium 14B announced as coming shortly; good for on-device/offline, latency-bound and cost-constrained simpler tasks; easier, more affordable fine-tuning; "much better latency"; weaker on factual knowledge benchmarks): https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/
- Abdin et al. (Microsoft), "Phi-3 Technical Report", arXiv:2404.14219 (phi-3-mini 3.8B, "small enough to be deployed on a phone"; rivals Mixtral 8x7B and GPT-3.5; filtered web and synthetic training data): https://arxiv.org/abs/2404.14219
- Meta, "Llama 3.2: Revolutionizing edge AI and vision with open, customizable models" (1B and 3B lightweight models for edge and mobile; data never leaves the device; instantaneous-feeling local responses; pruning and distillation with logits from Llama 3.1 8B and 70B): https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/
- Google, Gemma models overview (Gemma 4 small sizes E2B and E4B for ultra-mobile, edge and browser; memory by precision; per-layer embeddings raise load memory above effective parameters): https://ai.google.dev/gemma/docs/core
- Qwen Team, "Qwen3: Think Deeper, Act Faster" (dense models 0.6B to 32B; 30B-A3B called the small MoE model, 30 billion total and 3 billion activated parameters; Qwen3-4B called a tiny model that can rival Qwen2.5-72B-Instruct): https://qwenlm.github.io/blog/qwen3/
- Hinton, Vinyals, Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531 (knowledge distillation into a single model that is easier to deploy): https://arxiv.org/abs/1503.02531
- Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", arXiv:2210.17323 (3 or 4 bits per weight with negligible accuracy degradation on ~175B-parameter models; larger models generally appear easier to quantize): https://arxiv.org/abs/2210.17323