The short answer to LLM distillation vs quantization: distillation trains a new, smaller model (the student) to copy a larger one (the teacher), while quantization keeps the same model and stores its weights with fewer bits. Distillation changes the architecture and needs a training run with GPUs and data. Quantization changes only the number format and usually takes minutes on one machine. If you want a model you already like to fit your hardware, quantize it. If you need a much smaller model with a different parameter count, distill, and then quantize the student too.
Both shrink a model, which is why people mix them up. They shrink different things, cost very different amounts, and fail in different ways. This guide explains each method, compares them side by side, and shows how real open weight releases use them together.
LLM distillation vs quantization at a glance
| Item | Knowledge distillation | Quantization |
|---|---|---|
| What changes | A new, smaller network with fewer parameters | The same network, weights stored in 8, 4 or fewer bits |
| Needs training | Yes, a full training or fine tuning run | No for post training quantization (PTQ); yes for quantization aware training |
| Needs data | Yes, prompts and teacher outputs or teacher probabilities | Little or none (a small calibration set for imatrix, GPTQ or AWQ) |
| Typical compute | GPU cluster or at least one large GPU for hours to days | One laptop or workstation, minutes to an hour |
| Who usually does it | Model labs and teams with training budgets | Anyone running models locally |
| Speedup source | Fewer layers or narrower layers, so fewer operations | Less memory traffic per token, smaller files |
| Main risk | The student never fully matches the teacher | Accuracy drops as bits go down, worse at very low bit widths |
| Tools | TRL, Transformers, DeepSpeed, Megatron-LM | llama.cpp, bitsandbytes in Transformers, Ollama import |
What is LLM distillation?
Knowledge distillation is a training method. You run a large teacher model, record what it produces, and train a smaller student to reproduce it. The idea was popularised by Hinton, Vinyals and Dean in Distilling the Knowledge in a Neural Network (2015), which compressed the knowledge of an ensemble of models into a single model that was much easier to deploy. The student learns from the teacher's soft probabilities over all possible answers, which carry more information than a single correct label.
For language models there are two common flavours:
- Logit distillation: the student is trained to match the teacher's probability distribution over the next token. You need access to the teacher's logits, so this usually means you run the teacher yourself.
- Sequence level distillation: the teacher writes complete answers, and the student is fine tuned on that text like any other supervised dataset. This works even when you only see the teacher's text output.
A well known early result is DistilBERT, which its authors report is 40% smaller than BERT, 60% faster, and keeps 97% of BERT's language understanding capability. Those are the paper's own figures for that encoder model, not a rule that applies to every LLM.
Distillation in open weight releases
You have probably already used distilled models without thinking about it:
- DeepSeek-R1 distills. The DeepSeek-R1 model card says the six dense distilled checkpoints (1.5B, 7B, 8B, 14B, 32B and 70B, built on Qwen2.5 and Llama 3 bases) were fine tuned on samples generated by DeepSeek-R1. That is sequence level distillation: the big model's reasoning traces became the small models' training data. These are the tags most people pull when they run DeepSeek locally with Ollama.
- Gemma 2. Google's Gemma 2 technical report states that the 2B and 9B models were trained with knowledge distillation instead of plain next token prediction. See the Gemma tool page for the current family.
- Minitron. NVIDIA's Minitron paper pruned a pretrained 15B Nemotron model to 8B and 4B and then retrained them with distillation. The authors report needing up to 40 times fewer training tokens per model than training from scratch.
How you would distill a model yourself
The simplest route is sequence level: generate answers with a strong teacher, filter them, and run supervised fine tuning on a smaller base model with TRL or Axolotl. This is the same workflow as any fine tune, so our guide to fine tuning an LLM locally covers the mechanics. Check the teacher's licence first, because some licences restrict using outputs to train other models.
For logit distillation, TRL ships a GKDTrainer in its experimental module, documented in the TRL GKD trainer docs. It implements Generalized Knowledge Distillation from Agarwal et al., which trains the student partly on its own generated outputs with feedback from the teacher. The paper's point is that a student trained only on fixed teacher text sees a different distribution at inference time, when it has to continue from its own mistakes. In TRL you pass both a model and a teacher_model, and the lmbda setting controls how much of each batch is student generated (on policy) text.
What is quantization in LLMs?
Quantization is a storage and compute format change. A model trained in 16 bit floats has every weight stored in 16 bits. Quantization maps those weights to fewer bits per weight, often in small groups that share a scale, so the file and the memory footprint shrink. The model architecture, layer count and parameter count stay the same.
The llama.cpp quantize README gives concrete sizes for Llama 3.1: the 8B model goes from 32.1 GB original to 4.9 GB at Q4_K_M, and the 70B model from 280.9 GB to 43.1 GB. No training is involved. You convert the model to GGUF and run llama-quantize.
LLM quantization techniques
The main families you will meet:
- Post training quantization (PTQ): quantize a finished model. GGUF k quants in llama.cpp, bitsandbytes 8 bit and 4 bit loading in Transformers, GPTQ and AWQ all sit here. Some use a small calibration set (an importance matrix in llama.cpp) to decide which weights deserve more precision.
- Quantization aware training (QAT): simulate low precision during training or fine tuning so the model learns to tolerate it. This costs a training run, which puts it closer to distillation in effort.
- KV cache quantization: shrinks the runtime context cache rather than the weights. See KV cache quantization.
For the hands on steps, read how to quantize LLM models. For picking between Q4_K_M, Q5_K_M and the i quants, read GGUF quantization explained.
LLM quantization vs performance
"Performance" means two things here, and the methods trade them differently.
Speed and memory. Local token generation is usually limited by how fast weights move from memory, so a 4 bit file that is a quarter of the size can generate faster on the same machine. Distillation also speeds things up, but through a different route: a 7B student simply does fewer operations than a 70B teacher.
Quality. Quantization loss grows as bits go down. Formats around 4 to 5 bits per weight are the common sweet spot for local chat, and quality loss becomes much more noticeable at very low bit widths. llama.cpp measures this with perplexity and KL divergence against the original model, so you can test a quant on your own text instead of guessing. A distilled student's quality depends on its size, the data and how long it trained. A 7B student will not match a 671B teacher such as DeepSeek-R1 on everything, even if it gets close on the tasks it was trained for. DeepSeek publishes per size benchmark tables on its card, and you can see the scores climb from the 1.5B distill to the 32B and 70B ones.
The rule of thumb: quantization keeps most of a given model's ability at a fraction of the memory, while distillation buys a much bigger size cut at the cost of a real training project and a less capable model.
Distillation and quantization together
These methods stack. The usual pipeline for local models is:
- A lab trains a large teacher.
- The lab distills smaller students (DeepSeek-R1 distills, Gemma 2 2B and 9B, Minitron).
- The community or the runner quantizes those students to GGUF or another low bit format.
- You run the quantized student on your laptop.
When you type ollama run deepseek-r1:14b, you are running a model that went through both steps: distilled from DeepSeek-R1 onto a Qwen2.5 base, then shipped as a quantized file. Our VRAM guide shows how to size that final file against your GPU.
Which should you use?
| Your situation | Use |
|---|---|
| You like a model but it does not fit your GPU or RAM | Quantize it, or download an existing quant |
| You want a faster version of a model for a narrow task and have training budget | Distill a smaller student, then quantize it |
| You need a reasoning style from a big model in a 7B to 14B size | Pick an existing distilled checkpoint first |
| You fine tuned a model and want to run it locally | Merge, then quantize |
| You are training a model family from scratch | Consider pruning plus distillation, as Minitron did |
| You have no GPU budget at all | Quantization only |
For most people reading this on a single machine, the honest answer is to quantize and to choose already distilled checkpoints when you need something smaller. Distilling your own student makes sense when you have a clear task, a dataset, a teacher whose licence allows it, and access to the compute in the training tools used by labs.
FAQ
Is quantization a form of distillation?
No. Quantization keeps the same model and changes how its numbers are stored. Distillation trains a separate, smaller model. Quantization aware training does involve training, but it still keeps the original architecture.
Is distillation better than quantization?
Neither is better in general. Distillation gives a bigger reduction in parameter count but needs data, a teacher and a training run. Quantization is cheap and keeps more of the original model's quality for each step of compression. Most local models use both.
What is the best LLM quantization method?
For local use with llama.cpp, Ollama or LM Studio, a 4 to 5 bit GGUF such as Q4_K_M is the common starting point, with an importance matrix when you quantize yourself. For GPU work in Python, bitsandbytes 4 bit loading is the easy path, especially for QLoRA style fine tuning.
Can you quantize a distilled model?
Yes, and it is the normal case. The DeepSeek-R1 distills, for example, are widely run as quantized GGUF files through Ollama and llama.cpp.
Does distillation need the teacher's weights?
Logit distillation needs the teacher's output probabilities, so in practice you run the teacher yourself. Sequence level distillation only needs the teacher's text answers. Check the teacher's licence before training on its outputs.
Does quantization make a model smarter or faster?
Faster and smaller, not smarter. A quantized model can only lose quality compared with the original weights, though at moderate bit widths the loss is often small.
