How to Quantize LLM Models

Learning how to quantize LLM models means shrinking the weights so a model fits your RAM or VRAM with less accuracy loss than you fear. In practice you pick a path: convert then run llama-quantize in llama.cpp for portable GGUF files, load 4-bit or 8-bit weights on the fly with Hugging Face Transformers and bitsandbytes, or import an already-quantized GGUF (or create from FP16) into Ollama. For most local chat use, start from full-precision weights, convert to a high-precision GGUF, then quantize to Q4_K_M.

This guide is the hands-on recipe. For which GGUF type to pick (Q4_K_M vs Q5_K_M vs i-quants), see GGUF quantization explained. Weight quantization is also different from KV cache quantization, which shrinks the runtime context cache, not the model file.

What is LLM model quantization?

Quantization stores each weight (or group of weights) with fewer bits. A 16-bit weight file can become roughly 4–8 bits per weight, so disks and memory shrink and generation often speeds up because less data moves through bandwidth. The trade-off is some accuracy loss, usually measured as perplexity or KL divergence against the original model.

llama.cpp’s quantize README states the process explicitly: convert the original model to GGUF, then quantize that GGUF. It also notes that a suitable importance matrix (imatrix) can reduce accuracy loss. The same docs publish size examples for Llama 3.1: an 8B model drops from 32.1 GB original to about 4.9 GB at Q4_K_M; a 70B model from 280.9 GB to about 43.1 GB.

You will see quantization applied to open-weight families such as Llama, Qwen, and Mistral. The steps below are the same idea for each: start from the publisher’s safetensors (or a high-precision GGUF), then compress.

How to quantize LLM models: three practical paths

If you only need a downloadable chat model and a community already published good GGUFs on the Hugging Face Hub, downloading beats reinventing the quant. Quantize yourself when you merged a fine-tune, need a rare type, or want an imatrix tuned on your own text.

Path A: llama.cpp convert, then quantize

This is the standard GGUF pipeline documented in the llama.cpp quantize README.

1. Install tools and Python deps

Build llama.cpp so llama-quantize exists under build/bin/, and install the repo’s Python requirements (the README shows pip install -r requirements.txt, or uv pip install ...). For newer architectures you may need a current Transformers install; the README notes that some models need Transformers 5 even when the default requirements pin an older major.

2. Convert Hugging Face weights to a high-precision GGUF

From the llama.cpp tree:

python convert_hf_to_gguf.py \
  --outfile model-bf16.gguf \
  --outtype bf16 \
  --remote org/model-name

Notes from the same README:

• If the model is already on disk, pass the local directory and drop --remote.
• When the source is already 16-bit, --outtype auto (or omitting --outtype) is usually fine.
• Multimodal models need a separate mmproj GGUF for vision or audio; those components are usually kept at bf16 or Q8_0 because they are smaller and quality-sensitive.

3. Quantize with llama-quantize

./build/bin/llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

Q4_K_M is the type the README uses in its main example. Quantize from F16 / BF16 / F32 when you can. The tool’s --allow-requantize flag lets you re-quantize an already-quantized file, but the README warns that requantizing can severely reduce quality compared with starting from 16-bit or 32-bit.

Useful options when you outgrow the defaults:

• --imatrix file.gguf — use an importance matrix during quant.
• --leave-output-tensor — keep output.weight unquantized (larger file, sometimes better quality).
• --keep-split — emit shards matching a split input instead of one file.
• --output-tensor-type / --token-embedding-type — override types for sensitive tensors.

4. Optional: importance matrix path

An imatrix is built by running calibration text through the model so the quantizer can spend bits where they matter. Pass it with --imatrix. Low-bit and i-quant types benefit most; for everyday Q4_K_M many people skip it. Details and naming live in GGUF quantization explained.

5. No local build? Use GGUF-my-repo

The quantize README also points to the Hugging Face Space GGUF-my-repo, which builds quants without a local llama.cpp install (it syncs from llama.cpp main on a schedule). Unsloth can export GGUF after fine-tuning if that is already your training stack.

6. Run a quick sanity check

./build/bin/llama-cli -m model-Q4_K_M.gguf -p "Say hello in one sentence."

Then compare answers and speed against a higher quant if quality looks off. Leave headroom for context: see how much VRAM you need.

Path B: how to quantize LLM models on Hugging Face (bitsandbytes)

When you stay in PyTorch—chat demos, eval scripts, or QLoRA training—use bitsandbytes through Transformers. Official docs: Bitsandbytes quantization.

Install:

pip install --upgrade transformers accelerate bitsandbytes

8-bit load (about half the memory of full precision, per the docs):

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(
    "org/model-name",
    device_map="auto",
    quantization_config=quantization_config,
)

4-bit load (further compression; NF4 is recommended for training 4-bit bases):

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "org/model-name",
    device_map="auto",
    quantization_config=quantization_config,
)

Hardware notes from the same page: LLM.int8() needs NVIDIA Turing or newer; NF4/FP4 needs Pascal or newer on CUDA. Nested (double) quantization can save extra memory with no extra performance cost claimed in the docs. For training adapters on a 4-bit base, pair this with PEFT / QLoRA rather than full fine-tuning—our LoRA vs QLoRA guide covers that choice.

This path does not produce a GGUF. If you need Ollama or llama.cpp later, merge or export full-precision (or high-precision) weights first, then use Path A.

Path C: how to quantize a LLM model in Ollama

Ollama’s current import docs are clear: Ollama does not quantize GGUF models during import. Prepare the GGUF with llama-quantize (Path A), then point a Modelfile at it.

Import a quantized GGUF

FROM /path/to/model-Q4_K_M.gguf

For split shards, keep original filenames and use a wildcard:

FROM /path/to/model-*.gguf

Create and run:

ollama create my-model
ollama run my-model

You can also import safetensors by setting FROM to the weights directory (or FROM . if the Modelfile sits next to them), then ollama create my-model.

Quantize from an FP16-style source via the create API

When the source is already an Ollama model (or other create-supported high-precision input), the Create a model API accepts a quantize field (examples in the OpenAPI samples use levels such as q4_K_M). That is separate from GGUF import: do not expect FROM some.gguf plus a quantize flag to re-quantize a GGUF—the import page says to quantize GGUF elsewhere first.

For runner trade-offs after you have a model, see Ollama vs llama.cpp vs LM Studio.

Choosing a quant type (short version)

Always leave memory for the KV cache and your context length. Shrinking the cache is a different knob—documented in KV cache quantization.

Common mistakes

1. Re-quantizing a Q4 into a Q3. Prefer F16/BF16/F32 sources; the llama.cpp README warns requantize quality loss.
2. Quantizing for Ollama by dropping a raw safetensors folder and expecting a specific GGUF type. Import safetensors or a prepared GGUF; for GGUF types, run llama-quantize yourself.
3. Confusing weight quants with KV-cache quants. One changes the model file; the other changes runtime cache memory.
4. Ignoring multimodal pieces. Vision/audio need mmproj files kept at higher precision.
5. No disk for intermediates. The README’s Llama 3.1 table shows originals at tens to hundreds of GB before Q4_K_M; plan disk and RAM for the convert step.

FAQ

What is the fastest way to learn how to quantize LLM models?

If you want a local chat file: convert with convert_hf_to_gguf.py, run llama-quantize ... Q4_K_M, then FROM that file in an Ollama Modelfile. If you want Python GPU inference: load with BitsAndBytesConfig(load_in_4bit=True). Download a trusted Hub GGUF when you do not need a custom quant.

How do I quantize an LLM with Hugging Face tools?

Install transformers, accelerate, and bitsandbytes, then pass a BitsAndBytesConfig into AutoModelForCausalLM.from_pretrained as shown above. That path quantizes for PyTorch runtime (and QLoRA-style training). It is not a substitute for GGUF export.

Can Ollama quantize a GGUF for me?

No. Per Ollama’s import documentation, prepare and quantize GGUF with a tool such as llama.cpp’s llama-quantize, then import with FROM /path/to/file.gguf.

Do I need an imatrix to quantize models?

Not for a first Q4_K_M. Use an imatrix when you push to lower bit rates or care about squeezing maximum quality at a given size. Build it from calibration text that resembles your real prompts.

Is weight quantization the same as KV cache quantization?

No. Weight quantization compresses the model parameters on disk and in memory. KV cache quantization reduces memory for the growing context during generation. You can use both, but they solve different problems.

Should I quantize before or after fine-tuning?

Fine-tune from high-precision (or QLoRA 4-bit) weights in the training stack, merge adapters if needed, then convert and quantize to GGUF for local runners. See how to fine-tune an LLM locally for the training side.