MACaQ (Model Agnostic Calibrated Quantization) is an open source 4 bit quantization method from Huawei's Computing Systems Lab in Switzerland. It shrinks both the weights and the activations of a model to 4 bits (W4A4, using the MXFP4 format) and calibrates each linear layer on its own, so the same recipe works on text, image, video and audio models without custom code for each architecture. The code is on GitHub under Apache 2.0 with backends for llama.cpp, vLLM, Diffusers and PyTorch, and the method is described in arXiv 2611.10649.
For local model users the most practical piece is MACaQ-R, a refinement pass that improves an existing K quant GGUF while keeping its format, so the result still loads in standard llama.cpp. The repository went public on November 10, 2026 and reached the Hacker News front page the next day. Below: what MACaQ quantization does, the numbers the authors publish, how the backends differ, and who should try it now. We have not run it ourselves yet (more on that below), so every result here is the authors' own measurement.
MACaQ quantization in plain terms
Quantization stores a model's numbers with fewer bits so it needs less memory and runs faster. Most local users meet it through GGUF files, where only the weights are compressed and the math still happens at higher precision. If that is new to you, start with GGUF quantization explained.
MACaQ goes further on GPUs: it also quantizes the activations, the numbers flowing between layers while the model runs. That is harder, because activations often contain a few very large values (outliers) that ruin a 4 bit grid. The usual fix is to rotate the data so the outliers spread out, but many existing methods push those rotations across neighboring layers, which means writing special handling for every model family.
MACaQ's idea is to keep every change inside one linear layer. Text, image, video and audio models all share that building block, so the same quantizer can process any of them. The project page even explains the name: it is meant to be easy enough that "even a macaque could get the hang of it."
How MACaQ works
Each linear layer is handled in three steps, according to the paper and the project page:
- Channel balancing. A per channel scale evens out the size of the input channels so one hot channel does not dominate the 4 bit range.
- A small Hadamard transform. A block Hadamard rotation (32 channels per block in the published recipes) spreads outliers across neighbors. The matching inverse transforms are folded into the weights ahead of time, so the layer's output is mathematically unchanged before rounding.
- Flip Refinement. Instead of plain round to nearest, a Hessian aware coordinate descent step nudges individual 4 bit codes to an adjacent value when that lowers the layer's error on calibration data.
Because nothing crosses layer boundaries, the quantizer can stream through a model one layer at a time. That is why the authors report quantizing a 122B parameter model in 64 minutes using 32.5 GB of GPU memory. The tradeoff is that the scale and rotation run online at inference time. The paper measures that extra cost at 2.3 percent end to end over plain MXFP4 on Qwen3-32B in vLLM.
The published numbers
All figures below come from the MACaQ README, project page, model card and paper. We did not reproduce them.
| Claim | Model and setup | Source |
|---|---|---|
| Perplexity 2.8994 vs 2.8463 for BF16 | Qwen3.8-27B, MACaQ W4A4 in vLLM | GitHub README |
| Prefill 2.33x, decode 2.58x vs BF16 | Qwen3.8-27B, vLLM | GitHub README |
| Decode 2.74x vs FP16 | Qwen3.8-27B, llama.cpp | Project page |
| KL divergence 0.013367 vs 0.015731 at the same 22.36 GB | Qwen3.6-35B-A3B, MACaQ-R vs Unsloth UD-Q4_K_XL | Hugging Face model card |
| Matches the best Llama-3-8B perplexity of MR-GPTQ and DuQuant++ with 11 times less quantization time | Llama-3-8B | Paper abstract |
| 2.3 percent transform overhead | Qwen3-32B, vLLM, batch 1, context 512 | Paper, Figure 3 |
Two cautions when reading these. First, perplexity and KL divergence measure closeness to the original model, not task accuracy; for how quality loss shows up in practice, see our distillation vs quantization guide. Second, speedups depend on the GPU and kernels. The repository does not list a minimum GPU generation, so check the backend README for your card before you plan around a 2x figure.
The four MACaQ backends
MACaQ is a toolkit, not a single file format. The repository asks for Python 3.12 and a separate environment per backend.
| Backend | What it quantizes | Output | Runs in |
|---|---|---|---|
| llama.cpp | Text models at 2, 3 or 4 bits (IQ2_XXS to IQ4_XS) | GGUF plus a recipe sidecar | Bundled runtime, CPU supported |
| llama.cpp, MACaQ-R | Existing Q4_K, Q5_K, Q6_K MoE GGUFs | Same GGUF format, refined codes | Standard llama.cpp |
| vLLM | Dense Qwen3 and Qwen3.5 family text models, text MoEs | Compact checkpoint, W4A4 default | vLLM with the MACaQ plugin |
| Diffusers | FLUX.2 klein 4B, Wan2.2 TI2V 5B, a Stable Audio checkpoint | W4A4 safetensors | Diffusers |
| PyTorch | Qwen3 TTS and Chatterbox Turbo speech models | W4A4 checkpoint, WAV output | PyTorch |
There is also Ascend support: W4A4 matmul kernels have been submitted as CANN pull requests, though the README notes that running a full model needs integration with an Ascend inference runtime.
The detail that matters most for local users: a full MACaQ GGUF made with the activation transforms runs through the bundled llama.cpp fork in the repo (pinned to an upstream commit from August 28, 2026), driven by macaq.py run. Only MACaQ-R files are documented to load in a normal llama.cpp install. If you use Ollama or LM Studio, MACaQ-R is the part you can use today.
MACaQ-R: better GGUFs for standard llama.cpp
MACaQ-R takes a quantized mixture of experts GGUF you already have, plus the original Hugging Face weights and a full precision teacher GGUF, and refines the Q4_K, Q5_K and Q6_K weight codes. Block scales, tensor types and file size stay the same; other tensors are copied unchanged. The defaults are 64 refinement iterations over 131,072 calibration tokens.
The first public release is Qwen3.6-35B-A3B MACaQ-R Q4_K_XL, built from the Unsloth UD-Q4_K_XL file. It is the same 22.36 GB size, and the card reports lower KL divergence to FP16 (0.013367 vs 0.015731). It is a text only pack without the MTP prediction block. To try it:
hf download huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF \
Qwen3.6-35B-A3B-MACaQ-R-Q4_K_XL.gguf --local-dir models/macaq-r
llama cli -m models/macaq-r/Qwen3.6-35B-A3B-MACaQ-R-Q4_K_XL.ggufUse llama serve with the same file for an OpenAI compatible endpoint. Hardware needs match any 22 GB GGUF of that model; our Qwen 3.6 local setup guide covers memory tiers and context settings.
How to quantize your own model with MACaQ
Making your own MACaQ GGUF is a heavier job than running llama-quantize. The llama.cpp path in the repo looks like this:
- Create a Python 3.12 virtual environment and install
requirements/llamacpp.txt(torch 2.11.0, transformers 4.57.6 and friends). - Build the bundled llama.cpp with CUDA, or follow the CPU only instructions (slower).
- Download the original Hugging Face checkpoint and convert it to an FP16 GGUF with
convert_hf_to_gguf.py. - Prepare representative calibration text, and keep your evaluation text separate.
- Run
macaq.py quantizewith a format such asIQ4_XSand--had-block 32or64. Raising--vq-sweepsfrom 2 to 4 or 8 can improve quality at the cost of time, with no change to file size. - Load the result with
macaq.py run, optionally after compiling preprocessing kernels for your GPU.
Plan for disk space. The Qwen3.6-35B-A3B MoE recipe asks for about 220 GB of working space for an output GGUF of about 18.86 GB, and the Qwen3.8 Flash Next recipe produces a GGUF of about 125 GB. For general quantization paths with standard tools, see how to quantize LLM models.
MACaQ vs the quantization tools you already use
| Tool | What it compresses | Typical use |
|---|---|---|
| llama.cpp K quants and IQ quants | Weights only | CPU and GPU local inference, Ollama, LM Studio |
| Unsloth Dynamic GGUFs | Weights only, mixed per layer | Ready made local downloads |
| MACaQ full workflow | Weights and activations (W4A4) | GPU serving in vLLM, diffusion and speech models |
| MACaQ-R | Weights only, refined codes inside a K quant | Drop in GGUF for standard llama.cpp |
Pick by your runtime. If you serve text models in vLLM on a recent GPU, the W4A4 path is aimed at you. If you generate images or video with Diffusers, MACaQ is one of the few quantizers with a published recipe for FLUX.2 klein and Wan2.2. If you run GGUFs from Unsloth or Ollama, wait for MACaQ-R files of the models you use. If you are short on memory for long prompts, compressing the cache helps too; see KV cache quantization.
Pros and cons
Pros
- One method for text, image, video, audio and 3D models, Apache 2.0 licensed.
- Layer local design keeps memory low during quantization (one layer resident at a time).
- MACaQ-R files work in standard llama.cpp with no new runtime.
- Published commands, pinned revisions and resume support for long runs.
Cons
- Brand new: one MACaQ-R model on Hugging Face so far, no independent benchmarks yet.
- Full MACaQ GGUFs need the bundled llama.cpp fork, not Ollama or LM Studio.
- Quantizing big MoE models needs hundreds of gigabytes of disk and a capable GPU.
- Speed claims come from the authors' hardware, which the README does not fully specify.
Who should try MACaQ
- Local GGUF users with a 24 GB class GPU or a 32 GB Mac: try the Qwen3.6-35B-A3B MACaQ-R file against the Unsloth file you already have, on your own prompts.
- People serving models in vLLM who want 4 bit activations: test the W4A4 path on a Qwen3 or Qwen3.8 model and measure throughput yourself.
- Image, video and speech tinkerers: the Diffusers and PyTorch backends are the clearest published W4A4 recipes for those models.
- Everyone else: wait a few weeks for community GGUFs and outside comparisons.
How we checked this: on November 12, 2026 we read the MACaQ repository (README files for every backend, requirements and the pinned llama.cpp commit), the project page, the arXiv abstract and the MACaQ-R model card. We did not run MACaQ: our test box has no GPU, no Python 3.12 and not enough free memory for the CPU path.
FAQ
What does MACaQ stand for?
Model Agnostic Calibrated Quantization. "Model agnostic" means the same per layer recipe applies to any network built from linear layers, without architecture specific code.
Does MACaQ work with Ollama or LM Studio?
MACaQ-R GGUFs are documented to load in standard llama.cpp, so they should load in tools built on it once those tools support the base model. Full MACaQ GGUFs with activation transforms need the bundled runtime from the repository.
Is MACaQ better than Unsloth's GGUFs?
On one published comparison, yes: the Qwen3.6-35B-A3B MACaQ-R Q4_K_XL file reports lower KL divergence to FP16 than the Unsloth UD-Q4_K_XL file it was refined from, at the same 22.36 GB. That is the authors' measurement on one model, not an independent test.
Can I run MACaQ on a CPU?
The llama.cpp backend has CPU only install instructions and supports CPU execution, but the README warns that quantization takes longer. The vLLM, Diffusers and PyTorch W4A4 paths are built around GPU kernels.
What is W4A4 quantization?
W4A4 means 4 bit weights and 4 bit activations. MACaQ uses the MXFP4 format, which stores values as 4 bit floating point numbers in blocks of 32 that share one scale.
Who made MACaQ and what license does it use?
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli at Huawei's Computing Systems Lab in Switzerland. The code and the MACaQ-R model are released under the Apache License 2.0.
