llama-modchip: DeepSeek V4 Flash on CPU Only

llama-modchip is an open source set of patches and tools for llama.cpp that runs large mixture of experts models entirely on a CPU. Its headline claim is that the 284 billion parameter DeepSeek V4 Flash (the 0731 checkpoint) fits in a 64 GB desktop with no GPU, at about 9 to 15 tokens per second, by squeezing the model to an average of 1.71 bits per weight. The speed numbers come from the project itself and a handful of community reports, so treat them as promising but unconfirmed, and expect a real quality cost from a quant that small.

The project went up on GitHub on October 29, 2026, shipped versions 2.1 to 2.4 between November 2 and November 4, and reached the Hacker News front page on November 8 with more than 400 points and nearly 200 comments. This guide explains what it actually changes, what the numbers mean, the risks, and when plain llama.cpp gets you most of the way. Nothing here was run first hand; every figure is attributed to its source.

What llama-modchip is

The llama-modchip repository (Apache 2.0 license) describes itself as ggml and llama.cpp kernel patches, quantization transcoders and runtime tools for sparse mixture of experts (MoE) checkpoints on x86 processors with AVX2 or AVX-512 and on ARM64 chips. It is not a new inference engine. It wraps and patches llama.cpp, adds a ./modchip command line tool, and publishes its own GGUF files on Hugging Face.

It targets two models above all:

The reference machine is a laptop class AMD Ryzen 9 9955HX (16 Zen 5 cores, 64 MB of L3 cache) with 64 GB of dual channel DDR5. The README also lists Qwen3 30B A3B and Nemotron H 8B, plus guides for Windows, Apple Silicon, Fedora and AWS Graviton.

How it fits 284B DeepSeek V4 Flash into 64 GB

A 64 GB Linux machine has roughly 62.5 GiB usable, according to the project. Unsloth's smallest public build of the same checkpoint, UD-IQ1_S, is listed in the README at 76.87 GiB, which does not fit and ends in swapping or an out of memory kill. llama-modchip closes that gap with three changes.

1. Even smaller routed experts. Most of the 256 routed experts are pushed to IQ1_S (about 1.56 bits per weight) with a scale correction the author calls SwiGLU H2. Sensitive boundary layers stay at IQ2_XXS or IQ1_M.
2. A custom 64 weight block format. Some attention projections are transcoded to a new type the project calls Fractal 64 (GGML_TYPE_Q2_0, 3.75 bits per weight).
3. Keeping the important parts larger. Embeddings, the first dense layer, the shared experts and parts of the attention stack stay at Q4_K, Q5_K or 32 bit.

The result, per the Hugging Face model card, is a three part GGUF of about 56.6 GiB. The repo lists DeepSeek's MIT licensed DeepSeek V4 Flash 0731 as the base model. The README says peak memory with a 256K context is 57.61 to 58.9 GiB, so it just fits.

If quant names like IQ1_S and Q4_K are unfamiliar, our GGUF quantization explainer covers what each one means in bits and file size.

The CPU speedups it claims

The size trick only gets the model loaded. The speed claims come from kernel work in the patched ggml CPU backend:

• Cache aware matrix tiling for prefill. The README says prompt processing on the Zen 5 test machine rose from 25.0 to 44.36 tokens per second (a 77 percent gain) by processing several columns at once in a schedule sized to the L1, L2 and L3 caches.
• Streaming hints for expert weights. Cold expert weights are loaded with a non temporal hint so they do not push the KV cache out of L3.
• 2 MB huge pages. For Nemotron, the FAQ says stock llama.cpp runs at about 17 tokens per second because scattered expert reads cause millions of address translation misses per second, and huge pages plus thread pinning lift that to 25 to 33.
• A tiny speculative drafter. Instead of a neural draft model, a 1.64 MB lookup based drafter (named after the Stockfish chess engine) guesses upcoming tokens from text already in context. The project reports its biggest gains on code refactoring and tool calls, where output repeats what is in the prompt.
• Fast context restore. KV cache snapshots in /dev/shm let a long agent prompt (the example is Claude Code with 40 tools) reload in about 13 to 39 ms instead of being processed again.

By default the DeepSeek build evaluates all six routed experts per layer. An optional LLAMA_MOE_TOP_K setting of 5 or 4 trades a small measured quality drop for faster output.

What the community found

The Hacker News thread is a useful reality check, because several people ran it on their own hardware.

• One user on a Ryzen 7 5700U laptop chip with 64 GB of DDR4 reported a steady 12 to 14 tokens per second at about 15 watts.
• Another, on a Ryzen 9 7945HX with 96 GB of DDR5, said Nemotron ran "just about the same as plain old llama.cpp on the cpu", except prefill seemed much faster.
• Several commenters questioned quality at this size. One who had used DeepSeek V4 Flash 0731 said they were "exceedingly skeptical" of good results from a build this small. Others pointed out that the model has since been superseded by DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash.
• Many complained that the README is very hard to read and appears heavily AI written, which makes it difficult to tell measured results from projections.

The community leaderboard in the README lists submissions from about a dozen machines, mostly for the 30B Nemotron model, linked to GitHub issues. Those are self reported too.

Is a 1.71 bit DeepSeek worth running?

This is the real question, and the honest answer is: only if you need a very large model on hardware you already own, and you accept weaker output.

The project's own quality figures compare the modchip build against the same model at UD-IQ1_S, not against the full precision original. Agreement with an already 1 bit build tells you the patches did not break much further; it does not tell you how far the model is from the real DeepSeek V4 Flash. No independent benchmark of the Fractal build against the original existed when we checked on November 9, 2026.

General experience with quantization points the same way. Quality usually holds well down to about 4 bits per weight and falls faster below that, with 2 bit and lower builds most at risk on reasoning and exact facts. A smaller model at 4 bits, such as a 30B MoE, often beats a giant model at under 2 bits for daily work. Our guide to open source LLMs to run locally lists good options in that range.

Risks before you try it

• Custom tensor type. The Fractal files use a tensor type that stock llama.cpp, Ollama and LM Studio reject with an unknown tensor type 40 error. You are tied to the patched binary until the format is merged upstream, if it ever is.
• Prebuilt binaries. The quickstart downloads a precompiled llama-server-modchip-linux-x86_64 from the Hugging Face repo. Running an unaudited binary from a project a few weeks old is a real security tradeoff. Building from the published patches against llama.cpp source is the safer path.
• Remote script piping. The readiness check is a curl ... | python3 one liner. Download and read the script first.
• Fast moving code. Four releases landed in three days. Expect flags and file names to change.
• Old checkpoint. It targets DeepSeek V4 Flash 0731. For the newer 1017 release, see our guide to running DeepSeek V4.1 Flash locally.

How to try it

If you still want to test it, the README gives two routes. The automated one clones the repo and runs ./modchip chat, which detects your CPU, memory and GPU. The manual route downloads the GGUF and the patched server from Hugging Face:

hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
./dsv4/llama-server-modchip-linux-x86_64 \
  -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf \
  -c 32768 -t 16 -fa on --port 8080

Set -t to your number of physical cores, not threads. You need about 57 GiB of free RAM for the DeepSeek build and about 61 GB of disk space. Close browsers and other large programs first, and check that the machine is not using swap while it runs. The hf command comes from the Hugging Face Hub CLI.

The stock llama.cpp alternative

Much of what llama-modchip does on a hybrid machine is already in mainline llama.cpp. The llama-server documentation lists --cpu-moe to keep all expert weights on the CPU and --n-cpu-moe N to keep only the first N layers' experts there, plus --load-mode mlock to keep a model resident in RAM and --threads and --threads-batch for thread counts. Note that stock llama-server now listens on port 9931 by default, not 8080.

With a GPU of 12 to 24 GB and enough system RAM, putting attention on the GPU and experts in RAM is the well tested route, and it works with ordinary GGUF files from Unsloth and others. Our guide on whether a local LLM can use RAM instead of VRAM walks through those flags, and how to use llama.cpp covers installation.

Pros and cons

Pros

• Gets a 284B model running on a 64 GB machine with no GPU at all.
• Open source under Apache 2.0, with patches you can read.
• The CPU tuning ideas (huge pages, thread pinning, cache aware tiling) help on 30B models too.
• Community reports of usable speed on modest laptop chips.

Cons

• At 1.71 bits per weight, expect clearly weaker answers than the full model.
• Speed and quality figures are mostly self reported, and the documentation is hard to follow.
• Custom files lock you to a patched build.
• Targets an older DeepSeek checkpoint.

Who it is for

llama-modchip suits tinkerers with a 64 GB desktop or mini PC and no GPU who want to see a frontier sized open weight model run at home, and people interested in CPU inference tuning. It is not a good pick if you need dependable answers for work, if you cannot audit what you install, or if you already own a GPU, where stock llama.cpp with expert offload is simpler. To size hardware for other models, see how much VRAM you need for an LLM.

FAQ

What is llama-modchip?

It is an open source collection of llama.cpp patches, a quantization transcoder and a command line tool that runs large mixture of experts models on CPUs. Its best known result is DeepSeek V4 Flash 0731 (284B parameters) in under 59 GiB of RAM with no GPU.

Can DeepSeek V4 Flash run on CPU only?

Yes, if you have enough RAM. llama-modchip's 56.56 GiB build fits a 64 GB machine, and the project reports 9 to 15 tokens per second of output on a 16 core Zen 5 laptop chip. A standard 1 bit GGUF such as Unsloth's UD-IQ1_S is about 77 GiB on its own, so you need roughly 96 GB of RAM or more.

What quantization does llama-modchip use?

The DeepSeek build averages 1.71 bits per weight. Most routed experts are IQ1_S, some layers are IQ2_XXS or IQ1_M, some attention weights use the project's own Fractal 64 format at 3.75 bits, and key shared tensors stay at 4 to 5 bits or higher.

Does llama-modchip work with Ollama or LM Studio?

Not with its Fractal files. Those use a custom tensor type that unpatched llama.cpp, Ollama and LM Studio reject. You need the patched binary or a build from the project's patches.

Is llama-modchip faster than regular llama.cpp?

The project reports large prefill gains and 28 to 53 percent faster output on its test machines. One Hacker News tester on Zen 4 saw similar output speed to plain llama.cpp for Nemotron, with faster prefill. Results seem to depend heavily on the CPU and memory.

Is it safe to run?

The code is open, but the quickstart uses prebuilt binaries and a piped install script from a very new project. Read the scripts and build from source if you are cautious, and run it on a machine without sensitive data until it has a longer track record.