What Is DeepInfer? DeepSeek's Engine vs vLLM

DeepInfer is the inference framework DeepSeek uses to serve its own models in production, released on GitHub under the Apache 2.0 license in October 2026. The open release runs offline (batch) inference for one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 class GPUs or Huawei Ascend 950 NPUs. It is not a local runner and not yet a full serving stack: DeepSeek left out the KV cache system, prefill and decode disaggregation, and load balancing. For a home machine, stay with llama.cpp or Ollama. For a GPU cluster that serves an API today, vLLM or SGLang is still the practical pick.

The repository appeared on October 16 and went to the top of Hacker News on October 20, where it passed 2,000 points. Below is what is actually in the code, what it needs, and how it compares with the engines most teams already run.

What DeepInfer is

DeepInfer is the engine behind DeepSeek's hosted service, per the DeepInfer README. The README says it was originally built mainly on vLLM, with ideas taken from SGLang and NVIDIA's TensorRT LLM. So think of it as a heavily specialized descendant of vLLM, tuned for one model family on DeepSeek's own hardware, rather than a general engine that loads any checkpoint.

The supported model list has a single entry: DeepSeek-V4.1-Flash. The default checkpoint is DeepSeek-V4.1-Flash-1017, the October 17 update. Its model card lists the weights under the MIT License with a 1M token context window, and Hugging Face reports about 763 billion parameters stored mostly in 8 bit formats. To run that same model on your own hardware instead, see how to run DeepSeek V4.1 Flash locally.

DeepInfer hardware requirements

This is where most readers can stop. The README lists two supported targets:

There is no consumer GPU, Mac or CPU path. The published performance charts were measured on 4 SM100 GPUs for prefill and 64 for decode. The example commands split the model with expert parallelism across 8 devices (--ep 8) or combine expert parallelism with data or context parallelism (--ep 2 --dp 8, --ep 2 --cp 8). In plain terms, you need at least a full multi GPU server, and DeepSeek's own numbers come from much larger groups.

Weight size matters too. The model has roughly 763B parameters, which is hundreds of gigabytes of files before any KV cache. Our VRAM guide explains how to estimate that budget.

What the open release includes

The README lists these features:

• Optimized kernels. It pulls in DeepSeek's FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect libraries plus custom TileLang kernels.
• Parallelism. Attention data parallelism with balanced context parallelism, and MoE expert parallelism with expert load balancing (EPLB).
• DSpark draft decoding. The model proposes draft tokens and a confidence score decides when to verify them, controlled by DEEPINFER_ENABLE_MTP_VARLEN=1. The default maximum draft length is 5. If the idea is new to you, read speculative decoding vs MTP first.
• Bounded replay for sliding window attention. Prefill replays only a short suffix of the prompt instead of the whole thing, which cuts prefill work.
• Asynchronous scheduling. CPU batch scheduling overlaps with GPU work, on by default.
• Batch invariant inference. With DEEPINFER_BATCH_INVARIANT=1, a request returns the same result no matter what else is in the batch (NVIDIA only).
• Deterministic sampling. Same input tokens, same parameters and a fixed seed give the same output (NVIDIA only).
• Structured outputs. Generation can be constrained to a regex, a JSON schema or an EBNF grammar.

Deterministic output is the feature most likely to matter outside DeepSeek. Most engines can return slightly different text for the same prompt when the batch changes, which makes evals and regression tests noisy.

What is missing

The README is direct about this. A complete inference service also needs three parts that are not in the release:

1. A KV cache system (one Hacker News commenter pointed this out as the notable gap).
2. Disaggregation of the vision encoder, prefill and decode stages onto separate workers. DeepSeek says most of the engine support is already there.
3. Load balancing across data parallel replicas for prefill and decode.

So today you can run offline batch jobs with examples/offline_inference.py, benchmark the engine and study the kernels. You cannot point an OpenAI compatible client at it the way you can with vllm serve. There are no tagged GitHub releases yet either, so you build from source with ./build.sh after setting DEEPINFER_ACCELERATOR=nvidia or ascend.

DeepInfer vs vLLM and SGLang

Both general engines are already tuning for this model. The vLLM repository has open pull requests from October 2026 for V4.1 Flash decode on GB300 and AMD MI325X support, and the SGLang repository has work on prefill and decode splits for the same model. Expect some DeepInfer ideas to show up there, since the projects share roots. For the general tradeoffs between engines, see vLLM vs llama.cpp. If you are picking between those two general engines for an NVIDIA DGX Spark, see SGLang vs vLLM on DGX Spark.

Who should try DeepInfer

• Teams that already have Blackwell or Ascend 950 capacity and want to run large offline batches of DeepSeek-V4.1-Flash.
• Inference engineers who want to read how a frontier lab schedules MoE models, draft tokens and sliding window prefill.
• Researchers who need bit exact reproducible outputs for evaluation.

Who should skip it

• Anyone running models on a laptop or a single gaming GPU. Our guide to running DeepSeek locally with Ollama covers the distilled models that fit.
• Anyone who needs a hosted API endpoint this week. Use vLLM or SGLang.

The prompt format catch

The V4.1 Flash model card notes that the release ships without a Jinja chat template. Instead DeepSeek publishes a Python reference encoder in the model repository and a Rust toolkit with Python bindings called deepseek-recipe that converts Chat Completions and Responses style requests into the model's prompt format. Whatever engine you use, check that it encodes prompts this way, or tool calls and thinking mode will break in subtle ways.

How to try it

On a supported Linux machine:

git clone https://github.com/deepseek-ai/DeepInfer
cd DeepInfer
# create and activate an empty Python 3.12 environment first
export DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh
python examples/offline_inference.py --ep 8

The first run downloads the checkpoint into the Hugging Face cache. Set DEEPINFER_HF_MODEL_DIR to use a local copy instead. For ways to shrink the cache on other engines, our KV cache quantization guide explains the options.

FAQ

Is DeepInfer open source?

Yes. The code is under the Apache 2.0 license. The DeepSeek-V4.1-Flash weights it runs are separately released under the MIT License.

Can I run DeepInfer on an RTX 4090 or 5090?

No. The README supports only NVIDIA SM100 data center GPUs and Huawei Ascend 950 NPUs, and the model is far too large for one consumer card.

Is DeepInfer faster than vLLM?

DeepSeek publishes throughput charts for its own hardware but no direct comparison with vLLM, so there is no fair answer yet. On supported hardware it carries model specific kernels and scheduling that general engines are still adding.

Does DeepInfer have an OpenAI compatible API?

Not in this release. It does offline inference only, and DeepSeek lists the KV cache system, stage disaggregation and load balancing as missing pieces of a full service.

Which models does DeepInfer support?

Only DeepSeek-V4.1-Flash, with DeepSeek-V4.1-Flash-1017 as the default checkpoint.