DeepInfer: Offline inference, service components

1. Supported models and hardware

DeepInfer is the inference framework powering DeepSeek's large-scale production services. This release provides offline inference through example scripts that run the model.

The supported model is DeepSeek-V4.1-Flash, and the license is Apache 2.0.

Supported hardware includes NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Adoption reviews should start with this combination of model and devices.

2. Processing efficiency and reproducibility

Asynchronous scheduling overlaps accelerator execution of current work with CPU preparation of the next request batch. It is enabled by default.

The release also includes a decoding acceleration feature that first generates candidate answer fragments, then uses confidence to determine when to verify them.

It offers consistent model results regardless of request batching, plus deterministic sampling for identical outputs with the same inputs and sampling settings. Both are NVIDIA-only. Deterministic sampling requires separate feature configuration and a fixed seed. A seed is a reference value for reproducing random choices.

Structured output constrains generated results to a specified grammar. Output formats can be restricted using regular expressions, JSON schemas, or EBNF grammars.

3. Building and preparing the model

The source is built on Linux and requires GCC 13.0 or later.

Create and activate an empty Python 3.12 environment before building. The repository provides the following build commands.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Accelerator-specific development environments are also required. NVIDIA requires CUDA 13, while the other supported accelerator requires CANN 9.2.

On its first run, the offline inference example downloads the default model files to the Hugging Face cache. An existing model copy can be used by setting its path through an environment variable.

4. Latency rises alongside throughput

Prefill is the stage that reads and processes input before generating an answer. DeepSeek's published graph shows both throughput and latency for this stage with uncached inputs.

The supported model was measured on 4 SM100 GPUs within one scale-up domain, a tightly connected group of devices. The parallel configuration is EP4DP4, with 384+32 expert modules sharing computation. The context length is 131,072, and the decoder replay span is 140 tokens.

At a batch size of 4,096 tokens per data-parallel unit, the graph shows about 60 thousand tokens per second per GPU at roughly 70 milliseconds. At 65,536 tokens, it shows about 90 thousand tokens at roughly 730 milliseconds. Throughput rises, but so does the time for one input-processing step, making throughput alone insufficient for choosing operational settings.

The decode graph, which covers output generation, was measured on 64 GPUs and cannot be directly compared with the results above. The repository also points to scripts for reproducing the benchmarks.

5. The boundary between runtime and service infrastructure

The public code makes it possible to examine execution flow and optimizations. However, it omits a KV cache system for reusing prior computations, a setup separating input processing from output generation, and load balancing across model replicas. Operational infrastructure must therefore be supplied separately. The repository says most engine features needed for that separation are implemented, so existing runtime features must be distinguished from additional integration work.

DeepInfer's adoption value should be judged by how well its released inference features fit each service's runtime requirements and implementation constraints.