DeepSeek-V4.1-Flash: Large-scale inference serving, P/D separation and Wide-EP

1. One rack delivers 7.9 times the previous aggregated throughput

According to the company, per-GPU throughput in the SemiAnalysis AgentX test was 954,000 tokens/s, 7.9 times the sum of previously published single-node results. This includes cached input, uncached input, and output tokens.

Per-user speed was 196 tokens/s at P90, the 90th percentile of the distribution. Using DeepSeek's off-peak pricing, the stated equivalent revenue was $38.96 per GPU-hour.

It also published a comparison with a model-specific engine. With 32 GB200 GPUs, a 131K context, nextn=3, and batch sizes of 8–256 requests, vLLM's generation throughput was within 6% of the published results for DeepSeek's DeepInfer engine.

2. Smaller groups for input, wider distribution for generation

Prefill, which processes input all at once, and decode, which generates the subsequent answer, use resources differently. P/D separation assigns them to separate GPU groups, allowing parallelism to be chosen for each stage.

DEP combines input data parallelism with distributed expert computation; the trailing number indicates the GPU count. The developers say smaller configurations favor prefill because work is easier to distribute evenly.

In AgentX's prefill-only test with 256 concurrent requests, DEP2 delivered 64.4K tokens/s per GPU. DEP4 fell to 53.3K. Smaller GPU groups were more efficient for input processing than larger ones.

3. Generation compute grew only modestly with long inputs

The model uses cross-layer sharing and compression to shrink the KV cache, which holds computational information from prior input. At 128K context with DEP4 and 64 requests per GPU rank, mixture-of-experts computation accounted for 62% of the main model's generation-stage time.

On GB300 with DEP32, the main model's execution time was measured with 16 requests per rank and 6 tokens per request. Increasing context from 128K to 855K raised execution time only from 8.93 milliseconds to 9.53 milliseconds.

Table: Main model execution time by context length

Conditions: DeepSeek-V4.1-Flash, GB300, DEP32, 16 requests per rank, 6 tokens per request, identical context lengths for all requests

Unit: milliseconds

With so little time variation by context length, generation requests were easier to distribute by request count rather than length. The developers measured model execution and actual serving time separately, using this command for model execution measurements.

vllm bench forward

4. Expert distribution gains depend on communication costs

Wide-EP distributes expert computation across more GPU ranks, reducing the expert weights each must read. Tests on GB200 ran from DEP4 to DEP32 with 128K context, 64 requests per GPU, and expert load rebalancing enabled. The main model's expert computation time fell from 13.4 milliseconds to 6.0 milliseconds.

These gains come with communication and work-distribution overhead. Other work stayed at about 10 milliseconds, while at DEP32, communication for the auxiliary model that proposes future tokens offset some parallelism gains.

The developers reduced load skew with EPLB, which monitors expert loads and changes their placement. They applied it separately to both the main and auxiliary models. An NVLink-based implementation changed expert placement across DEP32 in under 1 second without pausing decode.

5. More GPUs also require CPU optimization

Wide-EP synchronizes GPU ranks at every layer, so one slow task can delay everything. The developers changed how decode-side CPUs read input information synchronously while waiting for request results, reducing inter-CPU synchronization and new-request overhead. According to the company, improving KV transfer with NIXL push raised per-GPU throughput from 141K to 440K tokens/s on GB300 with 8P1D and 3,072 concurrent requests. The results show that surrounding processing must also accelerate to turn faster GPU computation into faster responses.

They also built a separate low-latency generation path for small user counts. In a real AgentX session with one user and model computation split across four B300 GPUs, it reached 826 tokens/s at P90. The standard path delivered 575 tokens/s under the same conditions.

vLLM's results highlight resource allocation tailored to input and generation, balanced expert loads, and communication and CPU cost management as keys to large-scale serving.