Local model runner TensorSharp added support for the latest DeepSeek model and made data exchange between GPUs GPU-centric. Developers serving local models across multiple GPUs can choose to have GPUs jointly compute selected expert operations rather than split whole layers. These are the computations selected for each input.
At a glance
The version released on October 23, 2026 supports unsloth's DeepSeek-V4.1-Flash-1017 in GGUF, a model weight file format.
The default uses 1 GPU, while tensor parallelism (TP) uses 2–8 GPUs.
YOCO is an approximation that skips some input processing and recomputes the tail, and this version enables it by default.
Performance: tensor parallelism moves onto the GPU
Previously, TP used the CPU to synchronize GPU work for each layer. In the default ggml_cuda path, GPUs now handle execution and result gathering, eliminating CPU waits within layers. Transfers go through pinned host memory, so this also works on PCIe hosts without direct GPU-to-GPU communication (P2P).
Prefill processes input, while decode generates the next token. With --tp 5, moving execution onto the GPU improved exact prefill by about 1.40× and decode by about 2.39×. This is separate from the comparison between placement methods.
Table: TP throughput before and after moving execution onto the GPU
Conditions: TensorSharp's own measurements, 5× A40, unsloth UD-IQ2_XXS, --tp 5. Exact prefill uses 2,048 random tokens.
Unit: tok/s
| Measurement | Before | After |
|---|---|---|
| Exact prefill | 402.3 | 562.9 |
| Decode | 15.2 | 36.4 |
Table: TP versus layer splitting at a representative input length
Conditions: TensorSharp's own measurements, 5× A40, UD-IQ2_XXS, ggml_cuda, random tokens, best of 3 runs. Input length and context depth: 2,048 tokens. Default prefill uses YOCO approximation; exact prefill uses TS_DSV41_YOCO_KEEP=0.
Unit: tok/s
| Measurement | --tp 5 | --layer-split 5 |
|---|---|---|
| Default prefill | 1,046.7 | 844.3 |
| Exact prefill | 562.9 | 463.5 |
| Synthetic decode | 36.4 | 34.5 |
TP's major gains were concentrated in input processing. In the document's decode profile, dense projections, which transform input vectors into other representations, accounted for 31% of GPU kernel time. Expert operations split by TP accounted for 21% across all 5 GPUs. Attention, shared experts, and the output head, which scores next-token candidates, remain on one GPU per layer. TP does not support attention partitioning or multinode TP.
YOCO's limited recomputation becomes the default
DeepInfer's YOCO uses a compressed cache shared from layer 20 onward. In these layers, it recomputes only the last 140 positions of each prefill call. It does not reduce image calls, speculative verification, or 1-token microbatches.
The same source measures quality by feeding ground-truth tokens in sequence. TensorSharp's own wikitext-2 evaluation uses 10 segments with 4,608-token windows. It prefills the first 2,304 tokens and scores the rest one token at a time. With layer splitting, perplexity, a measure of prediction difficulty, rose 0.6% from 2.4820 with exact prefill to 2.4970 with YOCO.
Mean KL divergence, the output distribution difference relative to llama.cpp, rose from 0.0160 to 0.0172. For the first 128 tokens, it was 0.015 with exact prefill and 0.026 with YOCO, but after 512 tokens both were 0.016. The effect immediately after the prompt stood out more than the average difference. YOCO quality should be judged against exact prefill.
Storage and memory requirements
Table: Estimated RAM and VRAM requirements by quantization level
Conditions: Estimates from the hardware requirements table in unsloth's official guide. They cover RAM and VRAM or unified memory, with Engram, the model's separate lookup table for retrieving information, stored on SSD. The estimates allow about 5–10 GB of headroom over non-Engram weights; runtime and context may require additional memory.
Unit: GB
| Quantization | Estimated requirement |
|---|---|
| 1-bit | 128–145 |
| 2-bit | 176–205 |
| 3-bit | 256–290 |
| 4-bit | 305 |
| 8-bit | 315 |
TensorSharp's 5× A40 measurement setup above has different memory conditions from this SSD-placement estimate table. It maps Engram into host memory, so the host or container memory limit must accommodate the table.
Cache improvements and configuration
A KV mirror that replicates the key-value cache on each GPU was also added. In TensorSharp's own layer-splitting measurements, decode with 5× A40 and a 2K context rose from 32.6 to 34.5 tok/s. Logits, the model's output scores before conversion to probabilities, were bit-identical.
DSpark is speculative decoding that first generates predicted tokens, then verifies them in batches with the target model. It is experimental in V4.1.
YOCO is enabled by default for V4.1's ggml_cuda and ggml_cpu backends.
Download the release assets and unsloth weights, and check the version and SHA-256 in the documentation. To select TP, replace --layer-split 5 in the server example with --tp 5. The environment variable for switching to exact prefill is:
TS_DSV41_YOCO_KEEP=0In these measurements, the main reason to choose TP when serving 1017 across multiple GPUs is input throughput rather than decode throughput.