NVIDIA has expanded support for sharing cached request prefixes in inference services running across multiple hosts. Service developers and operators can restore prefixes instead of recomputing them. They also receive improvements to speculative decoding, which proposes candidate outputs in advance.
1. A shared cache with all hosts participating
Hyperflare Inference v0.1.0, released on November 11, adds cache sharing for tensor-parallel configurations that split model computation across hosts. PR #140 extends the existing Selene cache pool to this configuration. The pool serves as shared storage for request prefix caches across instances and hosts.
A lead rank proposes the connection, and all ranks must approve it. If any rank rejects memory registration, all ranks use their own local storage tiers. Operators must therefore apply the same pool settings to every rank or omit them from all ranks.
2. Shorter restoration times in two-host validation
According to the company, cache restoration took about 0.1~0.3 seconds with TP2 across two hosts, versus 40~60 seconds for a cold start. The setup used Soft-RoCE on SM120, a software implementation of RoCE for remote memory communication over Ethernet. Pool connections were tested with plain TCP and encrypted TLS.
Table: Cache restoration validation results for a two-host configuration
Conditions: Two hosts, TP2, over Soft-RoCE on SM120, with plain TCP and TLS pool links. Restored continuations were compared with GPU-resident ones by top-20 logprobs.
| Item | Restoration time (seconds, approximate) | Cold-start time (seconds) | TV difference upper bound |
|---|---|---|---|
| Cache eviction and restoration | 0.1–0.3 | 40–60 | 8e-4 |
Continuations generated after restoration were compared with those using GPU-resident caches through the log probabilities of the top 20 tokens. TV in the table measures differences between probability distributions. The same validation showed short restoration times and small distribution differences, demonstrating the practicality of cache reuse.
Both ranks reconnected after the pool manager restarted. A test that terminated one rank, restarted it, and restored its cache also passed. This verified reconnection and restoration after interruptions as well as speed.
3. Speculative decoding updates candidate preparation and delivery
For speculative decoding, the NVFP4 draft head and the Markov head, which determines the next state from the previous one, are prepared as independent options. Changes also propagate candidate proposals in blocks and fuse BF16 Markov sampling operations. These updates address both candidate preparation and delivery during parallel execution.
DeepSeek speculative sampling now keeps per-vocabulary probabilities on each rank. The release also integrates DSpark draft scoring and prepares an optional MXFP8 draft vocabulary head.
The speculative decoding changes are confirmed by change titles in the supplied release notes, but neither source provides measurements of their effects. Cache validation also used software-based connections. Pool configurations with physical RoCE network cards and aarch64 were not tested.
4. Less recomputation requires attention to builds, storage, and security
Using RoCE communication on Linux x86_64 requires explicitly enabling the feature at build time and providing libibverbs in both build and runtime environments. The source gives the following build command.
cargo build --release --locked --manifest-path flare/Cargo.toml --features roceThe default build rejects multi-host serving before bringing up GPU resources, so deployment first requires a build with communication support. Multi-host local caches persist on disk. Restoring after clearing local disks and restarting requires records to have already reached the pool manager's storage.
Shared caching reduces prefix recomputation, but requires operators to manage communication-enabled builds and disk persistence. Pool and record connections send session secrets and transfer tokens in plaintext, so the cluster network also needs protection.
This release extends cache reuse across multiple hosts while refining candidate preparation and parallel processing paths for speculative decoding.