TensorSharp Muse Spark 816B: 2.07× 4K prefill in internal tests, 1.40× tensor-parallel decode

At a glance

Layer splitting assigns model layers to different GPUs, while tensor parallelism splits computation within a layer across GPUs.

Table: Test environment and target

Conditions: Internal tests used the same model, machine, and requests. Results are medians of 3 runs, except llama.cpp tensor splitting with 2 runs. GPU traffic goes through host memory because direct GPU memory access is unavailable. llama.cpp was built without NCCL.

Performance: Layer splitting for prompts, tensor parallelism for generation

The report was published on October 22. Prefill processes the input first, while decode generates a response one token at a time.

Table: Performance comparison by parallel mode

Conditions: Matching modes compared in the environment above. Multipliers are TensorSharp÷llama.cpp; 1.0× means equal performance.

Units: Throughput in tok/s; ratios in multiples.

The bar chart highlights each mode's strengths. The report says layer splitting overlaps GPU processing of input batches. Single-request decode was similar in both engines because tokens pass through all 10 GPUs in sequence.

For tensor parallelism, TensorSharp uses shared memory in the GPU communication library NCCL to sum results; llama.cpp copies and sums without NCCL. The report puts TensorSharp's summation bandwidth at about 1.8 GB/s. Muse Spark normalizes expert outputs before combining them, transferring 8 times the summed result's size, so tensor-parallel prefill is slower than layer splitting.

Correctness: wikitext-2 evaluation with fixed reference tokens and internal checks

Next-token probability distributions were compared using fixed reference tokens. The metrics were KL divergence, which measures distribution differences, and top-token agreement, the share of positions with the same highest-probability token.

Table: wikitext-2 next-token distribution comparison

Conditions: TensorSharp internal tests evaluated 12,276 wikitext-2 positions with identical token IDs. The baseline was llama.cpp with 512-token microbatches, small input batches computed at once.

TensorSharp layer splitting showed smaller distribution differences than those between two llama.cpp runs. Tensor parallelism had 9% higher mean KL divergence and a higher maximum, but better top-token agreement. The report links this difference to the path that sums per-expert partial results at BF16 precision during prefill.

In 14 internal integration checks, each run once, tensor parallelism passed all checks, layer splitting passed 13, and both llama.cpp modes passed 12. These are not general model quality tests. Both llama.cpp modes failed cache reuse. TensorSharp layer splitting and both llama.cpp modes failed the staggered-arrival request check. The report says these runs ended before 512 tokens when selecting the highest-probability token, rather than due to engine errors.

What changed

Handling interruptions caused by GPU connection limits and raising the executor limit from 8 to 16 GPUs resolved the issue that prevented both modes from running on this machine.

The report says adjusting input batch size reduced a 16K call from 15.3 seconds to 10.2 seconds. Generation moved from a managed path delivering 1.9–2.4 tokens per second to native tensor parallelism.

How to try it

The code is on main. Reproduction settings are UD-IQ3_XXS, F16 KV cache, and flash attention enabled. The comparison uses llama.cpp 7170cad.

MAX_CONTEXT=32768 TS_MUSE_SPARK_PLAN_SLOTS=5 TS_CODE_EXEC=0 \
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model <shard 1> \
  --backend ggml_cuda --layer-split 10 --no-webui --no-skills --no-multi-agent

Switch to --tp 10 for tensor parallelism. The llama.cpp equivalents are -sm layer and -sm tensor, respectively. The reproduction script is eng/muse-spark-bench.py.

Limitations

Image input, speculative decoding, expert component offloading to the CPU, contexts beyond 16K tokens, and other GPU configurations were outside the scope of measurement.

Layer splitting is worth trying first for large-prompt workloads, and tensor parallelism for interactive responses and concurrent requests.