In an October 10 r/LocalLLaMA post, Basalt’s developer reported Qwen3.8-Flash-Next decode speeds of 665 tok/s for structured output and 354 tok/s for prose.

These are the developer’s own measurements. This article could not reopen the post body and carries over only the two figures in its title.

Within one engine, changing only the kind of work nearly halves throughput. That is why a single number is hard to read as a promise for interactive work in general.

A deployment boundary tied to Blackwell sm_120 and CUDA 13

According to its README, Basalt is a local inference engine forked from Strata and narrowed to Qwen3.8-Flash-Next. It requires Linux x86-64 with AVX2, an NVIDIA Blackwell GPU with compute capability 12.0, and CUDA 13.

Experts (the MoE sub-networks) that do not fit in VRAM live in system RAM, optionally with a second GPU as a peer expert tier.

The narrow target allows focused implementation. Other GPU generations or operating systems are not covered merely because they run other CUDA inference software.

Engine settings that the model file does not carry

The README calls the engine’s built-in defaults untuned. They are a 4096 context, fp16 KV, no speculative decoding and no expert cache, and the model file carries no engine flags.

Settings go into one basalt_config.json passed with --config, with precedence of command line, then config file, then built-in defaults.

Starting a downloaded model pack on defaults therefore does not recreate the conditions behind a tuned result.

--parallel slots and KV cache allocation

--parallel N serves 1 to 8 requests at once in the engine’s batch slots, and --max-context is shared among them. With --parallel 4 --max-context 262144, each slot gets 65536 tokens.

--slot-full-context gives every slot the whole --max-context, at N times the KV cache. Slot KV cache is taken from the expert cache.

Raising concurrency therefore trades away context per request or memory left for experts.

The test scope of the precision document

The precision document’s test setup is one RTX 5090 at a 400 W power limit plus an RTX 5060 Ti as the peer expert tier. The 400 W figure is not the combined draw of both GPUs.

With expert tiers pinned, all 275 greedy prompts produced identical tokens with and without MTP. Prompts were 19 to 395 tokens with 256 new tokens, and the document itself says this tests the decode path, not long context.

It also notes that experts moving between CPU and GPU can change rounding. Quality is compared against a llama.cpp Q8_0 reference, with the 100-chunk results rather than the 20-chunk ones as the basis for comparison.

These are the project’s own tests and do not guarantee the same outcome on every coding or reasoning task.

Quality acceptance and memory layout as separate tests

These conditions split into two acceptance questions. Does the chosen configuration give acceptable answers for the intended tasks and prompt lengths, and does its memory layout meet the required latency at the expected number of users?

A short deterministic parity test helps track an implementation change but does not replace a workload check. A higher aggregate throughput across requests also does not mean a single request finishes sooner.

Running with the effective configuration and slot memory in view

A trial on a Blackwell workstation keeps the model-pack revision, engine build and config file together and measures prose, structured output and concurrent serving separately. The README documents the launch and metrics check in this form.

bin/basalt serve --config basalt_config.json --parallel 4 --max-context 262144
curl http://127.0.0.1:8080/metrics

Per the README, the server prints one [server] kv cache: line per device at startup and warns when slots eat too far into the expert cache. /metrics reports total, active and queue_depth under live.slots.

For workloads that depend on long histories, evaluating those histories directly is more realistic than extrapolating from short-prompt parity. Basalt was not run for this article.

Basalt’s throughput figures read most accurately as the product of a model pack, a config file and a slot memory layout, not as raw engine speed.