Strata is an inference engine that runs models larger than GPU memory by splitting them across GPU, RAM, CPU and SSD. By default it keeps all 24,576 experts in RAM and caches the most-used experts in GPU memory.

This article examines that layout from v0.1.39, released October 4, and the docs at a pinned commit. Performance figures are reports from the project and public testers, not results devotd reproduced on hardware.

Not every part of a large model is busy at every moment

Qwen3.8-Flash-Next, the model Strata targets, is a mixture-of-experts (MoE) model in which a router picks the experts each input needs. The official model card lists 125B parameters with 6B activated, plus a separate 51B n-gram embedding and 4B of MTP parameters.

Experts unused now may be chosen for the next token or in another layer. So all the weights must stay reachable while the part needed now is used quickly.

Choosing which weights to keep close

Frequently chosen experts are worth keeping in fast memory. Putting even rare experts on the GPU raises VRAM needs, while keeping everything far away lets data movement eat the benefit of small compute.

Strata's model guide covers only Qwen3.8-Flash-Next and its quants and derivatives. A shared file format alone gives no basis for expecting the same layout effect on other models.

What each memory tier holds

VRAM holds the attention and DeltaNet mixers, routers, shared experts, output head, MTP draft layer and KV cache. From 64K, only the most-read part of the KV cache stays in VRAM and the rest streams from RAM.

The remaining VRAM is filled by an expert cache of the most-used experts, which adapts to the conversation. So total GPU memory is not the expert cache capacity.

RAM holds all 24,576 experts, pinned. The SSD holds a 28.8GB n-gram table, of which only a few rows per token are read, bypassing the OS cache.

Capacity, bandwidth and latency

A layout can be evaluated by separating capacity (how much fits), bandwidth (how much moves per unit of time) and latency (how long a request waits for data).

More experts in VRAM means less movement, but the KV cache and compute buffers use the same VRAM. Choosing a longer context changes the room left for experts.

Where weights are stored is not where they are computed

Weights in RAM are not necessarily computed by the CPU. The GPU handles experts in its cache, and the CPU computes the experts the GPU lacks in place, in RAM.

The CPU work runs while the GPU processes the cached experts. The docs also treat separately the share of missing experts copied to the GPU instead of computed on the CPU.

Work that has to finish together

If the CPU's share finishes late, a step cannot complete however fast the GPU is. If data movement is the main wait, adding compute alone may do little.

The adaptive cache changes which experts sit on the GPU according to usage. It does not change the experts the model picks; it adjusts where the same computation runs.

The SSD n-gram table is read in small pieces

The model card describes the n-gram embedding as a component indexed by short n-grams and says it is well suited to offloading on memory-constrained accelerators.

Strata's IQ4_NL table is 320,001,536 rows of 90 bytes (26.8GiB). A token needs 16 rows on 16 different pages, each fetched with an unbuffered 4KiB read.

Rows already fetched stay in a bounded row cache. For this kind of lookup, whether small reads finish on time matters more than the SSD's sequential read speed.

Two variants of the low-RAM mode

By default all experts (23–50GB) are copied into RAM, but when RAM is short setup chooses a low-RAM mode that maps them from a file in the model folder. The docs say that with a small GPU most experts then come from the SSD and it is much slower.

Engine 0.1.30's resident variant applies when the experts the GPU lacks fit in RAM, keeping the RAM copy to exactly those experts. Within the same low-RAM mode, speed depends on which tier the needed weights end up in.

The scale of feeding experts from the SSD

The experimental UD-Q4_K_XL doc was validated on one PC with Windows 10, an RTX 5070 12GB, 64GB of RAM and an AVX-512 CPU. With a 40GiB RAM budget, six runs of the same short greedy prompt gave 6.7–8.6 tok/s, a mean of 7.9.

It read 1.1GB from the SSD per verify round, about 0.33GB per token. These are the developer's figures from one specific PC.

Multiplying 0.33GB per token by 8 tok/s gives about 2.64GB/s of expert reads. This is a conversion of the doc's figures, not a separately measured SSD throughput.

The SSD can ease the capacity limit, but the cost of that data movement remains.

Reading the prompt versus writing the answer

Prefill processes a long input in token chunks, so expert weights prepared once serve many tokens. Strata's prefill path streams experts the GPU lacks to it in a fixed order through a ring buffer.

A source comment records that, with 8,192-token chunks, growing the ring from 96 to 384 slots went from 1,153 to 1,294 tok/s. A bigger ring takes expert cache slots, so chunk size and cache space need balancing.

Decode picks the next token from the tokens already produced, so it is hard to batch much work at once as prefill does.

That is why reading a long document to give a short answer and giving a long answer to a short question may favor different setups.

MTP, raising generation speed with draft tokens

Strata uses the model's MTP layer to draft up to 3 tokens and checks them in one pass over all 48 layers. The docs say this yields 2.4–3.2 tokens per pass on average.

How many are accepted depends on what is being generated. It is one reason tokens per second change with the prompt and the kind of answer on the same PC.

Caveats: the same model does not mean the same tokens

The docs explain that even at temperature 0, the same prompt can end in a different (equally good) answer when drafting, cache state or a resumed conversation differ. Byte-identical repeats need separate options.

Running the same model and producing the same tokens under every setting are claims of different strength.

Tokens per second versus the time actually waited

The engine 0.1.26 benchmark ran on an RTX 5070 12GB (PCIe 5.0 x16), a Ryzen 5 7600, 64GB of DDR5-5200 and Windows 10. It used setup's defaults with MTP and greedy decoding, generated 256 tokens with images off, and ran each condition once.

In the raw record, Q2_0's 32K input was 31,566 tokens, with prefill at 2,170.85 tok/s, generation at 81.75 tok/s and 16.02 seconds to the first token. The 81.75 tok/s is the speed once the answer starts; the 16.02 seconds waited before that is separate.

For a service that reads documents and answers briefly, the wait for the first response weighs heavily; for long drafts, generation speed does. For coding agents, completion time including tool runs and retries matters.

PP (prefill), TG (generation) and TTFT (time to first token) measure different stretches.

The engine in that record is 0.1.26. The v0.1.39 release notes say the README speed table was not re-measured for this release.

So these figures are one well-documented reference point, not a guarantee for the latest version.

Speeding up one request versus handling many

In GitHub issue #127 on September 29, rhgo1749 described an experimental setup with an independent Strata engine per GPU, assigning each request to one GPU lane. The large host expert arena is shared between lane processes via Linux MAP_SHARED.

On three RTX 5070 Ti 16GB cards with IQ3_XXS and 262K context per lane, warm generation speeds were 78.8, 78.4 and 80.1 tok/s. The author stated that the 237.3 tok/s total is a plain sum of lane values, not a synchronized aggregate throughput.

Because it is an experimental implementation in a separate fork, it cannot be read as stock Strata performance.

For personal chat, one request's wait matters; for a service running several agents, keeping concurrent requests from queuing up matters too.

Aggregate throughput hides individual users' waits, and single-request tests miss contention on shared resources.

A smaller pack is not the same as the right model

The model guide lists expert sizes of 34GB for Q2_0, 35.5GB for IQ2_XS, 43GB for IQ3_XXS, 50GB for IQ3_S and 23GB for the Coder. By default about 10GB more is needed for Windows and other programs.

The Coder is a derivative that keeps only 256 of the 512 experts, chosen on code data. The guide says it is weaker outside code and in languages other than English.

The install guide's main targets are Windows 10/11 or Linux, an x86-64 CPU with AVX2, and an NVIDIA RTX 20–50 series card with 12GB or more of VRAM, or a supported AMD card. Older NVIDIA cards such as Pascal and Volta were added as experimental paths in v0.1.39.

Disk needs are about 70–80GB for the model and about 6GB for the MTP layer, and Q2_0 on an AVX-512 CPU writes a further one-time copy of about 40GB.

The Strata engine is MIT-licensed, but the official Qwen model card shows the Qwen Community License 1.0. Model files carry their own licenses.

A memory layout that fits your own work

Strata is worth comparing where the supported model's quality is needed and VRAM is short but RAM and CPU have headroom. The first question is not whether the biggest pack runs but where the needed pack's weights stay while answering.

Separating first runs from warm runs and recording input length, concurrent requests, engine and model versions and hardware conditions makes it possible to trace why results change.

Strata's speed depends less on the size of the model it runs than on which tier the weights stay in and where they are computed.