devotd.
Recent developments, checked against their sources.
Basalt forks Strata into an engine for one target, Qwen3.8-Flash-Next on Linux and NVIDIA Blackwell (sm_120). The developer’s reported 665 and 354 tok/s come from a particular model pack, workload and tuned configuration; built-in defaults and context splitting under concurrency shape the result just as much.
strata-mlx is an independent MLX engine that reads Strata's model files. A user's measurements on a real 32GB MacBook Air showed that cutting the expert budget from 20.5GB to 16GB nearly doubled generation speed and sharply reduced cumulative swap-out.
audio.cpp v0.9.1 replaces the PocketTTS CPU Mimi upsampler’s 32 MiB dense diagonal weights with a 64 KiB depthwise module. In the maintainer’s measurements, four-thread decoder time per frame fell from 30.50 to 10.11 ms, but GPU paths are unchanged and some release binaries may require AVX-512.
On October 8 MSI opened Hong Kong preorders for the Prestige N16 Flip AI+, an RTX Spark-based 2-in-1. Up to 128GB is shared by the CPU and GPU, and for local AI the question is whether a real software stack on Windows on Arm runs within that space.
WHIRL v0.1.4 narrows the CORS default of its native Windows LLM server to web pages served from loopback addresses. Inference code and speed are the same as v0.1.3, and a streaming client that stops reading no longer blocks other users.
Strata runs MoE models that do not fit in GPU memory by splitting them across GPU, RAM, CPU and SSD. Where each weight lives and where it is computed shapes prompt processing speed, first-response wait and generation speed in different ways.
Lemonade stable v2026.40.0 sizes streaming models on AMD integrated GPUs against the GTT pool. The same release dropped openmoss:rocm and sd-cpp:rocm on Windows, so upgrade problems need to be separated into the two.
llama.cpp and Ollama return typed decisions instead of generated text through /v1/systemone. Parsing gets simpler, but probabilities do not guarantee correct answers, and confidence cutoffs differ by model.
Bas Nijholt runs an LLM on two RTX 3090s but calculates that electricity and hardware can make an API cheaper. The outcome changes with utilization and with which API model is compared.
LocalAI 4.11.0 puts ordered local and remote models behind one name. It can switch targets on selected failures, but only before the response starts.