devotd.
Recent developments, checked against their sources.
Strata runs MoE models that do not fit in GPU memory by splitting them across GPU, RAM, CPU and SSD. Where each weight lives and where it is computed shapes prompt processing speed, first-response wait and generation speed in different ways.
Lemonade stable v2026.40.0 sizes streaming models on AMD integrated GPUs against the GTT pool. The same release dropped openmoss:rocm and sd-cpp:rocm on Windows, so upgrade problems need to be separated into the two.
llama.cpp and Ollama return typed decisions instead of generated text through /v1/systemone. Parsing gets simpler, but probabilities do not guarantee correct answers, and confidence cutoffs differ by model.
Bas Nijholt runs an LLM on two RTX 3090s but calculates that electricity and hardware can make an API cheaper. The outcome changes with utilization and with which API model is compared.
LocalAI 4.11.0 puts ordered local and remote models behind one name. It can switch targets on selected failures, but only before the response starts.