devotd.
Recent developments, checked against their sources.
Support for a new model's text, image, and video inputs has been added. The cache manager's Python implementation is gone, so existing settings and event collection paths need checking before an upgrade.
Transformers adds support for Meta's multimodal model and native loading for InclusionAI's model. Input handling and loading are unified. Deployments still need separate checks for large-model memory needs and expert-parallel inference slowdowns.
NVIDIA released weights for training and evaluation, plus memory-saving deployment weights, for a model handling long documents, images, and video. Built-in prediction can accelerate generation, but actual speed and quantization's quality impact must be assessed by deployment conditions and task.
A small language model stores weight signs in 1 bit, with a 292.1MB distribution file that also runs on CPUs. Its average across ten company-evaluated tasks fell by 8.28 points. Runtime memory and generation speed were not measured.
Support for distributing model expert computations across GPUs has expanded on Rubin and Blackwell workstations. The workstation path leaves input delivery and output summation to the caller. Published single-layer performance figures do not represent speedups for an entire distributed service.
AIBrix v0.8.0's experimental ModelClaim separates model lifecycles from Pods and puts idle engines to sleep to share GPU memory. However, the first request to a sleeping model receives HTTP 503 rather than waiting for a result. If its reserved space is also shared while it sleeps, waking up depends on available capacity.
vLLM TPU Inference v0.31.0 adds optional dispatch that reduces inter-chip MoE communication and W4A16 linear operations on the JAX path. Development PRs reported a 4.5% reduction in total decoding step time for one MoE configuration and fit Gemma 4 31B QAT on one v6e chip. These results address different bottlenecks; they do not establish general speedups or preserved quality.
The Ling-3.1-flash model card advertises a 1M-token context, while SGLang's deployment guide specifies 262,144 tokens. Active-parameter figures also differ: 25B versus about 17B. Self-hosting plans need the actual checkpoint and server configuration, not just the model description.
Mistral Large 4 opened its public API preview on October 6, while model weights were promised for month-end. Hugging Face still marked them as upcoming on October 14. Evaluating the preview service and planning future self-hosting require different evidence.
vLLM 0.31.0 keeps processed model weights in a GPU daemon so a restarted engine can avoid loading them again. But its checkpoint fingerprint checks safetensors headers, not weight values; different fine-tuned models can therefore match the same cache entry. One user measured roughly 32 seconds per restart rereading 66 headers on remote storage.