ollama/ollama v0.40.17-rc0: Integrated graphics loading, MLX concurrent generation

1. Addressing loading failures and generation waits separately

Released on November 10, 2026, v0.40.17-rc0 addresses bottlenecks in model loading and request processing. The changes target loading issues on shared-memory graphics and wait times when multiple requests overlap.

The integrated graphics change reduces duplicate model data use within the same memory budget, the memory limit available to the device. The MLX change lets requests share compute resources instead of waiting for the previous request to finish.

2. Vulkan integrated graphics use memory-mapped loading

The previous loading method stores CPU-resident model data in a Vulkan host buffer. This storage draws from the same device memory budget as the weights, the numerical information learned by the model. Ollama's pull request says this can roughly double memory use on integrated graphics and cause loading failures. Reported symptoms included pinned memory allocation failures and errors on Windows AMD systems.

Ollama now automatically selects mmap for memory-mapped loading when it detects Vulkan integrated graphics. Direct I/O, which bypasses the operating system cache, could not avoid the host buffer, so the loading path itself was changed. This applies to integrated graphics that share memory with the CPU.

Below is an excerpt from the modified function.

// appendLoadModeArgs selects llama-server's single model loading mode. Integrated
// GPUs share memory with the CPU: Linux CUDA/ROCm use direct I/O to skip the page
// cache, while Vulkan uses mmap because non-mmap loads place CPU-resident tensors
// in a host buffer drawn from the same device budget as the weights.
func appendLoadModeArgs(params []string, opts api.Options, gpus []ml.DeviceInfo) []string {
	for _, g := range gpus {
		if !g.Integrated {
			continue
		}
		if strings.EqualFold(g.Library, "Vulkan") {

3. MLX handles multiple generation requests together

MLX models can use OLLAMA_NUM_PARALLEL to set the number of concurrent generation requests. The default is one request, and raising it requires more memory for each active request. Setting it to 2 can let a queued request start receiving tokens sooner when two requests arrive together.

Requests batch work that can be computed together while keeping separate caches, temporary storage for reusable data, and response streams. Instead of processing all input for one request before the next, prompts are split into 2048-token chunks and interleaved with response generation. Prompts are the instructions and context supplied to the model. An example setting follows.

OLLAMA_NUM_PARALLEL=2

The company's benchmarks ran NVFP4-quantized models on an Apple M5 Max, with results reported as median seconds. Quantization reduces model numerical precision to save memory. Two clients submitted requests simultaneously; the baseline processed them sequentially, while the modified version interleaved them. The table shows when the queued request began receiving a response and when both requests finished.

Table: Response start and total completion times for concurrent requests

Conditions: NVFP4, p4096/g256, Apple M5 Max; OLLAMA_NUM_PARALLEL=2; two clients submitting simultaneously; baseline uses sequential processing on main, modified version uses interleaving

Unit: seconds (median)

Under these conditions, the queued request started receiving a response sooner for all three models. Total completion time increased for Gemma and Qwen and was similar for Nemotron. The pull request says single-request performance remained close to the baseline.

4. Trading response wait time for processing costs

Concurrent processing responds sooner to queued requests by sharing the graphics compute resources used by the first request. The technique that speeds generation by guessing upcoming content also pauses while multiple requests are running. Operators gain shorter waits at the cost of more memory per request and longer overall completion times for some models.

Separate per-request caches retain memory even while requests are idle and store duplicate shared input prefixes. The developers plan follow-up improvements to share cache budgets, reclaim idle memory, and reuse common input.

This is a preview release, and the measurements apply to the models, settings, and hardware listed in the table.

v0.40.17-rc0 reduces loading pressure in shared-memory environments and lets MLX respond sooner to queued requests, while concurrent generation requires weighing memory use against overall completion time.