Developers running AI models of different sizes on one server can now select request targets automatically based on remaining resources. The new llama-swap version routes requests to a predefined candidate that is less demanding to run now. This change suits setups that can use smaller models or variants with shorter contexts, the input range a model can handle at once.
1. Candidates that are easier to run now take priority over larger models
Added in v273, fit is a selector that chooses which candidate model receives a request. It asks the capacity router, which manages model execution and swapping within resource limits, for each candidate's swap plan. It first considers what must be removed to make room.
Selection is based on the sum of evict_cost values assigned to models that would be unloaded. This value represents the cost assigned to unloading a model from memory; candidates with lower totals come first. Ties favor fewer unloaded models, then ready, starting, and stopped states, in that order. If still tied, candidate list order decides.
If a ready small model and a large model that needs starting can both run without unloading others, the small model takes priority. Listing a large model first therefore does not mean it runs first. Current execution state and the burden of unloading other models take precedence over model size.
2. Configure the capacity router and candidate list
This strategy must be used with the capacity router. Candidates are limited to local models. Setting each model's resource usage in capacities lets the router assess feasibility and unloading plans. The documentation linked from the release provides this example.
selectors:
any-llama:
strategy: fit
targets: [llama-70b, llama-8b-q8, llama-8b-q4] # order = preference
settings:
no_evict: false # true: 429 instead of choosing a target that unloads a model
name: "Any Llama"In this example, the candidate list groups multiple models under one request name. List order applies only when all preceding criteria tie, so operators configure actual resource usage for each candidate alongside preference order. Even requests using the same name trigger a fresh resource check to determine which model handles them.
The default evict_cost is 0. Without explicit costs, the number of models unloaded, readiness, and list order determine selection. To prioritize keeping a particular model in memory, its unloading cost can be reflected in the configuration.
3. Selection can also avoid unloading models
Enabling no_evict skips candidates that would unload running models. If no candidate can run without unloading, the selector rejects the request with HTTP 429 without forwarding it to the router. Even models with an unloading cost of 0 cause exclusion if they would actually be unloaded, so unloading is checked separately from cost.
Omitting a candidate's capacities treats it as consuming no resources, making it always feasible. Also, no_evict applies at candidate selection time. If another request starts a model swap after that check, the router may still unload models.
4. Per-request selection work shifts to candidate management
The benefit is that developers can group alternatives behind one name instead of manually changing request targets whenever resource availability changes. Smaller variants or shorter contexts can be included if they can handle the requests.
In return, operators must manage resource usage for each variant and express which models should stay in memory through cost values. Choosing to avoid unloading preserves existing models but may cause new requests to be rejected. Adoption hinges on balancing the flexibility of using alternatives against the need to retain currently running models.
Rather than choosing the largest model first, v273 broadens selection criteria to route requests according to current resources and the burden of swapping models.
