The AIBrix team introduced ModelClaim in v0.8.0, an experimental feature that manages models separately from their execution environments. It runs per-model engines within a long-lived GPU pool and puts unused engines to sleep so multiple models can share GPU memory.
The introductory post is dated October 12, while the GitHub API records October 13 as the publication date of the v0.8.0 release.
The project addresses a problem with rarely requested models: they continue to occupy dedicated GPU capacity. Conversely, removing an idle model's execution resources entirely requires a cold start to prepare it again for the next request.
ModelClaim keeps Pods, Kubernetes' execution units, running for a long time while managing the model engines inside them separately. Instead of retaining an entire execution environment for each model, it adjusts engine lifecycles and memory use.
Reserved space for sleeping engines
The controller places models and sets KV cache limits. kvcached manages the KV cache, which reuses information from earlier tokens, and flexibly adjusts the space available across engines.
By default, a sleeping model retains its full reservation to secure space for waking up. Enabling noWakeReserveWhileAsleep allows other models to use that reserved space too.
Giving up reserved space changes the wake-up process. The system may need to put idle neighboring models to sleep first, load a new engine in another Pod, or wait for space to become available.
The gateway does not hold requests until this process finishes. For models that are asleep, starting, or waking up, it returns HTTP 503 with Retry-After: 10.
Retry-After is 20 when waiting for space and 30 when moving to another Pod. These values are retry guidance in seconds, not guarantees that the model will be ready or the request completed after that interval.
39 resident models versus concurrent serving
The project reports keeping 39 models, ranging from 1.5 billion to 32 billion parameters, resident on a node with eight NVIDIA H20 GPUs. Among them, sleeping 8-billion-parameter models retained 1.7–3.2 GiB of GPU memory.
The comparison is with approximately 15 GiB needed just to load the weights of such a model. These figures therefore show the memory cost of keeping sleeping models resident, not a result demonstrating more models running concurrently at the same throughput.
The same report described approximately five resident models per GPU, with about three awake at once. It also presented a case of service resuming roughly four seconds after a request, conditional on spare GPU memory being available.
Test coverage
The project's residency test report did not provide the full request mix, test duration, service targets, or a comparison of failure rates. Resident model counts alone are insufficient to judge serving performance under production load.
The current guide identifies its tested configurations as vLLM on H20, with tensor parallelism splitting a model across one or two GPUs within a single Pod. Describing a node with eight GPUs does not mean that running one model across eight GPUs was also validated.
Tensor parallelism above 2, more than two GPUs per Pod, and other GPU types are untested. Pipeline parallelism, which splits model layers, is also untested; data parallelism, which distributes requests across replicas, is unsupported. One ModelClaim runs one engine.
Memory accounting responsibilities
According to the current guide, GPU placement trusts the operator-declared perGPU.maximumFootprint and perGPU.kvFloor. The controller does not measure the model's actual memory footprint, and kvcached limits only the KV cache. Understating the footprint can exhaust GPU memory.
Host RAM is a separate concern. Level 1 sleep moves weights from GPU memory to host RAM, but the controller does not factor this usage into placement or sleep decisions. As sleeping models accumulate, the node can therefore run out of RAM regardless of whether their GPU reservations are shared with other models.
Runtime and access controls
The publicly available runtime described in the current guide is based on vLLM and is available only for linux/amd64. It is distributed as an artifact from main, not included in AIBrix releases. SGLang requires a user-supplied image and is untested.
The guide states that the runtime agent's port 8080 has no authentication and that engine ports 20000–20999 are used for sleep and wake controls. It recommends restricting network access to these control paths.
ModelClaim's benefit is keeping more rarely requested models available. Retaining reservations protects room for waking up; sharing them increases GPU capacity available to other models but may require reclaiming space when a model wakes.
The roughly four-second figure reported by the project is a conditional example of engine recovery, not the time it takes a client to receive a response. A client following Retry-After: 10 may send its next request only after 10 seconds, even if the engine is ready sooner.
Separate records for the first request and readiness
In a local deployment, the first check can be whether noWakeReserveWhileAsleep is configured to retain sleeping models' reserved space. The first request's timestamp, HTTP status, and Retry-After value can then be recorded together.
Next, observing the ModelClaim's readiness state provides a timestamp for confirmed readiness, while the client's retry time and result can be recorded separately. These records distinguish the request's status, the server's retry guidance, actual readiness, and the client's waiting time.
Beyond resident model counts, the points to watch are wake-up delays caused by reusing reservations and host RAM consumption during level 1 sleep. For collections of rarely requested models, these two resource constraints and client retry handling are likely to determine practical usefulness.
ModelClaim keeps more models available, but that benefit is separate from any guarantee of immediately serving a sleeping model's first request.