GLM-5.3: Distributed inference on two Stations, a single API

1. A model larger than one GPU even after quantization

GLM-5.3 NVFP4 has 753 billion total parameters, with 40 billion active per token. It computes with only a task-relevant subset, but all candidate weights must remain in memory. NVIDIA's blog puts the checkpoint at about 464GB, exceeding a GB300 GPU's 252GB of memory.

The model card, a document covering specifications and usage, describes 4-bit quantization of linear operations in its mixture-of-experts architecture, which selects some submodels for each input. NVIDIA says reducing these weights and activations from 8-bit to 4-bit cuts disk and GPU memory requirements by about 1.66x.

The setup uses tensor parallelism to split weights and computation across two GPUs rather than offloading weights to CPU memory. One machine receives API requests, but both process every request, so users connect to one OpenAI-compatible API.

2. Prepare model copies and a high-speed link on both machines

Each Station stores a full checkpoint, the files containing the trained model state, on disk, requiring at least 464GB of free space per machine for the model. The procedure downloads the model on one machine, copies it to the other, and compares file lists and sizes. When loading it onto the GPUs, vLLM splits the weights between them.

The machines use RDMA over a ConnectX-8 link to exchange computation results directly between GPU memories. NVIDIA advises checking the 400000Mb/s link speed, successful transmission of large packets, and time synchronization during preparation.

After startup, check API health and communication logs. Startup logs show about 218GiB of weights loaded onto each GPU, roughly half the full model.

3. Start the worker first, then send requests to one API

The documented vLLM 0.30.0 setup uses multi-token prediction (MTP) to draft four tokens per step and verify them with the full model to accelerate output. Prefix caching also reduces recomputation of previously processed prompt prefixes.

The FP8 KV cache stores context for ongoing requests, halving cache memory compared with 16-bit storage.

Start the worker node first, followed by the API-serving node. Once the logs show service startup is complete, verify responses with a health check and a request. Use the original article's request example after configuring containers and environment variables on both machines.

docker exec "$CONTAINER" curl -fsS --max-time 120 \
 "http://127.0.0.1:$PORT/v1/chat/completions" \
 -H 'Content-Type: application/json' \
 -d '{"model":"nvidia/GLM-5.3-NVFP4",
 "messages":[{"role":"user","content":"Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one sentence and calculate the percentage change from Q1 to Q2."}],
 "temperature":0,"max_tokens":4096,
 "chat_template_kwargs":{"reasoning_effort":"low"}}' |
python3 -c 'import json,sys; c=json.load(sys.stdin)["choices"][0];
print(c["message"]["content"]); print("finish_reason:",c["finish_reason"])'

The example prints both the response content and finish reason to check for normal completion. This request uses low reasoning effort, while the load tests below used the default maximum effort.

4. More concurrent requests trade response speed for throughput

NVIDIA measured load using AIPerf and 80 SPEED-Bench Coding conversations. At each level, warm-up requests numbered three times the concurrency and measured requests ten times it; warm-ups were excluded from statistics. Output tokens include reasoning before the answer.

The following company-reported results keep the concurrency cap fixed at 16.

Table: Throughput and latency by request count with a concurrency cap of 16

Conditions: Two DGX Stations, GB300 GPUs, GLM-5.3 NVFP4, tensor parallelism 2, vLLM concurrency cap 16, context 98304 tokens, SPEED-Bench Coding, prefix caching, four-token MTP, FP8 KV cache, reasoning max, output 8192 tokens

Units: Throughput in tok/s, latency in s

The company reported total throughput of 607.8tok/s with 16 concurrent requests in this setup. More concurrent requests raised total throughput but also increased average latency. This shows the trade-off between serving more users and the time each user waits.

A separate test raised the concurrency cap to 32 and sent 32 concurrent requests, keeping other conditions unchanged. The company reported total throughput of 385.0tok/s, average per-user throughput of 13.0tok/s, and average request latency of 453.54 seconds.

5. Local execution requires operating two machines and their network

Pairing two machines enables local execution of a model larger than one GPU's memory. NVIDIA suggests local inference for sensitive data, agent experiments, and repeated prompting without per-token fees. However, both GPUs and the network must remain available, and each Station must store a full model copy. Operations also cover RDMA connectivity, synchronization, and network access control.

By default, the documented API accepts requests without authentication on all network interfaces of the receiving machine. Before exposing it externally, configure a firewall, authentication, and TLS, a security protocol that encrypts communications.

Splitting GLM-5.3 across two Stations overcomes one GPU's memory limit, but the optimal throughput setting depends on the workload and must be measured directly.