One GPU, One Model

By the end of this session you will be able to:

• Start vLLM 0.26 on one GPU and reach it from any OpenAI-compatible client without changing that client's code
• Find the four startup log lines that tell you how the GPU's memory was actually spent
• Predict, before you send a single request, how many requests the server can hold in flight at your chosen context length

A serving engine is a memory allocator with a model attached

When vLLM starts, it claims a fixed slice of the GPU up front and then never asks the driver for more. That slice is gpu_memory_utilization times total device memory. In vLLM 0.26 the default is 0.92, not the 0.9 you will find in most blog posts written before 2026. On an 80 GiB H100 that is roughly 73.6 GiB to spend, and it goes to three things in a fixed order: the model weights, the peak activation memory for a forward pass, and the remainder.

The remainder is the KV cache, and the KV cache is the thing that decides how many users you can serve at once. Notice that it is a residual, not an allocation you request. The weights are fixed by the checkpoint. The activation peak is measured rather than guessed: vLLM runs a profiling forward pass during startup to find it. Everything left over becomes cache. This is why loading a model that is 8 GiB larger does not cost you 8 GiB of headroom, it costs you 8 GiB of concurrency, and on a card where the cache was already the smaller half of the budget that can halve the number of users you can hold.

Starting it, and proving it speaks OpenAI

One command:

vllm serve Qwen/Qwen3-8B --max-model-len 32k --port 8000

Port 8000 is the default, so that flag is there to be explicit rather than because you need it. --max-model-len accepts human-readable suffixes in 0.26, so 32k and 32768 are the same thing. You are declaring a ceiling here, and the rest of this lesson is about what that declaration costs.

When the log stops moving, check the server from a second terminal:

curl http://localhost:8000/v1/models

The id that comes back is Qwen/Qwen3-8B, the exact string you passed to vllm serve, and it is the string every request has to use in its model field. Change it with --served-model-name if you want something shorter. Then send a real request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Reply with the word ready."}],
    "max_completion_tokens": 16
  }'

max_completion_tokens is the current field name. max_tokens still works but vLLM marks it deprecated, and new code should not be written against it. The same endpoint works from the official client:

from openai import OpenAI
 
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
 
resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Reply with the word ready."}],
    max_completion_tokens=16,
)
print(resp.choices[0].message.content)

vLLM ignores the key unless you started it with --api-key, but the SDK refuses to construct a client without one, so pass any non-empty string.

The four numbers

Scroll back through the startup log. Buried among the compilation chatter are four lines that describe your entire memory situation:

INFO 08-04 09:14:02 [gpu_model_runner.py:5385] Model loading took 15.26 GiB memory and 8.442136 seconds
INFO 08-04 09:14:29 [gpu_worker.py:564] Available KV cache memory: 54.71 GiB
INFO 08-04 09:14:30 [kv_cache_utils.py:2201] GPU KV cache size: 398,384 tokens
INFO 08-04 09:14:30 [kv_cache_utils.py:2203] Maximum concurrency for 32,768 tokens per request: 12.16x

Model loading took N GiB memory. The weights as they actually landed on this device, after quantization and after any tensor-parallel split. This is your reality check on the checkpoint: if an 8B model reports 15.26 GiB you loaded bf16, and if you expected a quantized checkpoint at roughly half that, you fetched the wrong repo.

Available KV cache memory: N GiB. The residual, after weights and after the measured activation peak. This is the only one of the four that moves when you touch gpu_memory_utilization, and it is the number to quote when someone asks what a bigger card would buy you.

GPU KV cache size: N tokens. The previous line converted into token slots. It is a total across all sequences at once, not a per-request figure. This is the number that actually gets divided up between your users.

Maximum concurrency for M tokens per request: Xx. Token slots divided by max_model_len. It answers one question: if every request in flight ran all the way to the ceiling you declared, how many of them fit.

Line numbers shift between builds, so match on the text when you grep, never on kv_cache_utils.py:2201.

Turning the fourth number into a prediction

The arithmetic is deliberately boring. 398,384 token slots divided by 32,768 tokens per request gives 12.16, and vLLM prints exactly that. What people get wrong is the direction of the inference.

That 12.16x is a floor under a worst case, not a description of your traffic. If your real requests average 2,000 tokens of context and output, the same cache holds close to 199 of them, and your server is nowhere near as small as the log suggests. Run the same model with --max-model-len 131072 because that is what the model card advertises, and the cache does not shrink, but the printed concurrency drops to 3.04x and every capacity conversation you have from then on is wrong by a factor of four.

The ceiling is not free in the other direction either. vLLM refuses to start if a single request at max_model_len will not fit in the cache, and the error tells you plainly to raise gpu_memory_utilization or lower max_model_len. Set the ceiling to the largest context you actually intend to accept, then read the concurrency line as the guarantee you can defend under review.

Try it

Start the same model twice and record all four numbers each time:

1. vllm serve Qwen/Qwen3-8B --max-model-len 32k
2. Stop it, then vllm serve Qwen/Qwen3-8B --max-model-len 8k

Success condition: GPU KV cache size in tokens changes by less than about five percent between the two runs, while Maximum concurrency goes up by roughly four times. That is the whole point. The cache is the same pool of memory in both runs. Only your declared ceiling changed, and the ceiling is a claim about a request, not a claim about the hardware. If the token count moved a lot, something else on that GPU changed between runs, and you should find out what before you trust either number.

Common mistakes

• Reading maximum concurrency as a throughput number. It counts sequences that fit in memory at once. It says nothing about requests per second, which depends on how fast tokens come out, not how many fit.
• Copying 0.9 as the memory utilization default. In 0.26 it is 0.92. On an 80 GiB card that gap is about 1.6 GiB, which is over ten thousand KV tokens, and a sizing sheet that starts wrong stays wrong.
• Leaving max-model-len at whatever the model card advertises. You either fail to start, or you publish a concurrency figure four times smaller than the one your traffic will actually see.
• Sending a model string that does not match /v1/models. You get a 404 that reads like the server is down. Check /v1/models first, every time.

Where this goes next

You now have a server and a memory budget you can explain. In Prefill and Decode you take one request apart and find two completely different workloads inside it: a prefill phase that saturates the GPU's compute, and a decode phase that starves on memory bandwidth while the compute sits idle.