An inference engine is the server software between an application and the GPU. It takes each chat request, turns it into tokens, runs the model, and streams the reply back, while doing the same for many other users at once.
What it does for every request
- Applies the model's chat template to the request.
- Splits the text into tokens, the units the model reads and writes.
- Checks its cache for parts of the prompt it has already computed.
- Schedules the request into the next GPU step, alongside other users' requests.
- Prefill: reads the whole prompt in one pass and produces the first output token.
- Decode: produces the rest of the reply, one token per GPU step.
- Turns the output tokens back into text and streams them to the user.
The next chapter follows one real request through these seven steps on each engine.
Terms used in this guide
| Term | Meaning |
|---|---|
| Token | A piece of text, often part of a word, that the model reads or writes. The sample request in this guide is 84 tokens; its reply is 63. |
| Chat template | Fixed text the engine wraps around every request: model instructions, the date, and markers for who is speaking. |
| Prefill | The pass that reads the prompt. All prompt tokens are processed together. |
| Decode | Generating the reply. Each GPU step adds one token to every active request. |
| KV cache | The intermediate results (keys and values) the model keeps for every token, so they are not recomputed at each step. |
| Prefix caching | Keeping the KV cache of a prompt's opening so a later request that starts the same way can reuse it. |
| Continuous batching | Adding new requests to the GPU's work at the next step instead of waiting for current requests to finish. |
| Throughput | Output tokens per second across all users: how much one GPU can serve. |
| Time to first token | How long a user waits before the reply starts to appear. |
The engines in this guide
| Engine | What it is |
|---|---|
| vLLM | Open-source serving engine built for many concurrent users on data-centre GPUs; the most widely deployed. |
| SGLang | Open-source serving engine with similar goals, known for its radix-tree prefix cache. |
| Ollama | Tool for running models on a laptop or single machine, aimed at one user at a time. |
One GPU serves many users by running their requests in the same GPU steps. How well an engine does this sets both how much the GPU can serve and how quickly each user gets an answer.
- Each GPU step runs the model once for every request in the current batch. In this benchmark a vLLM step serving 64 users took about 2.2 times as long as a step serving one user (per-user speed fell from 303 to 137 tok/s), so the GPU did 64 times the work in about twice the time.
- vLLM and SGLang use continuous batching: a new request joins at the next step instead of waiting for the others to finish.
- More users in a batch raise total output, but each user gets a smaller share of each step, so per-user speed falls and the wait for the first token grows.
Compare the engines at one load level
Ollama was configured for 64 parallel users and tested up to 64; vLLM and SGLang up to 256.
Time to one complete answer (about 450 tokens)
The animation replays each engine's median first-token wait and per-user speed at the selected load, scaled so the slowest engine finishes in about 8 seconds.
Prefix caching keeps the computed keys and values for a prompt's opening, so the next request that starts the same way skips that work. It matters most when every request carries the same long system prompt.
- Every request in the sample trace shares its first 64 tokens (the chat template) with every other request; with caching, only the last 20 are computed.
- Assistants and agents often send the same long instructions, policies or tool definitions on every call, so most of each prompt can be reused.
- vLLM matches cached work in 16-token blocks; SGLang matches it token by token in a radix tree.
Prefix cache on and off
Every request carries the same 1,515-token store-policy system prompt, then a short customer question.
The diagram is illustrative. The bars below are measured, and the bold row is the vLLM setting you selected.
Measured effect
Total throughput, output tok/s, shared 1,515-token prompt
Wait for the first token, 95th percentile, seconds
- At 128 users, vLLM with caching produced 8,449 tok/s against 1,756 without (4.8x), with a first-token p95 of 0.70 s against 3.82 s.
- vLLM and SGLang both cache prefixes by default and stay close up to 128 users. At 256 users SGLang is ahead, at 8,262 against 6,913 tok/s and a first-token p95 of 1.7 s against 2.7 s.
- The effect applies to assistants and agents that send the same long instructions, policies or tool definitions on every call.
At peak, self-hosted vLLM produces a million output tokens for $0.10, a third of the hosted price for the same model and a hundredth of a frontier model's price. Ollama on the same GPU costs 49x more per token.
- These figures assume the GPU runs at peak. At 30% average utilisation, vLLM's $0.10 becomes about $0.34, level with the hosted API; self-hosting is cheaper only above about 34% utilisation. Engineering, on-call and platform costs are not included.
- A 20B open model and a frontier model differ in quality, so compare them on cost per successful task rather than cost per token.
- On the same GPU at the same hourly price, the engine that keeps the GPU busier produces cheaper tokens.
Cost calculator
Peak throughput is measured. Utilisation is the average share of that peak your traffic uses. Prices are November 2026 list prices; 1 USD = 95 INR.
Beats the hosted gpt-oss-20b price ($0.30) when
Against API list prices, USD per 1M output tokens (log scale, $0.01 to $100)
This chapter follows one real request through each engine: how it is tokenized, what the cache reuses, how it shares the GPU with other users, and how the reply is generated token by token. With 63 other users active, the same reply takes 0.6 s on vLLM and 15.8 s on Ollama.
vLLM, SGLang and Ollama follow the same seven steps, but they split the work across processes differently, find cached prompts differently and handle many users differently.
vLLM
SGLang
Ollama
| vLLM | SGLang | Ollama | |
|---|---|---|---|
| Many users | Continuous batching: one decode token per active request plus new prompt tokens in each step, up to a token budget | Continuous batching, plus an overlap scheduler that prepares the next batch on the CPU while the GPU runs the current one | A fixed number of parallel slots, each with its own context |
| Prompt cache | KV cache in fixed 16-token blocks; reuse in whole blocks | Radix tree of cached tokens; reuse matched token by token | Per slot; reuse not measured here |
| Built for | Production serving, broad model and hardware support | Production serving, long shared prompts and agent workloads | Laptops, prototyping and single users |
On one H100 serving gpt-oss-20b, vLLM peaked at 12,011 output tokens per second and Ollama at 246. The engine was the only difference.
What we tested
- OpenAI's gpt-oss-20b (open weights, about 3.6B active parameters per token) ran on one NVIDIA H100 80GB, served in turn by Ollama, vLLM and SGLang from the same MXFP4 weights.
- The load generator simulated 1 to 256 concurrent users (Ollama up to its 64 configured slots). Each user sent three chat requests of up to 512 output tokens. Every level ran three times and the median is shown. Of 20,976 requests, 1 failed.
- Two workloads were tested: independent retail prompts, and requests that all carry the same 1,515-token store-policy system prompt, as assistant and agent traffic does.
- Each run recorded total output per second, speed per user and the wait for the first token.
Capacity under load
Total throughput, output tokens per second, all users (users on a log scale)
Speed per user, output tok/s (median user)
Wait for the first token, 95th percentile, seconds
- With one user, vLLM and SGLang stream about 300 tok/s and Ollama about 100.
- At 64 users, vLLM produced 7,275 tok/s, SGLang 6,386 and Ollama 246. With 32 users, vLLM kept the GPU 100% busy and Ollama about 15%.
- vLLM and SGLang add every waiting request to the next GPU step (continuous batching). Ollama gave each of its 64 users a slot, but its total output barely rose, so each added user slowed the others.
- Higher throughput costs latency. At 256 users, per-user speed falls to about 60 tok/s and the first-token wait rises to 4-6 s. With the wait held under 1 s, one H100 on vLLM serves about 128 users.
What drives the results
| What changed | What happens on the GPU | Measured effect |
|---|---|---|
| Users, 1 to 256 (vLLM) | More requests share each GPU step | Total throughput up 41x, 296 to 12,011 tok/s |
| Users, 1 to 256 (vLLM) | Each step serves more people, so each gets a smaller share | Speed per user down from 303 to 58 tok/s |
| Users, 128 to 256 (vLLM) | The batch is full; new requests wait | First-token p95 0.68 s to 3.81 s, throughput only +12% |
| Engine, Ollama to vLLM (64 users) | Continuous batching keeps the GPU busy; Ollama left it about 85% idle | 246 to 7,275 tok/s (30x); first-token p95 6.50 s to 0.30 s |
| Engine, vLLM to SGLang (independent prompts) | Similar batching and kernels | Within 10% at every level |
| Prefix cache, off to on (vLLM, 128 users) | The shared 1,515-token prompt is computed once, not per request | 1,756 to 8,449 tok/s (4.8x); first-token p95 3.82 s to 0.70 s |
| Peak throughput, 246 to 12,011 tok/s | The same $4.39 GPU hour produces more tokens | Cost per 1M tokens $4.95 to $0.10 |
- Capacity depends on the latency target. With the first-token wait held under 1 s (p95), one H100 on vLLM serves about 128 concurrent users at 10,721 tok/s, about 100 tok/s each.
Use vLLM or SGLang for anything served to more than a few people, and Ollama for laptops and single-user work.
| Engine | Use it for | Evidence from this benchmark |
|---|---|---|
| Ollama | Laptops, prototyping, a single developer, offline demos | 100 tok/s for one user; plateaus at 246 tok/s with the GPU about 15% busy |
| vLLM | The production default: broadest model and hardware support | Highest peak, 12,011 tok/s; best per-user speed up to 128 users |
| SGLang | Long shared prompts at very high concurrency: agents, assistants, multi-turn chat | Within 10% of vLLM on independent prompts; ahead at 256 users on the shared prompt |
What the numbers do and do not cover
- One model (gpt-oss-20b), one H100, synthetic retail prompts, default engine settings. Longer contexts, other models, tuning or more GPUs change the numbers.
- vLLM and SGLang repeated within 10% at most levels. Ollama's runs varied by up to 1.9x at 4-32 users. The differences between engines are much larger than this variation.
- Cost figures assume the GPU runs at peak and exclude people and platform costs.
- vLLM and SGLang now have similar features. For a production decision, benchmark your own model and traffic the same way.