An inference engine is the server software between an application and the GPU. It takes each chat request, turns it into tokens, runs the model, and streams the reply back, while doing the same for many other users at once.

What it does for every request

  1. Applies the model's chat template to the request.
  2. Splits the text into tokens, the units the model reads and writes.
  3. Checks its cache for parts of the prompt it has already computed.
  4. Schedules the request into the next GPU step, alongside other users' requests.
  5. Prefill: reads the whole prompt in one pass and produces the first output token.
  6. Decode: produces the rest of the reply, one token per GPU step.
  7. Turns the output tokens back into text and streams them to the user.

The next chapter follows one real request through these seven steps on each engine.

Terms used in this guide

TermMeaning
TokenA piece of text, often part of a word, that the model reads or writes. The sample request in this guide is 84 tokens; its reply is 63.
Chat templateFixed text the engine wraps around every request: model instructions, the date, and markers for who is speaking.
PrefillThe pass that reads the prompt. All prompt tokens are processed together.
DecodeGenerating the reply. Each GPU step adds one token to every active request.
KV cacheThe intermediate results (keys and values) the model keeps for every token, so they are not recomputed at each step.
Prefix cachingKeeping the KV cache of a prompt's opening so a later request that starts the same way can reuse it.
Continuous batchingAdding new requests to the GPU's work at the next step instead of waiting for current requests to finish.
ThroughputOutput tokens per second across all users: how much one GPU can serve.
Time to first tokenHow long a user waits before the reply starts to appear.

The engines in this guide

EngineWhat it is
vLLMOpen-source serving engine built for many concurrent users on data-centre GPUs; the most widely deployed.
SGLangOpen-source serving engine with similar goals, known for its radix-tree prefix cache.
OllamaTool for running models on a laptop or single machine, aimed at one user at a time.