Ollama vs. vLLM vs. SGLang: Two Different Worlds of LLM Serving

1. Executive Summary#

When developers move from local LLM prototyping to production inference, the most common mistake is treating all model servers as interchangeable wrappers around weights. In reality, running a model for a single developer on a laptop is a fundamentally different systems problem than serving 500 concurrent requests on an H100 cluster.

Core Thesis: Ollama packages local developer ergonomics on top of llama.cpp for single-stream interactive latency; vLLM and SGLang are distributed inference engines designed to maximize GPU compute utilization and memory bandwidth under continuous multi-tenant load.

2. The Core Bottleneck: Why LLM Inference is Memory-Bandwidth Bound#

LLM inference consists of two distinct phases:

• Prefill Phase (Compute-Bound): Processing the user's input prompt in parallel. Every token attends to every preceding token in a massive matrix-matrix multiplication (GEMM) operation. GPU Tensor Cores are saturated here.
• Decode Phase (Memory-Bandwidth Bound): Generating output tokens one by one autoregressively. To generate a single token for a 70B parameter model in FP16, the GPU must stream ~140 GB of weights from High Bandwidth Memory (HBM) into on-chip SRAM just to perform a tiny matrix-vector multiplication (GEMV).

Because the GPU spends most of its time waiting for weights and Key-Value (KV) cache states to move across memory buses rather than doing math, single-request inference leaves 90%+ of a datacenter GPU's compute units idle.

3. Under the Hood: Engine Architecture Comparison#

A. Ollama (Built on llama.cpp / GGML)#

• Primary Design Goal: Zero-configuration local execution, cross-platform portability (Apple Silicon Unified Memory, consumer CUDA GPUs, CPU AVX2/AVX-512), and fast Time-To-First-Token (TTFT) for a single user.
• Memory Management: Allocates contiguous ring/slot buffers for the KV cache. While recent versions support parallel slots, memory must be statically carved out per slot.
• Quantization Format: Uses GGUF (k-quants such as Q4_K_M, Q5_K_M, Q8_0), which are optimized for dequantization on CPUs and consumer GPUs rather than datacenter Tensor Core throughput.

B. vLLM (PagedAttention & Continuous Batching)#

• Primary Design Goal: High throughput (tokens/second across all users) and near-zero memory waste on datacenter GPUs.
• PagedAttention: Traditional engines pre-allocate contiguous VRAM for a request's maximum possible context length (e.g., 32k tokens), wasting 60% to 80% of VRAM on internal and external fragmentation. vLLM borrows the concept of virtual memory paging from operating systems: it divides the KV cache into fixed-size blocks (pages) that can be stored non-contiguously in physical HBM, mapped via a logical block table.
• Continuous (Iteration-Level) Batching: Instead of waiting for every sequence in a batch to finish generating before starting new requests (static batching), vLLM schedules requests at the single-token iteration level. The moment Sequence A emits an EOS token, Sequence B is injected into the batch on the very next forward pass.

C. SGLang (RadixAttention for Agent Workflows)#

• Primary Design Goal: Ultra-fast multi-turn agent loops, structured outputs, and tree-of-thought branching.
• RadixAttention: Instead of discarding KV cache states after a request finishes, SGLang maintains a Least-Recently-Used (LRU) Radix Tree of KV cache pages in GPU memory. When an agent makes 15 sequential tool calls sharing a 10k-token system prompt and conversation history, SGLang matches the longest shared prefix in the radix tree and skips prefill for those tokens entirely—reducing TTFT by 5x to 10x.
• Compressed Finite State Machines (FSM): For JSON schema enforcement, SGLang compiles schemas into jump-forward decoding graphs that emit multiple deterministic structural tokens (like {"key": ") in a single forward pass.

4. Senior Engineer Decision Matrix#