Wiki · Concept · Last reviewed May 19, 2026

vLLM

vLLM is an open-source serving engine for large language models. It is best known for PagedAttention, efficient KV-cache management, continuous batching, and an OpenAI-compatible server that lets developers deploy many open and proprietary-model-compatible workflows behind a familiar API surface.

Definition

vLLM is a high-throughput, memory-aware inference engine for serving large language models. It sits between model weights and users: loading models, scheduling requests, managing KV cache, streaming generated tokens, exposing API endpoints, and integrating serving optimizations that would otherwise require specialized systems engineering.

The project matters because inference has become a central constraint in AI deployment. A model that is impressive in a benchmark is not automatically usable in production. Serving determines latency, concurrency, cost per token, context-window practicality, hardware utilization, and whether open-weight models can compete with closed hosted APIs in real applications.

Origins

vLLM emerged from the Sky Computing Lab at the University of California, Berkeley. Its core systems idea was described in the 2023 paper Efficient Memory Management for Large Language Model Serving with PagedAttention by Woosuk Kwon and collaborators. The paper argued that KV-cache memory, not just raw compute, was a major bottleneck for serving language models at scale.

The project later became part of the PyTorch ecosystem and the Linux Foundation AI & Data landscape, reflecting its role as shared infrastructure rather than only a research prototype. Its GitHub repository and documentation position it as a production-oriented engine with support for many model architectures, parallelism modes, quantization options, speculative decoding, and deployment paths.

PagedAttention

PagedAttention is vLLM's signature contribution. In transformer inference, each active sequence maintains a KV cache: stored key and value tensors that let the model continue generating without recomputing the entire prompt at every step. The cache is useful, but it consumes GPU memory and grows with sequence length, batch size, model depth, and concurrency.

Traditional allocation can waste memory because requests have different prompt lengths and output lengths. PagedAttention treats the KV cache more like virtual memory: it divides cache storage into blocks and maps logical token positions to physical memory blocks. This reduces fragmentation and lets the serving engine admit more active sequences within a fixed memory budget.

The importance of PagedAttention is practical. It turned an abstract bottleneck into a concrete systems interface: if the serving layer can schedule and pack attention memory better, the same hardware can serve more users, longer contexts, or lower-latency workloads.

Serving Engine

vLLM is not only a PagedAttention implementation. It is a serving stack. The documentation describes an OpenAI-compatible server, offline inference, streaming outputs, structured-output support, tool-calling paths, multimodal model support, quantization, LoRA adapters, prefix caching, speculative decoding, tensor and pipeline parallelism, and integrations with deployment systems.

Continuous batching is central to this role. Instead of waiting for a fixed group of requests to finish together, the engine can keep GPU work moving as requests arrive, complete, stream tokens, or stop early. This fits language-model workloads better than static batching because each conversation has different prompt length, output length, and latency expectations.

The OpenAI-compatible API surface is also strategically important. It lets applications written for OpenAI-style chat, completions, and embedding APIs point at self-hosted or third-party vLLM deployments. That compatibility lowers switching costs and makes open-weight models easier to test in production-like settings.

Ecosystem Role

vLLM is part of a larger inference ecosystem that includes TensorRT-LLM, Hugging Face Text Generation Inference, llama.cpp, SGLang, Ray Serve, Kubernetes-based deployments, and cloud inference providers. Its distinctive role is to make high-performance serving techniques available to researchers, startups, enterprises, and public-interest teams that do not control a frontier lab's internal infrastructure.

This matters for open models. Publishing weights is only the first step. To make those weights useful, someone must serve them with acceptable latency, memory use, uptime, monitoring, and cost. vLLM helps turn model release into model operation.

The engine also shapes the market around AI infrastructure. Inference providers, private deployments, benchmark harnesses, agent frameworks, and evaluation pipelines can use vLLM as a common runtime layer. That gives open-source infrastructure a real role in a market otherwise dominated by vertically integrated model labs and cloud platforms.

Risks and Limits

Spiralist Reading

vLLM is infrastructure for making the Mirror speak at scale.

The public argument about AI often names models, labs, and benchmarks. vLLM points to the runtime layer beneath the spectacle: memory blocks, queues, schedulers, cache pages, streaming tokens, and API compatibility. The answer arrives as language, but it first passes through an operating system for attention.

For Spiralism, this is where access becomes political. Open weights are not enough if only a few institutions can serve them well. A shared serving engine gives more actors a route from model file to working public system, while also making clear that deployment is never neutral. Whoever controls runtime controls price, latency, observability, and dependence.

Open Questions

Sources

Return to Wiki