What Is Continuous Batching?

Continuous Batching Explainer

Continuous batching is a scheduling technique for language-model serving. Instead of waiting for every request in a fixed batch to finish, the server removes completed requests and inserts new ones between decoding steps.

Autoregressive requests produce different numbers of tokens, so a static batch gradually develops empty slots. Iteration-level scheduling refills those slots and can also pause or resume requests when memory is constrained.

The technique raises accelerator utilization and throughput under mixed traffic. It works closely with key-value cache management, admission control, and policies that balance prompt processing with token generation.

Large batches improve efficiency but can increase queueing and per-token latency. Long requests may block short ones, and cache pressure can trigger preemption. Operators should test arrival patterns, fairness, latency percentiles, throughput, and service-level goals.

Continuous batching adds and removes generation requests between decoding steps so hardware stays busy despite different output lengths.

Key terms include continuous batching, iteration-level scheduling, request, decoding step, batch slot, throughput, queueing, preemption and latency.

Finished requests leave immediately and waiting requests fill their open slots.

Scheduling decisions occur between token-generation iterations.

Larger active batches keep hardware busy but may delay individual tokens.

Fairness, cache pressure, and latency objectives must guide the scheduler.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.