Continuous batching architecture

Traditional batching processes a fixed group of requests together and waits for the slowest one to finish before starting the next group. This causes the GPU to idle between batches.

With continuous batching, at every generation step, the scheduler checks for finished requests and replaces them immediately with waiting ones. Short requests are kicked out as soon as they’re done, and the GPU stays occupied the entire time. The result is significantly higher throughput and lower average latency.

Request lifecycle

A request moves through four states from submission to completion.

       WAIT IN QUEUE          LOAD PROMPT INTO KV       STREAM OUTPUT           DONE
    ┌────────────────┐    ┌─────────────────────┐    ┌─────────────────┐    ┌────────┐
    │    PENDING     │───▶│     PREFILLING      │───▶│    DECODING     │───▶│FINISHED│
    │  ○ ○ ○ · · ·   │    │ ████████░░░░░░░░░░  │    │  → → → → · ·    │    │   ✓    │
    └────────────────┘    └─────────────────────┘    └─────────────────┘    └────────┘
         others ahead            prompt chunks              +1 token/step

1. Pending — the request is queued and waiting to be scheduled.
2. Prefilling — prompt tokens are processed in a forward pass.
3. Decoding — output tokens are generated one at a time.
4. Finished — generation is complete and the result is available.

A request is moved from pending to prefilling when the scheduler finds enough token budget and cache space to admit the request. If the prompt is too long to fit in a single step, the scheduler splits it across multiple forward passes (chunk prefill).

Chunked prefill

Processing a long prompt in a single step is expensive because it blocks other requests from generating. Chunked prefill solves this by splitting large prefills across multiple steps.

Blocking (long prefill blocks the batch)
  Step:   1        2        3        4
  Req A  [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓]   ← one huge prefill
  Req B  ················ wait ················
  Req C  ················ wait ················

Chunked (same prompt split and interleave with decode)
  Step:   1          2          3          4
  Req A  [▓▓ pre]    [▓▓ pre]   [▓▓ pre]   [→ dec]
  Req B  [→ dec]     [→ dec]    [▓▓ pre]   [→ dec]
  Req C  [→ dec]     · idle ·   [→ dec]    · idle ·

When a new request’s prompt exceeds the available token budget (set by max_batch_tokens in ContinuousBatchingConfig), the scheduler processes as many tokens as possible and holds the rest. On subsequent steps, it continues from where it left off, interleaving prefill work with ongoing decode steps. This keeps the batch productive and reduces time-to-first-token for other requests.

Scheduler

The scheduler decides which requests join each forward pass based on two budgets and a request cap.

• Token budget — the maximum number of query tokens processed in a single forward pass, set by max_batch_tokens in ContinuousBatchingConfig.
• Cache budget — the total number of KV pages that can be read in a single pass, which is bounded by the total cache size.
• Request cap — the maximum number of requests in a single forward pass, set by max_requests_per_batch. The logits tensor scales with the vocabulary size, so this cap bounds peak memory for large-vocabulary models.

FIFO

FIFOScheduler is the default scheduler. It fills the batch in priority order. Active decode requests are processed first, then active prefill requests, and then new waiting requests in arrival order. This maximizes throughput by pushing existing requests through before pulling in new ones.

PrefillFirst

PrefillFirstScheduler completes chunked prefill operations before resuming decode. This reduces fragmentation for workloads dominated by long prompts.

Set the scheduler in ContinuousBatchingConfig.

cb_config = ContinuousBatchingConfig(scheduler_type="prefill_first")

Memory management

The KV cache is paged so requests of different lengths can share a fixed memory pool without fragmentation. A shared pool of memory chunks, implemented as CachePool, supplies those chunks to each attention type allocator, which divides them into blocks and pages.

Shared pool (first two chunks reserved for padding / never allocated)

  chunk N  [ ======================================== ]
               |                            |
    full attn: [ FULL BLOCK 0 | FULL BLOCK 1 | … ]
               [ page L0 | page L0 | … ]   (one page per full layer)

    sliding:   [ ----------- SLIDING BLOCK 0 ----------- ]
               [ page L1 | page L2 | page L3 | … ]  (one page per sliding layer)

  page  = page_size tokens of KV for one layer
  block = one page per layer of that attention type
  chunk = pool unit sized so full and sliding blocks tile the same bytes

Continuous batching supports two attention types only: full_attention (FullAttentionCacheAllocator) and sliding_attention (SlidingAttentionCacheAllocator). Hybrid models get one allocator of each kind. Attention type comes from the model (layer_types / sliding_window), not from ContinuousBatchingConfig.

Sliding layers keep only a fixed window of recent tokens by recycling a small set of pages. They do not support kernel block tables or block sharing.

The cache reserves two chunks ahead of the data chunks for padding reads and writes that requests never use. Those chunks are never allocated to requests.

Cache sizing

The manager infers how many chunks fit at startup from free GPU memory. It accounts for KV tensors, attention masks, activations, and bookkeeping indices, then sizes the pool to fit inside max_memory_percent of the available memory. When unset, it defaults to a value according to the hardware and continuous batching parameters. MPS reserve less space for the cache, where the CPU and GPU share memory, to avoid swapping. num_blocks expresses capacity as whole model blocks of page_size tokens when you set it explicitly.

You can pin the values in ContinuousBatchingConfig.

cb_config = ContinuousBatchingConfig(
    page_size=256,
    num_blocks=4096,
    max_batch_tokens=512,
    max_memory_percent=0.8,
)

Admission

Before a request joins the batch, the scheduler checks that enough free capacity exists for every attention type allocator. If any allocator would fall short, the request is rejected and nothing is allocated.

To avoid filling the cache until offloading is the only option, the scheduler enforces a safety_margin. Once free cache capacity falls below safety_margin, new prefills are held back and only active decodes continue. The FIFO scheduler defaults to 0.15 (15% of capacity held in reserve), while prefill_first defaults to 0.0. Set safety_margin in ContinuousBatchingConfig to tune it, where 0 disables the margin.

Prefix caching

When two requests share a prompt prefix, they can share the blocks that hold the KV for that prefix. Each completed block is content-hashed. A later request with a matching prefix reuses the block and skips the prefill for those tokens. Shared blocks are reference-counted and only return to the free pool once every request using them has finished.

allow_block_sharing defaults to True. Prefix sharing is active only when every allocator supports sharing, so only full-attention-only models use it. Pure sliding and hybrid models disable it. Set allow_block_sharing=False for workloads with short prompts and long generations, where the bookkeeping outweighs the savings.

Eviction

Freed blocks aren’t wiped and they stay “initialized” with their content and hash intact so a later prefix match can reuse them. When the uninitialized pool runs low, the manager lazily demotes the most recent initialized block back to uninitialized, dropping their hash. The prefix cache is a best-effort layer that the allocator discards to keep serving new requests.

Soft reset

A soft reset is the fallback path for offloading. When the manager can’t copy a request’s KV cache to the CPU swap pool, because CPU offloading is disabled or the pool is full, it soft resets the request instead. It appends the request’s generated tokens to its original prompt, frees its cache blocks, and adds it back to the queue. The Offloading section covers which requests are chosen.

When the cache has space again, the request resumes from where it left off with its generation history encoded in the prompt.

CUDA graphs

Every generation step involves CPU overhead, from assembling the batch to dispatching GPU kernels and reading the results. CUDA graphs eliminate the overhead by recording the full GPU execution sequence once and replaying it for batches with matching shapes.

Because the batch shapes change every step, the continuous batching system handles this with padding and caching.

1. Query lengths are padded to the nearest multiple of q_padding_interval_size.
2. KV lengths are padded to the nearest multiple of kv_padding_interval_size.
3. Recorded graphs are stored in a cache.

When a batch’s padded shape matches a cached graph, the graph replays without any CPU dispatch. New shapes trigger a new graph capture.

CUDA graphs require static control flow and are incompatible with attention masks. They’re auto-detected by default and disabled when conditions aren’t met.

Async batching

Async batching uses two I/O buffer pairs and two CUDA streams to overlap CPU and GPU work.

Sequential
  CPU  ── [prep N] ·········idle········· [prep N+1] ·········idle·········
  GPU  ·········idle········· [compute N] ·········idle········· [compute N+1]

Async
  CPU  ── [prep N] [prep N+1] [prep N+2] ──
  GPU  ── ········ [compute N] [compute N+1] ──

While the GPU computes batch N, the CPU prepares batch N+1. However, overlapping the two requires roughly double the VRAM.

Async batching requires CUDA graphs to be active, since graph replay provides the stable tensor addresses needed for stream overlap to be correct.

Offloading

Requests that generate long outputs can grow until the KV cache has no room to allocate the next block. Rather than crash, the manager offloads active requests back to the waiting queue to free blocks for the requests that remain.

Offloading is demand-driven. The scheduler reports the stalled requests, the active requests that failed to allocate the blocks they need, along with how many blocks each one demands. The manager offloads the fewest requests needed to free enough blocks to cover that demand. It offloads newest first, so requests already scheduled into the current batch keep their cache, and the last remaining active request is never offloaded.

Two situations trigger offloading.

• A batch is scheduled, but some active requests stalled. The manager offloads enough of the stalled requests that the next batch can allocate blocks for the rest.
• The cache is full and no batch can be scheduled at all. The manager offloads requests and retries scheduling in a loop within the same step, until a batch fits or no request can be offloaded. It no longer waits for the next forward pass to recover.

An offloaded request takes one of two paths. When cpu_offload_space is greater than 0.0, the manager copies the request’s KV cache blocks to a pre-allocated pinned CPU buffer, batching every request that fits the swap pool into a single transfer. After GPU cache space becomes available, the manager copies the blocks back to the GPU and resumes the request without recomputing its prompt and generated tokens. Requests that don’t fit the pool, or every offloaded request when CPU offloading is disabled, fall back to a soft reset.

Next steps

• The Continuous batching blog post covers KV caching, chunked prefill, and dynamic scheduling with performance benchmark numbers.
• For usage examples, see the Continuous batching doc.