← Learn / Notebooks

Models & toolsPractitionerStudy guide

Model Serving: Batching and Request Queues

Compare a batch barrier with slot refilling using a visible request timeline and a deterministic scheduler.

Before you start: Know that requests can finish at different times. No GPU or serving framework is required.

AI-assisted study notebook · Published 14 Nov 2026. Source-support checks and synthetic examples are recorded; automated review is not independent expert fact verification.

In [1]: Architecture

1. Architecture and scope

A fixed batch can leave capacity unused when one sequence finishes before its batch peers. Continuous batching changes active membership as requests finish and new work enters. The Hugging Face explanation develops that idea from inference mechanics; the current manager documentation shows request-oriented control. [1] [2]

Admission policy is a separate question from batch membership. A queue can preserve arrival order, prioritize classes or distribute turns among users. TNG’s operational report describes per-user queues and a separate scheduling layer. Its example also shows why filling a downstream queue can undermine a fairness policy implemented upstream. [3]

Scope. A discrete scheduling example. Ticks are abstract units, not token latency or measured GPU time.

Workflow and scope notes
  1. Accept work. Keep pending requests until execution capacity is available. The experiment gives every request arrival time zero.
  2. Admit requests. Choose which queued request enters each free slot. The toy uses FIFO; real fairness and admission policies are separate design choices.
  3. Advance work. Each active request consumes one unit of its declared duration. This deliberately excludes prefill costs, memory limits and unequal per-step work.
  4. Release slots. A static batch waits for its longest member; refilling admits new work as slots free. Compare idle cells and completion time under the same input durations.
In [2]: Experiment

2. Static batches and continuous slot refilling

Four requests arrive together: A needs 2 ticks, B 6, C 1 and D 4. Both policies use the same slot count. Static batches have a barrier between successive groups; refilling replaces a finished request on the next tick.

Static batches and continuous slot refilling

Original synthetic experiment · Change an input to update the figure.

Every active request advances by one unit per tick. There is no memory pressure, prefill stage, network overhead or cancellation. Slot occupancy is busy slot-ticks divided by capacity slot-ticks, not GPU utilization.

Request timelinesStatic 10 ticks (65.0% occupancy) · Refill 7 ticks (92.9% occupancy). A needs 2 ticks, B 6, C 1, D 4. Empty cells are idle slots. Horizontal axis is ticks; S labels are slots.STATIC10 ticksS1AACS2BBBBBBDDDD012345678910REFILL7 ticksS1AACDDDDS2BBBBBB01234567
Static 10 ticks (65.0% occupancy) · Refill 7 ticks (92.9% occupancy)
Calculation record
{
  "inputs": {
    "slots": "2"
  },
  "static": 10,
  "refill": 7,
  "static_occupancy": 0.65,
  "refill_occupancy": 0.9285714285714286
}
CaseResultInterpretation
Static policy at 2 slots10 ticksA/B form the first batch (6 ticks); C/D form the second (4 ticks).
Refill policy at 2 slots7 ticksC starts after A, then D uses that slot while B continues.
Total useful work13 slot-ticksBoth policies perform the same declared work; the idle capacity differs.
In [3]: Interpretation

3. Interpretation and limitations

At two slots, the static timeline contains 20 capacity slot-ticks and 13 busy slot-ticks, so occupancy is 65%. Refilling finishes in 7 ticks, giving 13/14 ≈ 92.9%. This synthetic difference is caused by the barrier; it is not a measured speedup for a serving product.

Changing to four slots lets all four requests start together, so both policies finish at tick 6. This counterexample matters: refilling creates no benefit in this fixture when there is no pending request to admit. Real batch construction also interacts with sequence length, memory and scheduling overhead. [1] [2]

Editorial operations exercise: record queue delay separately from execution time, state the admission policy and watch where requests accumulate. Do not infer fairness from good occupancy. A scheduler can keep every slot busy while making one user wait disproportionately. [3]

Practice exercise

Use three slots. Predict when request D starts under each policy. Then inspect the timeline and explain whether lower queue wait also implies a fair policy for multiple users.

In [4]: Python

4. Worked Python example

The core calculation below uses only the Python standard library. The full download adds the verification assertions and input fixtures. It runs locally on synthetic data; it does not load a model or contact a service.

lab.py · Core example · Python 3.13
from math import ceil

DURATIONS = [2, 6, 1, 4]


def static_time(slots):
    return sum(max(DURATIONS[i:i + slots])
               for i in range(0, len(DURATIONS), slots))


def refill_time(slots):
    # FIFO list scheduling: admit the next request in the first free lane.
    available = [0] * slots
    for duration in DURATIONS:
        lane = min(range(slots), key=lambda i: (available[i], i))
        available[lane] += duration
    return max(available)


def result(slots):
    static, refill = static_time(slots), refill_time(slots)
    return {"static": static, "refill": refill,
            "static_occupancy": 13 / (slots * static),
            "refill_occupancy": 13 / (slots * refill)}

Download full Python example and checks ↓

Run: python -I lab.py. Recorded run: Python 3.13; 4 fixture states plus scenario assertions passed. The JSON output is a reproducibility record, not a real-system benchmark.

In [5]: Knowledge check

5. Questions and explanations

Predict before opening the explanation. Answers refer to the stated scope and synthetic inputs.

Q1 · BarrierWhy can the static policy show idle cells?
Why can the static policy show idle cells?
Answer and explanation

Answer: It waits for the current batch’s longest request. Finished slots remain unused until that batch finishes in this declared static policy.

Q2 · UnitsWhat does 92.9% mean in the two-slot refill example?
What does 92.9% mean in the two-slot refill example?
Answer and explanation

Answer: Busy slot-ticks / available slot-ticks. It is a synthetic capacity occupancy ratio, not a hardware metric.

Q3 · QueueDoes continuous membership itself specify per-user fairness?
Does continuous membership itself specify per-user fairness?
Answer and explanation

Answer: No. Batch membership and request admission policy answer different questions. [3]

Q4 · CounterexampleAt four slots with four initially queued requests, what happens here?
At four slots with four initially queued requests, what happens here?
Answer and explanation

Answer: Both policies finish at tick 6. All requests start together; there is no waiting work to refill a slot.

In [6]: Sources

References and source links

  1. Ouazan Reboul, R., Zucker, A., & Georges, L. (2025). Continuous batching from first principles.https://huggingface.co/blog/continuous_batching
  2. Hugging Face maintainers. (Main documentation; accessed 2026). Continuous batching · Hugging Face.https://huggingface.co/docs/transformers/main/continuous_batching
  3. TNG Technology Consulting authors. (2025). Efficient Request Queueing – Optimizing LLM Performance.https://huggingface.co/blog/tngtech/llm-performance-request-queueing
Source checks and provenance

Primary publisher pages and relevant passages checked on 13 Nov 2026. Source support is not independent expert fact verification. Main documentation can change; retrieved-body hashes, dates, version labels and saved revision IDs are in the audit download.

[1] Continuous batching from first principles

Evidence: Continuous batching section. Conceptual introduction read in the saved public original and current publisher page. No benchmark result is reused.

Public ResearchNode original 7b44225dc8d09a3a6793eaef130699be
Revision 93ad81780057f3e35e2e0c92235deb0e. Captured 2026-11-06.

[2] Continuous batching · Hugging Face

Evidence: Continuous batching introduction. Current manager and request lifecycle documentation; the toy scheduler is not this implementation.

[3] Efficient Request Queueing – Optimizing LLM Performance

Evidence: Solution: Fair Scheduling; No Backpressure by Backend Queue. A first-party operational report, not a universal scheduling guarantee.

Public ResearchNode original 8f6d7dcacc39d11598a0bbeb01177459
Revision 9edc40898805d54aacf6d862a3bea560. Captured 2026-11-06.

Claim-to-evidence ledger ↓ · Review report ↓