1. Architecture and scope
A fixed batch can leave capacity unused when one sequence finishes before its batch peers. Continuous batching changes active membership as requests finish and new work enters. The Hugging Face explanation develops that idea from inference mechanics; the current manager documentation shows request-oriented control. [1] [2]
Admission policy is a separate question from batch membership. A queue can preserve arrival order, prioritize classes or distribute turns among users. TNG’s operational report describes per-user queues and a separate scheduling layer. Its example also shows why filling a downstream queue can undermine a fairness policy implemented upstream. [3]
Scope. A discrete scheduling example. Ticks are abstract units, not token latency or measured GPU time.
Workflow and scope notes
- Accept work. Keep pending requests until execution capacity is available. The experiment gives every request arrival time zero.
- Admit requests. Choose which queued request enters each free slot. The toy uses FIFO; real fairness and admission policies are separate design choices.
- Advance work. Each active request consumes one unit of its declared duration. This deliberately excludes prefill costs, memory limits and unequal per-step work.
- Release slots. A static batch waits for its longest member; refilling admits new work as slots free. Compare idle cells and completion time under the same input durations.
2. Static batches and continuous slot refilling
Four requests arrive together: A needs 2 ticks, B 6, C 1 and D 4. Both policies use the same slot count. Static batches have a barrier between successive groups; refilling replaces a finished request on the next tick.
Static batches and continuous slot refilling
Original synthetic experiment · Change an input to update the figure.
Every active request advances by one unit per tick. There is no memory pressure, prefill stage, network overhead or cancellation. Slot occupancy is busy slot-ticks divided by capacity slot-ticks, not GPU utilization.
Calculation record
{
"inputs": {
"slots": "2"
},
"static": 10,
"refill": 7,
"static_occupancy": 0.65,
"refill_occupancy": 0.9285714285714286
}| Case | Result | Interpretation |
|---|---|---|
| Static policy at 2 slots | 10 ticks | A/B form the first batch (6 ticks); C/D form the second (4 ticks). |
| Refill policy at 2 slots | 7 ticks | C starts after A, then D uses that slot while B continues. |
| Total useful work | 13 slot-ticks | Both policies perform the same declared work; the idle capacity differs. |
3. Interpretation and limitations
At two slots, the static timeline contains 20 capacity slot-ticks and 13 busy slot-ticks, so occupancy is 65%. Refilling finishes in 7 ticks, giving 13/14 ≈ 92.9%. This synthetic difference is caused by the barrier; it is not a measured speedup for a serving product.
Changing to four slots lets all four requests start together, so both policies finish at tick 6. This counterexample matters: refilling creates no benefit in this fixture when there is no pending request to admit. Real batch construction also interacts with sequence length, memory and scheduling overhead. [1] [2]
Editorial operations exercise: record queue delay separately from execution time, state the admission policy and watch where requests accumulate. Do not infer fairness from good occupancy. A scheduler can keep every slot busy while making one user wait disproportionately. [3]
Use three slots. Predict when request D starts under each policy. Then inspect the timeline and explain whether lower queue wait also implies a fair policy for multiple users.
4. Worked Python example
The core calculation below uses only the Python standard library. The full download adds the verification assertions and input fixtures. It runs locally on synthetic data; it does not load a model or contact a service.
from math import ceil
DURATIONS = [2, 6, 1, 4]
def static_time(slots):
return sum(max(DURATIONS[i:i + slots])
for i in range(0, len(DURATIONS), slots))
def refill_time(slots):
# FIFO list scheduling: admit the next request in the first free lane.
available = [0] * slots
for duration in DURATIONS:
lane = min(range(slots), key=lambda i: (available[i], i))
available[lane] += duration
return max(available)
def result(slots):
static, refill = static_time(slots), refill_time(slots)
return {"static": static, "refill": refill,
"static_occupancy": 13 / (slots * static),
"refill_occupancy": 13 / (slots * refill)}
Download full Python example and checks ↓
Run: python -I lab.py. Recorded run: Python 3.13; 4 fixture states plus scenario assertions passed. The JSON output is a reproducibility record, not a real-system benchmark.
5. Questions and explanations
Predict before opening the explanation. Answers refer to the stated scope and synthetic inputs.
References and source links
- Ouazan Reboul, R., Zucker, A., & Georges, L. (2025). Continuous batching from first principles.https://huggingface.co/blog/continuous_batching
- Hugging Face maintainers. (Main documentation; accessed 2026). Continuous batching · Hugging Face.https://huggingface.co/docs/transformers/main/continuous_batching
- TNG Technology Consulting authors. (2025). Efficient Request Queueing – Optimizing LLM Performance.https://huggingface.co/blog/tngtech/llm-performance-request-queueing
Source checks and provenance
Primary publisher pages and relevant passages checked on 13 Nov 2026. Source support is not independent expert fact verification. Main documentation can change; retrieved-body hashes, dates, version labels and saved revision IDs are in the audit download.
Evidence: Continuous batching section. Conceptual introduction read in the saved public original and current publisher page. No benchmark result is reused.
Public ResearchNode original 7b44225dc8d09a3a6793eaef130699be
Revision 93ad81780057f3e35e2e0c92235deb0e. Captured 2026-11-06.
Evidence: Continuous batching introduction. Current manager and request lifecycle documentation; the toy scheduler is not this implementation.
Evidence: Solution: Fair Scheduling; No Backpressure by Backend Queue. A first-party operational report, not a universal scheduling guarantee.
Public ResearchNode original 8f6d7dcacc39d11598a0bbeb01177459
Revision 9edc40898805d54aacf6d862a3bea560. Captured 2026-11-06.