BubbleTea fills MoE inference idle time with fine-tuning work
An arXiv paper describes a colocation system that predicts idle windows in expert-parallel MoE serving and reuses them for fine-tuning, reporting higher training throughput than prior systems.
In plain words
When large language models split their Mixture-of-Experts parts across GPUs, some GPUs sit idle waiting for busier ones. BubbleTea detects those idle windows and uses them to run fine-tuning on the same GPUs, reusing weights already loaded for serving. The authors report faster training than existing colocation systems while keeping most inference latency close to serving alone.
What was announced
The paper argues that expert parallelism creates a new source of idle time: because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one, and such bubbles still take 8–13% of prefill time even after load balancing. BubbleTea, described as the first colocation system to exploit these expert-parallel bubbles, predicts each GPU's bubble at every layer from the router's token assignment and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. Compiler passes derive these chunks automatically from the served model, without porting each architecture to a separate training engine.
In the authors' evaluation on three MoE models served on four A100 GPUs, fine-tuning throughput was 1.9–6.2× faster than LLMStation, a state-of-the-art colocation system, and p90 inference latency stayed within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, BubbleTea trained 3.3× faster than LLMStation and 14× faster than a stock trainer colocated under CUDA MPS, with p90 latency within 7% of serving alone. These are the authors' own measurements and have not been independently confirmed.
What it does not show
- The evidence is the paper's abstract only; full experimental details, baselines and setup are not available for verification.
- All throughput and latency figures come from the authors' own evaluation on four A100 GPUs and three MoE models; no independent confirmation is provided.
- The abstract does not state whether BubbleTea is available as a deployed or downloadable system.
For security and IT teams
- Scope
- The reported results cover three MoE models served on four A100 GPUs, so the figures may not transfer to other hardware or model configurations.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.