TEMPO: Expert-Parallel Load Balancing That Sees Both GPU Regimes

Serving Mixture-of-Experts (MoE) models in expert-parallel (EP) mode has a persistent problem: every layer synchronizes at the slowest GPU. One overloaded expert and the whole batch waits at the barrier. Dispatchers are supposed to fix this by distributing work across GPUs – except that all popular heuristics balance a single quantity: token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), quietly assuming expert time is linear in one of them.

The KlingAI Research (Kuaishou) team measured this on two datacenter GPU generations and shows in TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes (arXiv:2608.13057) that both assumptions are false. Expert time in decode is linear in neither tokens nor experts – and that changes how dispatch should work. On recorded batches, the choice of heuristic changes modeled block time by 1.4-1.6x (p95 up to 1.7x), and which heuristic wins depends on the GPU regime.

Two regimes, one threshold: 156-168 tokens

The authors measured expert time as a function of token count and the number of activated replicas (replicas are copies of the same expert spread across GPUs). Result: two regimes with a sharp threshold n* of about 156-168 tokens:

• Below the threshold – memory-bound regime. HBM weight streaming dominates. Execution cost attaches to the number of activated replicas: each replica reads the expert’s full weights from memory, regardless of how many tokens reach it. Token-count balancing is pointless here because cost is fixed per replica.
• Above the threshold – compute-bound regime. Grouped GEMM on tensor cores rounds the M dimension (tokens) to 128-tile chunks. Splitting an expert across more GPUs adds padding: every 128-tile must be filled, so sharding an expert in two can cost more than keeping it in one place.

Both regimes are captured by a single max-affine profile: t = max(a + bG, c + betaN), where G is the number of activated replicas and N the token count. A simple model, but it does the job: for a given batch you know which side of the threshold you are on.

Why existing heuristics fail

Real decode batches hold hot experts (many tokens, linear regime) and cold experts (few tokens, flat regime) simultaneously. A token-balancing heuristic works great on hot experts and hurts cold ones; a replica-balancing heuristic does the opposite. That is why proxy decisions diverge: on recorded batches, different heuristics differ by 1.4-1.6x in modeled block time.

Dispatch as a makespan problem

TEMPO formalizes per-batch dispatch as a fixed-charge makespan problem: each expert has an activation cost (fixed) plus a token-dependent cost. It is NP-hard even on two fully replicated GPUs, polynomial in degenerate limits, with an additive approximation guarantee under full replication. In practice the dispatcher solves an instance in milliseconds, off the critical path – it adds no latency.

The SGLang integration runs out-of-process and fuses dispatch with count collection into one in-graph kernel, so overhead is minimal.

Results: a phase diagram, not a universal win

On the microbenchmark (8 GPUs, Testbed A) TEMPO stays within 1% of the best fixed baseline everywhere and wins by up to 15.5% where regimes mix. End-to-end on Testbed B:

• Qwen3-235B (model inside the win region): +4-6% throughput, p99 latency down by ~15.6%;
• DeepSeek-V3 (outside, communication-dominated): mechanism cost only, no gains.

That is an honest framing: the authors do not sell TEMPO as a panacea. They deliver a phase diagram that predicts before deployment whether your model and batch profile land in the win region. If you serve a large MoE model with hot and cold experts (regimes mix every day), this is worth testing.

What it means for an engineer

• The assumption “expert time is linear in tokens” is simply false on modern GPUs. Any custom dispatcher or balancer should account for the ~160-token threshold.
• Expert replication has a hidden cost in the memory-bound regime: every replica reads the full weights. An argument against over-replicating cold experts.
• Grouped GEMM with 128-tiles means sharding an expert across GPUs only pays off above the threshold – below it you only add padding.
• SGLang is a mature integration point: an out-of-process dispatcher with kernel fusion is a pattern you can replicate in other stacks (vLLM, TensorRT-LLM).

This content was generated using artificial intelligence and reviewed by a human editor.

Sources

• TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes – arXiv:2608.13057 (KlingAI Research, published 2026-08-13)
• China AI Weekly Brief, period 2026-08-09 to 2026-08-16 (job 3d03e2d4fae9)

---

This article is also available in Polish (PL).