Conditional computation and parameter accounting
A sparse Mixture-of-Experts layer replaces one feed-forward computation with a bank of expert networks and a router. For token representation x, its output is y = Σᵢ gᵢ(x)Eᵢ(x). Experts with zero gates need not be evaluated for that token. Increasing the expert bank can therefore increase stored parameters without evaluating every expert on every token. This is conditional computation, not weight compression. [1]
Mixtral provides a concrete architecture: eight feed-forward experts per layer, with two selected for each token. Its report describes approximately 47 billion total parameters and 13 billion active parameters. Shared Transformer components mean that multiplying the model name’s 7B by eight is not valid parameter accounting. [4]
Scope and limits: this guide examines routing and its systems consequences, not model quality rankings. The arithmetic below uses explicit synthetic assumptions. Active-parameter counts are bookkeeping estimates, not measurements of FLOPs, memory consumption or latency. The lab contains no model execution, training, accelerator kernels or network communication.
Define L as the number of identically configured MoE layers, S as shared parameters per layer, F as parameters per expert, E as experts per layer, K as selected experts and B as a separately counted embedding/output allowance. Then N_total = B + L(S + EF), while the corresponding selected-path estimate is N_path = B + L(S + KF). Both expressions have units of parameters. The second deliberately includes the entire B allowance; it does not assert that every embedding-table entry participates in one token’s computation.
| Quantity | Assumption or result |
|---|---|
| L, E, K | 32 layers, 8 experts, 2 selected |
| S | 33,554,432 shared parameters per layer |
| F | 176,160,768 parameters per expert |
| B | 262,144,000 embedding/output parameters |
| N_total | 46,433,042,432 parameters |
| N_path | 12,610,174,976 parameters; approximately 27.16% of N_total |
Here S is a stipulated shared-layer allowance, not a reconstructed attention configuration. Router and normalization parameters are excluded unless explicitly absorbed into S; this example does not add them separately. Optimizer states, gradients, activations, KV caches, communication buffers and quantization metadata are outside both formulas. The counts reproduce approximately 46.43B and 12.61B, but the percentage must be called a selected-path parameter fraction, not an active-compute percentage.
The ledger also makes a useful sensitivity calculation possible. Adding one expert to every layer adds LF = 5,637,144,576 stored parameters. With K fixed, it adds no expert parameters to this selected-path estimate. Increasing K by one instead adds that same amount to N_path while leaving N_total fixed. Neither operation has a latency prediction attached: the formulas contain no device, precision, batch size or interconnect variable.
Top-k selection, normalization and noisy routing
Mixtral describes gates obtained by retaining the top-k linear logits, masking the others to negative infinity, and applying softmax. Selected weights consequently sum to one. [4] Shazeer et al.’s noisy variant first forms Hᵢ(x) = (xW_g)ᵢ + εᵢ softplus((xW_noise)ᵢ), with standard-normal noise, then applies that sparse normalization. Its selected gate values provide derivatives when k exceeds one; discrete expert identities themselves are not ordinary differentiable outputs. [1]
Switch uses a different convention: select one expert but retain its probability from a softmax over all experts. Renormalizing a single selected logit would instead give a constant weight of one. These operations are not interchangeable router implementations. [2]
Download editable routing diagram. Every purple arrow denotes a data dependency: x reaches the router, the selected branches receive x, and their output vectors reach the sum. The grey bank represents unselected experts. Attention, normalization and the surrounding residual connection are omitted; the picture depicts only the MoE sublayer.
For a hand-checkable example, choose four logits [ln 3, −2, −3, 0] and K = 2. Experts 0 and 3 win. Exponentiating just those logits gives [3, 1], hence weights [0.75, 0.25]. Assume their output vectors are [2, −1] and [−2, 3]. The combined vector is 0.75[2, −1] + 0.25[−2, 3] = [1, 0]. The gate weights are scalar multipliers; they do not select coordinates inside either vector.
Now compare normalization orders. A full four-way softmax allocates a positive probability to the two losing experts. Keeping only the winning probabilities without renormalizing would make their sum less than one. Dividing those retained probabilities by their own sum recovers [0.75, 0.25]. The example distinguishes three operations that are easy to conflate: choosing identities, calculating probabilities and deciding which probability normalization weights the outputs.
The lab adds a separate seeded noisy run. Its noise projections are synthetic inputs standing in for xW_noise; they are not learned weights. A deterministic example remains available with noise disabled. Sorting breaks exact ties by expert index, making that implementation choice explicit. Noise is not a proof of balanced dispatch, and a single noisy run is not a training experiment.
Dispatch imbalance and auxiliary objectives
Shazeer et al. report a self-reinforcing imbalance: favored experts receive more training and are selected more often. This motivates balancing mechanisms; it does not establish that every unregularized router must collapse. Their original approach includes importance and load objectives. [1]
For the separately defined Switch top-1 case, let pᵢ(x) be the full router probability, fᵢ the fraction of tokens whose argmax is i, and Pᵢ the batch mean of pᵢ(x). The balancing term is L_aux = αEΣᵢfᵢPᵢ. Gradients flow through P, while f is a discrete count. [2]
Uniform fᵢ = Pᵢ = 1/E gives L_aux = α. That is a reference value, not a universal lower bound. Consider ten tokens and two experts. Nine probability vectors are [0.51, 0.49]; the tenth is [0, 1]. Argmax counts give f = [0.9, 0.1], while averaging probabilities gives P = [0.459, 0.541]. Thus EΣfᵢPᵢ = 2(0.9 × 0.459 + 0.1 × 0.541) = 0.9344. With α = 0.01, the loss is 0.009344, below 0.01.
The zero probability is a convenient boundary case rather than the output of a finite-logit softmax. Replacing it with a sufficiently small positive probability preserves the inequality. The example exposes why the loss scalar cannot certify uniform assignments: the hard counts and average confidence need not move together. Nine narrow wins can coexist with one very confident loss.
The proposed diagnostic is therefore to inspect counts and confidence separately. In this example, expert 0 gets nine assignments despite its mean probability being below one half. A reader who saw only the mean probabilities might infer nearly balanced demand. A reader who saw only the auxiliary value might mistakenly conclude that the distribution was better than uniform. Both interpretations disappear once the actual assignments are printed.
Top-2 requires its own counting convention. In the lab’s four-token example, expert counts sum to eight assignments. Dividing counts by T gives per-token selection rates whose sum is two; dividing by TK gives assignment shares whose sum is one. Neither vector is the top-1 f used above. The lab does not feed noisy top-2 counts into a function labeled as Switch balancing.
Capacity limits and architectural alternatives
Switch allocates expert capacity proportional to (T/E) × capacity factor. Overflow can skip the expert computation while the residual path carries the representation onward. [2] MegaBlocks instead formulates expert computation with block-sparse operations to support dropless execution. This addresses the tradeoff between discarded assignments and fixed-capacity padding; it does not make arbitrary load imbalance free. [7]
| Method | Assignment rule | Distinction |
|---|---|---|
| Sparsely-gated MoE | Noisy top-k | Learned noise scale; sparse softmax [1] |
| Switch | One expert per token | Full-softmax gate; capacity-limited implementation [2] |
| Expert Choice | Each expert selects a fixed number of tokens | Equal slot counts; variable selections per token [3] |
| Mixtral | Two of eight experts | Selected-logit normalization [4] |
| DeepSeekMoE | Smaller routed experts plus shared experts | Fine-grained segmentation and shared computation [5] |
| DeepSeek-V3 | Eight of 256 routed experts plus one shared expert | Bias-adjusted selection and node limits [6] |
To make capacity concrete, use synthetic selections [0,3], [0,1], [0,2], [1,2]. The expert counts are [3,2,2,1]. A teaching capacity C = ceil(TK/E) = 2 gives eight slots overall, exactly matching eight requested assignments, yet expert 0 exceeds its two slots. One assignment is rejected while expert 3 has an unused slot. Aggregate free space is not interchangeable with capacity at the selected destination.
The lab’s rejection policy keeps the earliest assignments in input order. It does not reroute or renormalize survivors. Increasing C to three admits all eight assignments but allocates twelve slots, four of them unused. A dropless count keeps all eight requested assignments without imposing this per-expert cap. These are allocation calculations, not a sparse-kernel implementation or measured throughput comparison.
Expert Choice reverses the selection direction: each expert chooses its highest-affinity tokens up to a fixed budget. Equal expert counts do not imply every token has an assignment. [3] For example, with three tokens and two experts, let affinity rows be [0.5,0.5], [0.6,0.4], [0.4,0.6]. If each expert selects two tokens, counts are exactly [2,2], but token participation is [2,1,1]. If each selects only one, participation becomes [0,1,1]. Uniform expert occupancy and token coverage answer different questions.
DeepSeekMoE splits the FFN intermediate dimension into smaller experts and includes always-active shared experts, with the stated goal of reducing duplicated common knowledge. This is an architectural motivation supported by the authors’ experiments, not a guarantee that each expert acquires a clean human-labeled specialty. [5]
A synthetic segmentation comparison illustrates the combinatorics. Selecting two of eight coarse experts permits 28 subsets. Halving each expert’s width, doubling the pool to sixteen and selecting four permits 1,820 subsets. Under an idealized FFN count proportional to intermediate width, both choices select two coarse-width equivalents. The larger combination count is arithmetic; it does not measure useful specialization, quality or runtime.
DeepSeek-V3 selection biases and sequence balancing
DeepSeek-V3 uses sigmoid affinities. A per-expert bias affects top-k selection, but output gates normalize the selected unbiased affinities. Biases decrease for overloaded experts and increase for underloaded experts. The report also retains a small sequence-wise balancing loss; auxiliary-loss-free therefore does not mean that all auxiliary balancing terms disappear. Its stated configuration uses one shared expert, eight selected routed experts out of 256, and a maximum of four destination nodes per token. [6]
A small synthetic calculation separates selection from weighting. Let affinities be [0.8,0.7,0.6], biases [−0.3,0,0.2] and K = 2. Adjusted selection scores are [0.5,0.7,0.8], so experts 2 and 1 win. Their output weights are 0.6/1.3 and 0.7/1.3, approximately 0.4615 and 0.5385. Using the biased values would produce 0.8/1.5 and 0.7/1.5 instead, reversing which selected expert has greater output weight.
For a separate update illustration, suppose counts are [6,1,1] and the target is their mean, 8/3. With γ = 0.01 and initially zero biases, the next biases become [−0.01,0.01,0.01]. This sign update discourages the overloaded destination on the next selection. It does not retrospectively change current assignments. The lab intentionally models only this local arithmetic, not the report’s full training procedure, node constraints or sequence loss.
Expert placement, dispatch and combine
Expert parallelism distributes expert computation across devices. The DeepSeek-V3 report describes all-to-all dispatch and combine phases, including a deployment implemented with direct point-to-point transfers. These are logical communication phases, not a requirement that every system issue exactly two particular collective API calls. Shared-layer partitioning is a separate deployment choice. [6]
Download editable placement diagram. The two containers are synthetic device boundaries. The downward local arrows carry x into expert 0 and its output into the combiner. The outward cross-boundary arrow dispatches x to expert 3; the returning arrow carries that expert’s result to rank A. Routing metadata and gates must remain associated with the token so the combiner can restore order and weights. Other tokens, unused experts and backward communication are omitted.
Place experts [0,1] on rank A and [2,3] on rank B. Let the four tokens from the capacity example originate on ranks [A,A,B,B]. Requested expert work totals five assignments on A and three on B. Three assignments cross a device boundary, while the remaining five stay local. The next paragraph enumerates the three remote pairs.
Explicitly enumerating the remote pairs gives (token 0, expert 3), (token 2, expert 0), and (token 3, expert 1). For hidden width d and b bytes per element, separately transferring a full input and output for each remote assignment gives 2 × 3 × d × b payload bytes. With d = 4,096 and b = 2, that is 49,152 bytes. This excludes metadata, protocol overhead, padding and backward traffic, and assumes no shared-copy optimization.
After dispatch, tokens can be grouped by expert for matrix computation; MegaBlocks supplies a block-sparse implementation of this variable-work problem. [7] A simplified weight-traffic model explains why batch composition matters. A matrix with P weights used for n vectors costs roughly 2nP multiply-add FLOPs and, assuming one weight read, Pb weight bytes. Ignoring all other traffic gives intensity 2n/b FLOPs per byte: at two-byte precision, n = 1 gives one FLOP per byte and n = 32 gives 32. This is an idealized arithmetic ratio, not an observed accelerator result.
Sparse routing can split a batch across several matrices, reducing n for each one. However, an active-parameter-matched dense matrix-vector operation can also have low reuse. Storage of inactive experts alone does not prove that every weight is streamed each step or that MoE is always slower. A real comparison needs expert occupancy, caching, precision, kernel shapes and communication timing. Larger batches may improve reuse while also changing request latency; this guide makes no universal serving guarantee.
Illustrative Python lab and interpretation
The original standard-library-only lab.py is illustrative and uses synthetic inputs. It implements stable softmax, softplus-scaled noise, assignment counts, capacity admission, a separate top-1 loss calculation, expert-choice counts, bias arithmetic, communication payload accounting and parameter estimates. It contains assertions but no claimed execution transcript.
def switch_top1(probabilities, alpha=0.01):
# Illustrative: full probabilities, not sparse top-2 gates.
tokens = len(probabilities)
experts = len(probabilities[0])
counts = [0] * experts
for row in probabilities:
winner = max(range(experts), key=lambda i: row[i])
counts[winner] += 1
fractions = [count / tokens for count in counts]
means = [sum(row[i] for row in probabilities) / tokens
for i in range(experts)]
loss = alpha * experts * sum(
f * p for f, p in zip(fractions, means)
)
return fractions, means, loss
rows = [[0.51, 0.49] for _ in range(9)] + [[0.0, 1.0]]
f, p, loss = switch_top1(rows)
# Analytical expectation: loss is 0.009344, below alpha.
Work through the deterministic run before inspecting the noisy run. First verify that every top-2 gate sums to one. Then verify that counts sum to TK, rather than T. Next compare requested counts with admitted counts under capacity two. Finally inspect the top-1 counterexample separately. These invariants test different concepts; a correct total assignment count cannot establish balanced expert occupancy.
For an extension, change the origin ranks without changing logits. Expert selections and parameter totals should stay the same while remote assignment counts can change. Alternatively, change capacity without changing the requested routes: admitted work changes, but requested counts do not. These controlled edits separate mathematical routing, admission policy and physical placement.
Eight explanatory questions
Q1 · CapacityWhat does sparse conditional computation decouple?
Stored expert parameters and the expert subset evaluated for one token. In the synthetic ledger, changing E changes storage while fixed K preserves the selected expert count. That does not fix total runtime. [1]
Q2 · Balance lossWhy multiply dispatch fractions by mean probabilities?
The discrete fractions weight the differentiable probability term. The ten-token example also shows why α is a uniform reference, not a universal minimum or certificate of balance. [2]
Q3 · CollapseWhat failure motivates balancing mechanisms?
Concentration on favored experts can reinforce itself through uneven training. It is a documented failure mode, not an inevitable outcome for every router. [1]
Q4 · Expert ChoiceWhat is balanced when experts choose tokens?
Fixed expert budgets balance assignment counts. They do not guarantee token coverage: the three-token example can leave token 0 unselected. [3]
Q5 · Fine-grained expertsWhat distinguishes segmentation and shared experts?
Segmentation creates smaller routed units; shared experts always contribute common computation. More possible subsets do not establish semantic specialization by themselves. [5]
Q6 · Bias balancingDoes DeepSeek-V3 eliminate every auxiliary balance loss?
No. Selection biases are distinct from unbiased output affinities, and a complementary sequence-wise loss remains. The worked affinity example separates selection rank from output weight. [6]
Q7 · CommunicationWhat do dispatch and combine exchange?
Dispatch sends representations to selected remote experts; combine returns results for reconstruction at the origin. Local assignments need no cross-device transfer in the illustrated placement. [6]
Q8 · Weight reuseWhy can small decoding batches have low arithmetic intensity?
Few vectors reuse each loaded expert matrix. In the simplified model, intensity grows with vectors per expert. This also applies to dense matrix-vector work, so it does not prove MoE is universally slower than an active-FLOP-matched dense model.
References and source links
Primary research
- Shazeer et al. (2017), arXiv v1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
https://arxiv.org/pdf/1701.06538 - Fedus, Zoph and Shazeer (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 23.
https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf - Zhou et al. (2022), arXiv v2. Mixture-of-Experts with Expert Choice Routing.
https://arxiv.org/pdf/2202.09368 - Jiang et al. (2024), arXiv v1. Mixtral of Experts.
https://arxiv.org/html/2401.04088v1 - Dai et al. (2024), arXiv v1. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models.
https://arxiv.org/html/2401.06066v1 - DeepSeek-AI (2025 revision, v2; first submitted 2024). DeepSeek-V3 Technical Report.
https://arxiv.org/html/2412.19437v2 - Gale et al. (2022 preprint, v1). MegaBlocks: Efficient Sparse Training with Mixture-of-Experts.
https://arxiv.org/pdf/2211.15841
Source-check scope and limitations
Evidence methodology and example boundaries
Evidence distinguishes original research descriptions from synthetic calculations. Router equations are checked against their own method rather than treated as universal conventions. The claim ledger identifies passages, equations and example inputs. Model-specific implementation reports describe the authors’ systems; they are not independent performance verification.
Full primary texts listed above were accessed on 16 November 2026. Public library searches and a saved explanatory article were used for discovery; that secondary article is not evidence for the technical claims. All Python is original illustrative code. Numerical examples are analytical expectations, not measured model outputs. The diagrams describe the explicitly stated synthetic assignments and placement, with omitted components identified in their text alternatives.