{"sources":[{"id":"s1","url":"https://arxiv.org/pdf/1701.06538","title":"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer","publisher":"Shazeer et al.; arXiv","published_at":"2017-01-23","accessed_at":"2026-11-16","version":"arXiv:1701.06538v1","document_id":"","revision_id":"","limitations":"Original research; architecture and observed imbalance are not universal outcome guarantees."},{"id":"s2","url":"https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf","title":"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity","publisher":"Journal of Machine Learning Research","published_at":"2022-04","accessed_at":"2026-11-16","version":"JMLR 23, 2022, final PDF","document_id":"","revision_id":"","limitations":"Top-1 definitions and the reported capacity implementation. The text's uniform-minimum language is qualified using the explicit counterexample C17."},{"id":"s3","url":"https://arxiv.org/pdf/2202.09368","title":"Mixture-of-Experts with Expert Choice Routing","publisher":"Zhou et al.; arXiv","published_at":"2022-02-18","accessed_at":"2026-11-16","version":"arXiv:2202.09368v2, revised 2022-10-14","document_id":"","revision_id":"","limitations":"Fixed expert budgets balance slots, not necessarily token coverage; synthetic examples are separate."},{"id":"s4","url":"https://arxiv.org/html/2401.04088v1","title":"Mixtral of Experts","publisher":"Jiang et al.; arXiv","published_at":"2024-01-08","accessed_at":"2026-11-16","version":"arXiv:2401.04088v1","document_id":"","revision_id":"","limitations":"Rounded parameter counts are those in the report; the notebook ledger is not an exact reconstruction or benchmark."},{"id":"s5","url":"https://arxiv.org/html/2401.06066v1","title":"DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models","publisher":"Dai et al.; arXiv","published_at":"2024-01-11","accessed_at":"2026-11-16","version":"arXiv:2401.06066v1","document_id":"","revision_id":"","limitations":"Specialization and redundancy motivations are author-reported, not guarantees about semantic expert identities."},{"id":"s6","url":"https://arxiv.org/html/2412.19437v2","title":"DeepSeek-V3 Technical Report","publisher":"DeepSeek-AI; arXiv","published_at":"2024-12-27","accessed_at":"2026-11-16","version":"arXiv:2412.19437v2, revised 2025-02-18","document_id":"","revision_id":"","limitations":"Method-specific configuration and self-reported systems design; no independent benchmark reproduction."},{"id":"s7","url":"https://arxiv.org/pdf/2211.15841","title":"MegaBlocks: Efficient Sparse Training with Mixture-of-Experts","publisher":"Gale et al.; arXiv","published_at":"2022-11-29","accessed_at":"2026-11-16","version":"arXiv:2211.15841v1 as identified in the accessed PDF","document_id":"","revision_id":"","limitations":"Block-sparse training system; not evidence that every Mixtral deployment uses identical kernels or is free of padding overhead."}],"claims":[{"id":"C1","text":"Sparse gates permit skipping zero-gated expert evaluation and expanding the stored bank independently of selected expert count.","location":"conditional-computation; questions Q1","evidence":[{"source_id":"s1","passage":"We save computation based on the sparsity","location":"Section 2, equation 1 and following paragraph"}],"limitations":"Applies to expert evaluation; other computation and storage remain.","kind":"source","basis":"Read full-text section 2."},{"id":"C2","text":"Mixtral selects two of eight FFN experts, uses selected-logit softmax, and reports rounded 47B total and 13B active parameters.","location":"conditional-computation; router-mechanics; architecture table","evidence":[{"source_id":"s4","passage":"only uses 13B active parameters during inference","location":"Abstract; section 2.1 equation and Table 1"},{"source_id":"s4","passage":"taking the softmax over the Top-K logits","location":"Section 2.1"}],"limitations":"Model-specific; counts are rounded; synthetic ledger is separate.","kind":"source","basis":"Read abstract, architecture table and section 2.1."},{"id":"C3","text":"Noisy top-k adds Gaussian noise scaled by softplus of a learned projection before sparse softmax; selected gates have derivatives for k greater than one.","location":"router-mechanics; architecture table; lab.py noisy_gate","evidence":[{"source_id":"s1","passage":"we add tunable Gaussian noise","location":"Section 2.1, equations 3–5 and Training the Gating Network"}],"limitations":"Not a universal router definition; lab substitutes synthetic projections.","kind":"source","basis":"Read equations 3–5 and following gradient discussion."},{"id":"C4","text":"Switch top-1 keeps the selected probability from a full-expert softmax, unlike a one-entry renormalized softmax.","location":"router-mechanics; architecture table","evidence":[{"source_id":"s2","passage":"route to only a single expert","location":"Section 2.1, equations 1–2; Figure 15 router pseudocode"}],"limitations":"The one-entry comparison follows directly from softmax arithmetic.","kind":"source","basis":"Read section 2.1 and top-1 gate extraction in Figure 15."},{"id":"C5","text":"Uneven expert selection can reinforce uneven training; the original work uses importance and load balancing objectives.","location":"balancing; questions Q3","evidence":[{"source_id":"s1","passage":"This imbalance is self-reinforcing","location":"Section 4, opening paragraph; importance loss and Appendix A"}],"limitations":"Observed failure mode, not inevitability.","kind":"source","basis":"Read full-text imbalance passage and objective discussion."},{"id":"C6","text":"Switch defines alpha times E times the dot product of hard top-1 fractions and average full router probabilities; the latter supplies derivatives.","location":"balancing; questions Q2; code-balance; lab.py switch_top1","evidence":[{"source_id":"s2","passage":"the P-vector is differentiable","location":"Section 2.2, equations 4–6"}],"limitations":"Top-1 definition; no assertion of alpha as a global lower bound.","kind":"source","basis":"Read equations 4–6 and gradient explanation."},{"id":"C7","text":"Switch's fixed expert capacity scales with T/E and capacity factor; overflow skips expert computation with residual bypass.","location":"capacity-architectures; architecture table","evidence":[{"source_id":"s2","passage":"through the residual connection","location":"Section 2.2, equation 3 and following overflow paragraph"}],"limitations":"Specific capacity-limited implementation; rounding is implementation-dependent.","kind":"source","basis":"Read equation 3 and overflow behavior."},{"id":"C8","text":"MegaBlocks uses block-sparse computation to handle variable expert assignments without dropping tokens.","location":"capacity-architectures; expert-parallelism","evidence":[{"source_id":"s7","passage":"expressed as block-sparse operations to accommodate imbalanced assignment of tokens to experts","location":"Introduction contributions; sections 2–3"}],"limitations":"Does not imply zero padding, balanced work or universal speedup.","kind":"source","basis":"Read introduction and routing/permutation description."},{"id":"C9","text":"Expert Choice selects a fixed token budget per expert, allowing variable expert participation per token.","location":"capacity-architectures; architecture table; questions Q4; lab.py expert_choice","evidence":[{"source_id":"s3","passage":"tokens can be received by a variable number of experts","location":"Section 3.2, equations 1–2"}],"limitations":"No universal token-coverage guarantee follows from equal expert budgets.","kind":"source","basis":"Read section 3.2 and transposed affinity top-k equations."},{"id":"C10","text":"DeepSeekMoE combines fine-grained FFN segmentation with always-active shared experts to target reduced redundancy.","location":"capacity-architectures; architecture table; questions Q5","evidence":[{"source_id":"s5","passage":"splitting the FFN intermediate hidden dimension","location":"Introduction architecture description; section 3.1"},{"source_id":"s5","passage":"shared experts that are always activated","location":"Introduction architecture description; section 3.2"}],"limitations":"Design motivation and reported experiments, not universal semantic specialization.","kind":"source","basis":"Read architecture description in the primary full text."},{"id":"C11","text":"DeepSeek-V3 uses sigmoid affinities, bias-only selection adjustments, normalized unbiased gates, complementary sequence balancing, one shared expert, eight of 256 routed experts and four-node routing limits.","location":"deepseek-balancing; architecture table; questions Q6","evidence":[{"source_id":"s6","passage":"bias term is only used for routing","location":"Section 2.1.2, equations 12–20"},{"source_id":"s6","passage":"complementary sequence-wise balance loss","location":"Section 2.1.2, equations 17–20"},{"source_id":"s6","passage":"1 shared expert and 256 routed experts","location":"Section 4.2 Model Hyper-Parameters"}],"limitations":"Specific to report v2; lab omits node restrictions and the sequence loss.","kind":"source","basis":"Read section 2.1.2 and section 4.2."},{"id":"C12","text":"Expert-parallel execution has dispatch and combine communication phases, which can use point-to-point transfers rather than two mandated API calls.","location":"expert-parallelism; questions Q7","evidence":[{"source_id":"s6","passage":"direct point-to-point transfers","location":"Section 3.4.2; section 3.2.1 dispatch/MLP/combine decomposition"}],"limitations":"Logical communication pattern; shared-layer partitioning and physical collectives vary.","kind":"source","basis":"Read deployment and training communication passages."},{"id":"C14","text":"Synthetic parameter formulas yield 46,433,042,432 stored and 12,610,174,976 selected-path parameters; their ratio is about 27.16%, not a measured compute ratio.","location":"conditional-computation, table and scope; questions Q1; lab.py parameters","evidence":[],"limitations":"Stipulated S,F,B,L,E,K; router/norm counts excluded unless inside S; no runtime model.","kind":"original_example","basis":"Inspect lab.py parameters with L=32,S=33554432,F=176160768,B=262144000,E=8,K=2. JavaScript arithmetic in the author tool record returned total=46433042432, path=12610174976, ratio=0.2715776161871641; LF=5637144576."},{"id":"C15","text":"The four-logit example selects experts 0 and 3 with weights 0.75 and 0.25 and combines [2,-1] and [-2,3] into [1,0]; routing diagram arrows represent these dependencies.","location":"router-mechanics; routing.drawio; lab.py main","evidence":[],"limitations":"Synthetic outputs, not learned expert behavior; diagram omits surrounding Transformer components.","kind":"original_example","basis":"Inputs [ln(3),-2,-3,0]; selected exponentials [3,1]; coordinate-wise arithmetic and inspectable lab assertions. XML created using drawio create_diagram."},{"id":"C16","text":"The illustrative lab separates noisy top-2 routing from top-1 balancing and explicitly calculates selection rates, assignment shares and invariant checks.","location":"router-mechanics; balancing; routing-lab; code-balance; lab.py","evidence":[],"limitations":"Unexecuted illustrative Python; synthetic logits/projections; no learning or real dispatch.","kind":"original_example","basis":"Inspect lab.py softmax, softplus, sparse_gate, noisy_gate, assignment_counts and main. Four top-2 rows produce eight assignments; division by T versus TK changes normalization."},{"id":"C17","text":"Uniform f and P give alpha, but the ten-token counterexample gives 0.9344 alpha, disproving a universal alpha lower bound.","location":"balancing; questions Q2; code-balance; lab.py switch_top1","evidence":[],"limitations":"Boundary probability zero can be replaced by a sufficiently small positive value; not an optimization convergence analysis.","kind":"original_example","basis":"Premise is the equation audited under C6. Nine [0.51,0.49] rows and one [0,1] give f=[0.9,0.1], P=[0.459,0.541]; 2*(0.9*0.459+0.1*0.541)=0.9344. Author JavaScript arithmetic agrees; lab asserts loss=0.009344."},{"id":"C18","text":"Synthetic top-2 capacity two rejects one assignment despite enough aggregate slots; capacity three admits all with four unused slots.","location":"capacity-architectures; routing-lab; lab.py admit","evidence":[],"limitations":"First-come assignment admission; no rerouting or survivor renormalization; not a kernel benchmark.","kind":"original_example","basis":"Routes [[0,3],[0,1],[0,2],[1,2]] yield [3,2,2,1]. Capacity=2 gives [2,2,2,1], rejecting token 2/expert 0. Capacity=3 allocates 12 slots for 8 assignments."},{"id":"C19","text":"Equal Expert Choice budgets do not guarantee coverage; the three-token affinity example gives participation [2,1,1] or [0,1,1].","location":"capacity-architectures; questions Q4; lab.py expert_choice","evidence":[],"limitations":"Synthetic affinity matrix, not model data.","kind":"original_example","basis":"Column-wise ranking of [[0.5,0.5],[0.6,0.4],[0.4,0.6]] with two slots selects [1,0] and [2,0]; one slot selects [1] and [2]."},{"id":"C20","text":"The synthetic coarse/fine comparison gives 28 versus 1820 possible subsets at equal idealized selected width.","location":"capacity-architectures; questions Q5; lab.py main","evidence":[],"limitations":"Combinatorial possibilities do not measure useful specialization or runtime.","kind":"original_example","basis":"8 choose 2 = 28; 16 choose 4 = 1820; four half-width experts equal two coarse-width experts under the stipulated linear-width parameter model."},{"id":"C21","text":"The synthetic bias example selects experts 2 and 1 while weighting them by their unbiased affinities; sign updates depend on current overload.","location":"deepseek-balancing; questions Q6; lab.py biased_gate and update_bias","evidence":[],"limitations":"Only arithmetic inspired by C11; excludes full training, node constraints and sequence loss.","kind":"original_example","basis":"[0.8,0.7,0.6]+[-0.3,0,0.2]=[0.5,0.7,0.8]; normalize selected original [0.6,0.7]. Counts [6,1,1], mean 8/3 and gamma .01 yield [-.01,.01,.01]."},{"id":"C22","text":"Synthetic placement produces three remote assignments, five assignments on rank A and three on rank B, with 49,152 input-plus-output payload bytes under the stated assumptions.","location":"expert-parallelism; parallel.drawio; routing-lab; questions Q7; lab.py communication","evidence":[],"limitations":"No metadata, padding, protocol or backward traffic; no shared-copy optimization; diagram shows only token 0.","kind":"original_example","basis":"Routes [[0,3],[0,1],[0,2],[1,2]], origins [0,0,1,1], placement [0,0,1,1]. Remote pairs [(0,3),(2,0),(3,1)]. Payload=2*3*4096*2=49152. Editable diagram created with drawio tool."},{"id":"C23","text":"In a weight-only matrix model, intensity is 2n/b FLOPs per byte; this does not establish universal MoE latency or superiority over dense decoding.","location":"expert-parallelism; questions Q8","evidence":[],"limitations":"Assumes one read per weight and multiply-add counted as two FLOPs; excludes all other traffic, caching and timing.","kind":"original_example","basis":"For P weights and n vectors, stipulated FLOPs=2nP and bytes=Pb, so their ratio is 2n/b. At b=2, n=1 gives 1 and n=32 gives 32."},{"id":"C24","text":"The source disclosure describes public discovery, full-text primary reading, synthetic arithmetic and illustrative code without asserting independent verification.","location":"source-checks; bibliography","evidence":[],"limitations":"No execution, browser rendering or independent expert review is represented as completed.","kind":"process","basis":"Tool record: ResearchNode queries mixture experts, Switch and Sparsely-Gated; read_document d45a50e4558fdad28a4c4f8aa310a92d revision 13f224d12636fe6a9c75387efb5a8c59 used only as discovery. Live web opens/finds read the seven listed primary full texts. drawio get_drawio_reference and create_diagram calls produced both editable diagrams; duplicate IDs in an initial placement diagram were corrected."}]}