← News

Models & toolsSources: arXiv · 19 Nov 2026

DUME builds Mixture-of-Experts models from dense experts without extra training

The DUME method reuses dense experts trained on separate domains to assemble one multitask Mixture-of-Experts model, with no additional training required, according to the paper's abstract.

In plain words

A research team describes DUME, a way to combine several already-trained specialist language models into one larger model that can handle all their tasks. It does this without the usual extra training step, using a closed-form statistical solution instead of optimization.

What was announced

The authors frame the problem as follows: finetuning large language models for specific domains is expensive and can cause overspecialization, while multitask training risks interference and catastrophic forgetting, where a model loses earlier skills when learning new ones. Prior work combined dense experts inside a Mixture-of-Experts architecture, but that still required multitask finetuning. DUME instead reuses dense experts trained on different domains and assembles them into a unified MoE model that preserves their original capabilities without further training.

The method relies on the closed-form solution of ridge regression, which the authors say removes the need for further optimization and lets experts be added or removed dynamically. In their experiments across causal language modeling and reasoning tasks, with model sizes from 115M to 9B parameters, they report that DUME consistently beat baseline approaches. In reasoning settings, they claim it reached 104% of the dense expert's performance on average, and that it can be finetuned to improve complex reasoning further. Code is published on GitHub under the Gensyn AI organization.

What it does not show

  • The evidence is the paper's abstract only; the full article's methods, experimental setup and results were not reviewed, so the reported figures cannot be independently verified.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.