← News

Models & toolsSources: arXiv · 19 Nov 2026

FlashMorph2 picks hybrid attention layers on-policy for faster conversion

A new arXiv paper proposes an on-policy method for choosing which layers keep full attention when converting pretrained Transformers into hybrid attention models, reporting gains over FlashMorph.

In plain words

Hybrid attention models mix two ways of processing sequences: costly full attention and cheaper linear attention. Converting an existing model means deciding which layers keep the expensive version. FlashMorph2 makes that choice while training the model in its final form, and the authors report better results than earlier selection methods.

What was announced

Hybrid attention models combine full and linear attention to balance output quality against computational cost, and converting a pretrained Transformer into this form requires deciding which layers retain full attention within a set budget. The authors identify two mismatches in the earlier FlashMorph approach: its selection relies on continuous branch interpolation and teacher-forced hidden states that differ from discrete hybrid computation, and it keeps weights frozen, ignoring parameter adaptation during the later recovery stage.

FlashMorph2 addresses both issues by evaluating sampled discrete hybrid architectures end to end using their own hidden states, and by jointly optimizing layer-selection scores and model weights under the recovery objective. In experiments across Qwen3 model scales and linear-attention variants, the authors report average score improvements over FlashMorph of 2.42 points on RULER, 1.08 on LongBench, and 0.76 on recall-intensive tasks, with layer selection taking 0.72 to 3.57 GPU hours for models from 0.6B to 8B parameters.

What it does not show

  • The source is the paper abstract only; experimental setup details, baselines beyond FlashMorph, and full results are not available in the supplied passages.
  • The reported gains and runtimes are the authors' own measurements and have not been independently confirmed.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.