Sirius-Tiamat study maps how much compression a 421M decision encoder tolerates
The authors of the Sirius-Tiamat paper measure when quantisation, layer deletion and distillation change a one-pass decision encoder's answers, not just its size.
In plain words
A decision encoder is a compact model that reads a question with several options and picks probability scores for each in one pass. The Sirius-Tiamat authors compressed a 421M-parameter model of this kind and counted how often its answers changed. Small compressions changed few answers, but heavier cuts and out-of-family questions changed many more.
What was announced
The paper reports that bf16 rounding changes 0.2% of in-family decisions but 2.2% on out-of-family questions, while 6-bit grouped rounding changes 1.3% and 5.9%. Deleting six layers changes 0.9% of in-family decisions yet 9.1% out of family, and the expected calibration error rises from 5.9% to 39.2%. A single fitted temperature, which the authors say needs no labels, restores calibration to 3.6%. At equal stored size, rounding flips fewer decisions than deletion.
Heavier compression is less forgiving: CPU dynamic int8 changes 30% of decisions, nearly all through one layer, and one scale per tensor fails below six bits. Distilling a 16-layer student cuts its in-family flips from 44% to 9%, with 45% out of family. The authors also find an early exit saves layers over a static cut only if its confidence gate is recalibrated first. The data are synthetic and LLM-labelled, and cover one architecture.
What it does not show
- The evidence is the paper's abstract only, so the full article's methods and results are not covered here.
- All evaluation data are synthetic and LLM-labelled, and the study covers a single architecture, so results may not transfer to other models or real-world data.
- The measurements are the authors' own and have not been independently confirmed.
For security and IT teams
- Scope
- The findings apply to one 421M-parameter one-pass decision encoder evaluated on synthetic, LLM-labelled data, with out-of-family checks on ARC, MMLU and HellaSwag questions.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.