← News

Models & toolsSources: arXiv · 18 Nov 2026

Study finds moral answers in language models come from a general judgment circuit

A 14-model interpretability study reports that the attention heads steering moral verdicts also handle other normative judgments, while sparse autoencoder features track the word "morally" rather than moral content.

In plain words

Researchers wanted to know whether language models have a dedicated moral mechanism or a general one for judging by any norm. Across 14 open-weight models, the components that change a moral answer also serve judgments like prudence or etiquette. Separately, the features they found mostly detect the word "morally" itself.

What was announced

The authors built 227 minimal pairs of short phrases, validated by five annotators, where the two versions differ in a moral standard, in another norm such as prudence, etiquette, aesthetics or evidence, or in a descriptive quality. Showing these to 14 open-weight models from the Llama, Gemma and Qwen families, sized 1B to 14B, they found that the attention heads which change a moral answer belong to an evaluation circuit used for every kind of normative judgment tested. The heads carried a similar share of the effect for moral and non-moral pairs, and removing them also reduced the gap between acceptable and unacceptable ETHICS benchmark scenarios by 37 to 39 percent, versus 1 to 3 percent for ten random heads.

The sparse autoencoder features the team located responded mostly to the word "morally" in the prompt rather than to the subject matter of the phrase. The models did encode the category of each pair in their hidden states, and in the eight models tested, removing the direction carrying that category weakened moral verdicts by a few percent, more than other verdicts in seven of the eight. The authors release the pairs, ratings, code, SAE dictionaries and result files.

What it does not show

  • The evidence comes from a single arXiv abstract covering 14 open-weight models between 1B and 14B parameters; results for other model families, sizes or closed models are not established.
  • The reported effects are partial: removing the category direction weakened moral verdicts only by a few percent, and the abstract does not establish causal claims beyond the interventions described.

For security and IT teams

Scope
The findings apply to the 14 open-weight Llama, Gemma and Qwen models tested, from 1B to 14B parameters, in base and instruction-tuned versions, with and without the word "morally" in prompts.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.