← News

ResearchSources: arXiv · 18 Nov 2026

arXiv study links chain-of-thought format to entailment and coherence judgments in Qwen3-8B

Preregistered experiments report large prompt-and-readout effects on entailment and coherence judgments, while the authors keep any internal lock-in mechanism as an unconfirmed hypothesis.

In plain words

The authors tested whether large language model judgments about events and arguments change when the same model answers in different formats. Across thousands of controlled test items, the answer format strongly affected the results. They caution that this shows a pattern in outputs, not proof of what happens inside the model.

What was announced

The arXiv paper reports four preregistered, sealed experiments on Qwen3-8B-Q8_0 covering 16,640 formal units at temperature 0.0. Fifteen of sixteen hypothesis slots kept confirmatory status, twelve rejecting the null hypothesis under within-experiment Holm control, with one demoted to descriptive status. Reported gaps include a 25.00 percentage-point difference between Direct-CoT and MCU on telic high-confidence completion, a 91.25 percentage-point collapse of NLI-CoT on uninterrupted atelic items, and a 26.64 percentage-point rise in perceived coherence on implicit Moorean sentences for MCU-CF over Direct-CoT.

Cross-model extensions complicate the picture: a 1,800-call OpenAI ablation and a planned 4,240-unit extension across five external models found that discrete NLI-CoT and scalar Direct-CoT judgments dissociate across models, with gpt-4o-mini yielding 1,000 of 1,000 valid units. The authors state that confirmatory effects concern observable prompt-and-readout protocols only, and that internal premature lock-in remains a hypothesis. Code, stimuli, and preregistered analysis plans are posted on GitHub.

What it does not show

  • The source is the paper's abstract only; full methods, results, and peer review status are not available in the supplied excerpt.
  • Findings are centered on one primary model (Qwen3-8B-Q8_0); the cross-model extension reports dissociations but no independent replication of the main effects.
  • The authors themselves note that internal premature lock-in remains a hypothesis, so no mechanism claim is established.

For security and IT teams

Scope
The reported effects concern observable prompt-and-readout protocols in formal entailment and coherence tasks, not production system behavior.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.