← News

Agents & MCPSources: arXiv · 18 Nov 2026

Study finds most tested LLMs leak secrets hidden in system prompts

An arXiv audit of 363 OpenRouter models reports that an AI agent extracted a protected flag from 296 of them, mostly via non-English requests.

In plain words

Researchers hid a secret code inside each model's system prompt, the standing instruction a model is told to follow, then had an AI agent try to talk the model into revealing it. The agent succeeded against 296 of 363 models. The authors say this shows the models put user requests above their own instructions at least once.

What was announced

The study, described as an agentic capture-the-flag exercise, audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog. Each model guarded a 16-character flag placed in its system prompt, while an agent built on OpenCode and powered by DeepSeek V4.1 Flash, steered by a human researcher, tried to persuade the model to emit it. The flag was extracted from 296 models, plus four released after the catalog freeze, for 300 captures in total.

The authors report that 170 of the captures involved a non-English language pivot, most often French. They draw three practical lessons: do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and never let an LLM judge be the sole decider of consequential outcomes. The team also cautions that a capture only shows a model ranked a user request above its system instruction at least once; it does not prove resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.

What it does not show

  • The source is the paper's abstract only, so full methodology and results are not available for verification.
  • A capture shows only that a model once ranked a user request above its system instruction; it does not establish resistance to untested attacks, consistent leakage of sensitive data, or vulnerability to harmful-content jailbreaks.
  • The findings cover the 363 audited OpenRouter models and four later releases, not the broader LLM population.

For security and IT teams

Scope
The audit covers 363 text-in/text-out models from the OpenRouter catalog, not all deployed LLMs.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.