← News

Agents & MCPSources: arXiv · 19 Nov 2026

COTEX adds specialist vision tools to trained multimodal agents at adapter-level cost

An arXiv paper proposes a method for extending already trained multimodal agents with new specialist tools while preserving their existing tool skills.

In plain words

Multimodal AI agents use general tools like image cropping, search, and code execution, but usually cannot call specialist vision models. The paper's method, COTEX, trains a small adapter for each new tool so the agent can adopt it without retraining the whole system. The authors report gains on vision-language benchmarks while existing tool performance barely changed.

What was announced

The authors argue that multimodal agents depend on generic tools such as cropping, search, and code execution, which leaves specialist vision models outside their tool space, and that adding a new tool to an already trained agent is difficult because the agent must learn it efficiently without losing existing abilities. COTEX addresses this by identifying two sets of parameter directions: one that triggers a tool call and opens a multi-turn reasoning trajectory, and another that determines which tool is chosen. The adapter for a new tool is restricted to calling directions that leave earlier tool choices unaffected through an orthogonal constraint.

The paper also introduces a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories, providing training data without new annotation effort. In experiments on two representative agents and three specialist tools, the authors report gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools. These results come from the paper's abstract alone, so the experimental setup and baselines cannot be independently assessed.

What it does not show

  • The evidence is limited to the paper's abstract; full experimental details, baselines, and peer review are not available.
  • The reported gains and the 0.3-point drop are the authors' own claims and have not been independently confirmed.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.