← News

Agents & MCPSources: arXiv · 6 Nov 2026

Nubank screens customer-service AI agents in simulation before live deployment

A paper describes a hypothesis-driven simulation workflow that tested candidate customer-experience agents at Nubank, including its highest-volume chat-support agent in Brazil, before customer exposure.

In plain words

Nubank's researchers tested new customer-service AI agents by simulating conversations with synthetic customers and fake tool responses, so no real customers saw failures. Simulated results tracked production quality scores closely, and changes picked through simulation improved live performance in A/B tests.

What was announced

Customer-experience agents combine large language models with tools to handle requests, and the paper's authors note they must detect intent, follow operational policies and use tools reliably — hard to verify through manual end-to-end testing, while live experiments risk exposing customers to failures. Their workflow uses synthetic customers that react to agent responses and simulated tool outputs, allowing multi-step agentic workflows without touching production backends.

Applying the Snowglobe simulator to Nubank's Card Delivery agent and its successor, Card Management — described as Nubank's highest-volume chat-support agent in Brazil — the authors report high correlation between simulated and production evaluator scores across four deployed versions. Simulation-guided iteration raised transactional net promoter score by 36.69 points in a live A/B test. They also screened open-weight configurations across more than 16,000 simulated conversations; the selected model lifted self-service rate by 8.82 percentage points in a later A/B test, with no statistically significant tNPS change.

What it does not show

  • The evidence is the paper's abstract only; full methodology, baselines and caveats are not available for verification.
  • Correlation between simulated and production scores is reported by the authors and has not been independently confirmed.
  • The tNPS and self-service rate gains come from the authors' own A/B tests; no third-party replication is described.

For security and IT teams

Scope
The workflow screens candidate customer-experience agents before deployment using simulated customers and simulated tool outputs, without invoking production backends.
Status
The work is described in an arXiv paper (v2, replace announcement) covering Nubank's Card Delivery and Card Management agents; it is a research report, not a generally available product announcement.

Sources

Source-support review completed 13 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 13 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.