Nubank screens customer-service AI agents in simulation before live deployment
A paper describes a hypothesis-driven simulation workflow that tested candidate customer-experience agents at Nubank, including its highest-volume chat-support agent in Brazil, before customer exposure.
In plain words
Nubank's researchers tested new customer-service AI agents by simulating conversations with synthetic customers and fake tool responses, so no real customers saw failures. Simulated results tracked production quality scores closely, and changes picked through simulation improved live performance in A/B tests.
What was announced
Customer-experience agents combine large language models with tools to handle requests, and the paper's authors note they must detect intent, follow operational policies and use tools reliably — hard to verify through manual end-to-end testing, while live experiments risk exposing customers to failures. Their workflow uses synthetic customers that react to agent responses and simulated tool outputs, allowing multi-step agentic workflows without touching production backends.
Applying the Snowglobe simulator to Nubank's Card Delivery agent and its successor, Card Management — described as Nubank's highest-volume chat-support agent in Brazil — the authors report high correlation between simulated and production evaluator scores across four deployed versions. Simulation-guided iteration raised transactional net promoter score by 36.69 points in a live A/B test. They also screened open-weight configurations across more than 16,000 simulated conversations; the selected model lifted self-service rate by 8.82 percentage points in a later A/B test, with no statistically significant tNPS change.
What it does not show
- The evidence is the paper's abstract only; full methodology, baselines and caveats are not available for verification.
- Correlation between simulated and production scores is reported by the authors and has not been independently confirmed.
- The tNPS and self-service rate gains come from the authors' own A/B tests; no third-party replication is described.
For security and IT teams
- Scope
- The workflow screens candidate customer-experience agents before deployment using simulated customers and simulated tool outputs, without invoking production backends.
- Status
- The work is described in an arXiv paper (v2, replace announcement) covering Nubank's Card Delivery and Card Management agents; it is a research report, not a generally available product announcement.
Sources
Source-support review completed 13 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 13 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.