← News

ResearchSources: arXiv · 19 Nov 2026

HorizonBench tests whether AI assistants track changing user preferences

A new arXiv benchmark generates six months of simulated conversation in which user preferences shift after life events, and finds all 25 evaluated frontier models often cling to outdated preferences.

In plain words

A benchmark is a standard test for measuring how well AI systems perform a task. HorizonBench tests whether an assistant notices when a user's preferences change over months, rather than only remembering what the user once said. The authors found that every model they tested repeatedly answered with preferences the user had already outgrown.

What was announced

The arXiv paper introducing HorizonBench argues that existing benchmarks cannot distinguish whether a model forgot what a user said or failed to update its belief about that user, because they either hold preferences fixed or have the user state the change directly. HorizonBench instead samples how a user's preferences evolve and generates six months of conversation from that sequence, with each change tied to a life event the user mentions without stating the new preference. When a preference has changed, its earlier value appears among the answer options, so models that rely on stale information are visibly penalized.

The authors report that all 25 frontier models evaluated chose the outdated preference more often than evenly spread errors would predict, with the best model reaching 52.8% accuracy. Two of the newest models scored no higher when given only the relevant conversations than with the full history, yet adding one sentence that states the change raised their accuracy by 11 to 22 points. Retrieval and memory methods did not recover the lost accuracy. The team also collected and released a human dataset of 83,024 ratings from 160 people over 13 weeks, which sets how often and how much life events shift preferences. The benchmark, its generator, and the human dataset are released to support research on treating recall and current-preference tracking as separate capabilities.

What it does not show

  • The evidence is the paper's abstract only, so the evaluation methodology, model list, and error analysis cannot be verified from the supplied passages.
  • The reported results are the authors' own claims; no independent confirmation is included in the sources.
  • The human preference dataset covers 160 people over 13 weeks, which may not capture preference change over the six-month horizons the benchmark simulates.

For security and IT teams

Scope
HorizonBench, its data generator, and the human longitudinal preference dataset are released for research on personalization, per the paper's abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.