← News

ResearchSources: arXiv · 18 Nov 2026

WATERFALL automates reward discovery for reinforcement learning in partially observable environments

An arXiv paper introduces WATERFALL, a workflow that generates and refines reward programs without privileged information, using vision-language models to judge candidate behaviour in POMDP settings.

In plain words

Reinforcement learning systems need reward signals that tell them what counts as success, and designing these by hand is difficult when the system cannot see the full state of its environment. The WATERFALL paper describes a workflow that automatically generates candidate reward programs, tests them by watching recorded behaviour, and iteratively improves the best ones. The authors report it works without ground-truth metrics or task descriptions.

What was announced

The workflow, described in an arXiv abstract by its authors, has three parts. It synthesises diverse programmatic reward candidates through persona-conditioned generation, evaluates candidate behaviour from raw visual rollouts using a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and uses longitudinal assessment histories to mutate, escalate, or prune candidates over iterations.

The authors evaluated WATERFALL in both fully observable and reveal-gated partially observable environments, comparing it against sparse-reward RL, intrinsic-motivation, and foundation-model reward-synthesis baselines. They report that WATERFALL discovers and refines reward programs without privileged information, outperforming baselines that retain native information advantages, most clearly in the partially observable regime. Ablations indicate visual evidence, tournament selection, and iterative refinement are jointly important. The evidence is limited to the abstract; full results, benchmarks, and experimental detail are not available in the supplied passages.

What it does not show

  • Only the paper abstract was supplied; no full-text results, benchmark names, or numerical comparisons are available to verify the reported performance claims.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.