SignAligner trains one model to place subtitles and words in sign-language video
A new arXiv paper describes a prompt-conditioned span generator trained jointly on subtitle and word spans, then refined with overlap-based rewards for sign-language video alignment.
In plain words
Sign-language videos need time markers: where a subtitle applies and where a specific word's sign happens. The authors built a model that takes a video plus a subtitle or word query and outputs the start and end frames. It learns both marker types together and is then tuned with rewards for overlapping the correct span.
What was announced
The paper, a cross-listed arXiv abstract, introduces SignAligner as a visual-language span generator that maps a sign-language video and either a subtitle-sentence or target-word query to a textual start and end frame interval. The authors say it is first trained jointly on subtitle spans and word spans drawn from sign spottings, then refined with overlap-based rewards while keeping auxiliary word examples. They frame joint training on both span scales as underexplored, since word-level sign spotting already exists at corpus scale but subtitle alignment has not typically been trained this way.
On the subtitle test set of the BBC-Oxford British Sign Language (BOBSL) corpus, the authors report the refined model reaches 66.68 [email protected], which they describe as the best among compared systems that use no external sign-video resources. They state that overlap-reward refinement beats cross-entropy continuation from the same jointly trained model, and that joint training with word spans improves subtitle alignment in a matched pair of generators, an advantage persisting after both are refined. For word queries, they say the model follows the queried word rather than a fixed video position, making it far more accurate when the sign lies away from the centre. They position the result as one shared generator for subtitle alignment and word spans for human review.
What it does not show
- Evidence is limited to the paper's own abstract; the reported F1 score and comparisons are author claims on the BOBSL subtitle test set and have not been independently verified.
- The abstract is truncated mid-sentence, so full results, baselines, and details beyond the excerpt are not available here.
For security and IT teams
- Scope
- The reported results cover subtitle alignment on the BOBSL corpus test set and word-span queries; the authors do not claim evaluation on other sign-language corpora in the abstract.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.