← News

Models & toolsSources: arXiv · 18 Nov 2026

Trie-Head speeds generative retrieval serving without retraining

Researchers report a GPU-native decoding interface and shared attention scheme that cut serving overhead in trie-constrained generative retrieval, with throughput gains measured on LinkedIn workloads.

In plain words

Generative retrieval systems look up catalog entries by predicting short codes instead of ranking whole documents. The authors built a faster way for the model itself to produce only valid codes, and let many parallel guesses reuse the same stored prompt data. In their tests this made serving several times faster, with no retraining needed.

What was announced

Generative retrieval predicts catalog entries as short semantic identifiers, and trie-constrained beam search keeps those identifiers valid. The authors of the arXiv paper say wide beams make two serving costs visible: vocabulary-wide score tensors that are mostly discarded, and repeated attention over the same prompt. Their answer is Trie-Head, a model-native output interface that stores the identifier trie as validated GPU tensors. Compared with a Python custom logit processor, it reduces CPU-GPU coordination and memory traffic by avoiding Python constraint callbacks and vocabulary-wide candidate buffers, while scoring compact codebook candidates and streaming the full-vocabulary normalizer so valid-token scores are preserved without retraining.

A second mechanism, shared-context attention, reuses prompt key/value blocks across beams while keeping each beam's private decode tail. Both were implemented in SGLang and evaluated on candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform. The combined system improved closed-loop throughput by up to 5.57x over the custom logit processor baseline and up to 2.63x over BF16 FlashRec at a common beam width of 200 on the Qualification Retrieval Platform workload. These figures come from the authors' own abstract and have not been independently confirmed.

What it does not show

  • Evidence is limited to the paper's abstract; experimental setup details, baselines beyond those named, and results on other workloads are not described.
  • Reported throughput gains are the authors' own measurements on two LinkedIn workloads and are not independently verified.

For security and IT teams

Scope
The reported evaluation covers candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform, at a beam width of 200.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.