Skip to main content
ModelsIntermediate7 min read

QRAE: compressing DeepSearch agent context to save tokens and latency

i
iaLocale Team
•
QRAE: compressing DeepSearch agent context to save tokens and latency

When a DeepSearch agent answers a question, it doesn't make one jump: it searches, opens a page, reads a fragment, reasons, and searches again. Every result it finds — the chunks of retrieved text — stays in memory and gets re-read at every subsequent step. After ten searches, the context is often longer than the answer, and most of those fragments will never be used again. That is where the token budget and the latency go.

QRAE (Query-Relevant Attention Entropy) is a context compression technique that attacks exactly that point: deciding, before a fragment enters the reasoning context, whether it is worth keeping. The paper — QRAE: Query-Relevant Attention Entropy for Context Compression in DeepSearch Agents — measures it across multiple benchmarks and models, and the numbers are the point.

The problem: context that grows at every step

A DeepSearch agent (the kind that does deep research, like Tongyi DeepResearch or OpenResearcher) works in a cycle: it generates reasoning, fires a tool action (search, open, read), and folds the result into the context. The retrieved fragments — the chunks — make up most of the context, and they accumulate step by step.

The cost is twofold. First, every token that stays in the context gets re-processed at every subsequent call: the prefill re-pays itself at each step. Second, more context means more latency: the model spends time just re-reading material it will never use.

The classic solutions — summarizing the history with another model (ACON), removing tokens with a learned compression policy (LLMLingua-2), or folding completed subtrajectories into compact summaries (Context-Folding) — work, but they add an extra generative pass. In practice: you pay another model call to compress the model.

How QRAE works

QRAE's idea is to add no generative call at all: use the model itself, in prefill-only mode, to decide what to keep.

The framework maintains two separate areas:

  • A persistent whitelist — the admitted chunks, which stay in the reasoning context across steps.
  • A candidate cache — the uncertain chunks, held outside the context but retrievable: if new evidence arrives later, they can be re-evaluated and admitted.

For each new batch of chunks, QRAE runs two paired prefills: one with the query, one as a query-free control. The difference between the two attention responses isolates the reaction induced specifically by the query. It then measures the normalized entropy of that reaction within each chunk: a genuinely relevant fragment elicits a broad response across many supporting tokens (high entropy), while a coincidental match produces an isolated peak (low entropy).

The signal comes from a small family of query-relevant attention heads, localized once, offline, on 100 HotpotQA trajectories. Once localized they stay fixed: they are a property of the model, not of the localization dataset. The paper calls them "evidence-assessment heads" and shows they recur across different models and architectures.

No modified attention operator, no generative compression pass. Just two extra prefills, at a bounded cost that is independent of the candidate cache size.

The numbers

On the primary model DeepDiver-7B, compared against the uncompressed DeepSearch agent:

BenchmarkSuccess (base → QRAE)Tokens (base → QRAE)Latency (base → QRAE)
BrowseComp-Plus13.8% → 17.2%177k → 54k29.5 s → 20.5 s
XBench-DeepSearch38.0% → 44.0%319k → 117k276 s → 150 s
FRAMES40.9% → 42.8%140k → 94k158.7 s → 90.7 s
SimpleQA79.6% → 82.2%74k → 45k145.6 s → 98.6 s

On average, QRAE improves task success by 2.35 percentage points while cutting latency by 30% and token usage by 45%. On XBench-DeepSearch the tokens fall by 63% (from 319k to 117k) while success rises by 6 points. And across all four benchmarks the success rate never drops: it goes up.

The result generalizes beyond the reference model. On linear-time hybrid architectures (OpenResearcher-30B, Mamba–Transformer) and sparse-attention models (OpenPangu-2.0-Flash-92B, nearly 100 billion parameters), QRAE stays effective: on BrowseComp-Plus with the 92B model, tokens drop from 1077k to 660k (−39%) and success goes from 54.9% to 55.3%.

What changes for local setups

If you run research agents on your own hardware, the context is your bottleneck. The tokens that stay in the context are the tokens your GPU has to re-process, and on a local setup there is no cluster to dilute them: every second of latency is one you feel, and every watt is one you pay.

QRAE doesn't give you a faster model, but it shows you how to make the same model work better: stop dragging irrelevant fragments into the prompts. The 45% token reduction means, at the same model, shorter prompts at every step and lower operating cost. The 30% latency cut is pure perceived speed — on an agent that takes dozens of steps, it is the difference between an answer that arrives and one that makes you wait.

And the limits?

  • It is a method, not a package. QRAE is a research framework: you have to localize the relevant heads (once, offline) and integrate the context manager into your agent. It is not download-and-go.
  • Tested on DeepSearch. The gains are measured on agents that search and accumulate chunks. On a single-turn chatbot the problem does not exist, and neither does the gain.
  • Token savings don't guarantee accuracy. The paper says it explicitly: token counting is not FLOP counting, and savings do not automatically imply the same answers. Here the two go together, but that has to be verified case by case.
  • It depends on an internal signal. It works because those models contain query-relevance heads. On models where that signal is weak or absent, the method loses effectiveness.

How to carry the idea into your setup

Even without implementing QRAE, the principle is the one to adopt: never send the reasoning context everything you retrieved. Three concrete habits:

  1. Keep a bounded evidence context — keep in it only the fragments the current reasoning step is actually using, not the complete history.
  2. Maintain a candidate queue — if a fragment is uncertain, set it aside instead of discarding it: you can re-evaluate it when new context arrives.
  3. Measure tokens per step — if your context grows linearly with the number of searches, you are paying the prefill repeatedly for the same material.

The full paper, with the formulas and the experimental details, is available in the HTML version.

Sources

In summary

QRAE is a context compression technique for DeepSearch agents that decides, with two prefills and the model's own attention entropy, which retrieved fragments deserve to stay in the reasoning context. No extra generative pass, no modified attention operator. The results, across four benchmarks and models up to nearly 100 billion parameters: success +2.35 points on average, tokens −45%, latency −30%. The underlying idea — don't drag into the context what you will not use, and keep a candidate queue for what is uncertain — is the principle to adopt today, whether or not you implement the method itself.

Article generated with AI

In these reading paths

Share:

Enjoyed this article?

Subscribe to the newsletter for new tutorials and guides.

After signing up you receive a confirmation email (double opt-in). Without confirmation we send nothing. No spam, we promise.

Related Articles