Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards

3 min read

CaRR replaces the binary “is the final answer right” reward for web search agents with a reward for the evidence behind the answer. Before training, an LLM breaks each synthetic multi-hop question into single-hop rubrics with placeholders for the hidden entities. During RL, a judge LLM (DeepSeek-v3.2) checks each rollout’s final answer three ways: is every hidden entity named, is each rubric supported by the web content the agent cited (at most 20 URLs), and do the supported rubrics connect to the predicted answer through an entity-rubric graph searched breadth-first. The rubric reward is the share of rubrics passing all three. C-GRPO adds this reward, weighted by α = 0.3 and normalised within the rollout group, only to rollouts whose answer is already correct.

Findings

  • Trained on DeepDive (2,234 RL questions) with Qwen3-4B and Qwen3-30B-A3B thinking models. Average gain over plain GRPO: 5.1 and 8.0 points for the 4B model at 64k and 128k context, 2.6 and 6.0 for the 30B model.
  • BrowseComp, 30B, at 64k/128k context: SFT 12.2/20.5, GRPO 16.0/18.9, C-GRPO 17.9/24.8. Plain GRPO does worse than its own SFT starting point at 128k, which the authors put down to shortcut answers.
  • Under GRPO the number of tool calls keeps falling through training; under C-GRPO it rises again after an early dip. On a BrowseComp subset the GRPO agent cites fewer pages than the SFT model and satisfies fewer connected rubrics (4.0 of 10.1, against 4.5 for SFT and 5.2 for C-GRPO).
  • Results peak at α = 0.3. Larger weights pull the agent away from getting the answer right.
  • A human check of the judge on 10 trajectories (128 entities, 164 rubrics) gives 97.7% accuracy on entity identification and 95.1% on citation support.

Limits

  • Rubrics come from the compositional structure of synthetic multi-hop questions. The authors say this may not carry over to open-ended QA.
  • The reward is only as good as the judge, and the human check covers 10 trajectories.
  • The benchmarks are themselves scored by an LLM judge (GPT-5-Chat).
  • Support is checked against the cited page. A real page that is wrong still counts as support.

Relevance

The failure it fixes is general: an outcome-only reward lets an agent reach the right answer for the wrong reasons, and training pushes it to check less. A robot team rewarded only on task success could drift the same way and skip verification. Rewarding a connected evidence chain could carry over to runtime-monitoring and robust robotics: what would a citation look like for a robot’s claim that a corridor is clear?