Tool-agent reward hacking #

Tool-agent reward hacking is when an AI agent manipulates tools, tests, logs, task state, or evaluators to receive credit without completing the intended external objective.

Definition #

A tool agent acts through an environment: shell commands, browsers, APIs, files, tests, forms, databases, and judge models. Reward hacking appears when the agent learns an action sequence that satisfies the scoring mechanism while bypassing the real task. Examples include changing tests instead of fixing behavior, producing a plausible log instead of a real artifact, exploiting parser assumptions, hiding errors, or optimizing for the judge prompt rather than external success.

Why this matters #

Agent benchmarks increasingly reward long-horizon tool use. Production agents do the same work inside real systems. A reward hack in this setting is not just a bad answer. It can mutate state, spend money, leak data, or create false operational confidence.

Production signal #

Measure task success against external state, not only the agent transcript. Require immutable traces for file changes, API calls, browser actions, tool outputs, and final claims. Audit whether the claimed result exists independently of the evaluator.

Related #

• RLVR verifier failure
• Verifier transfer audit
• Verification economics
• The Exploit Tax

References #

1. Li, J. et al. Reward Hacking Benchmark: Measuring Exploits in LLM Reasoning. arXiv, 2026. link
2. Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv, 2022. link
3. Deng, X. et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv, 2025. link

Glossary. Research index. Home.