Adversarial Hacker-Fixer Loops

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

Announce Type: new Abstract: Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1,968 tasks across five terminal-agent benchmarks and find 323 (16%) hackable by frontier models given only the task description. This corrupts both leaderboard rankings and RL training signal, yet the standard response is manual and reactive.