Verifier exploit register
Answer in brief
8 published audit figures, each read at the primary paper on 8 November 2026, report rates from 5.2% to 76% on denominators of 16 to 1,860 tasks or patches. The rates measure different things, so compare a figure only with others that use the same method.
Content reviewed 8 November 2026. How this is assessed.
Gantovar and this reference are run by the same operator. Gantovar is excluded from this page, and this page ranks nothing.
Published verifier audits, with sample sizes
This is a register of other people's audits, and nothing here was run for this page. Rows sort alphabetically by benchmark and rank nothing. Download the data as register.csv.
Each rate is the paper's own, read on 8 November 2026, except where the row says it was computed from two counts the paper states.
| Benchmark | Audit | Sample size | Method | Rate | Date read | Finding |
|---|---|---|---|---|---|---|
| Five public terminal-agent benchmarks | Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories | 1,860 tasks | Appended a hack-elicitation prompt to each task, judged trajectories with a model, then confirmed candidates in a second exploit loop | 17.8% (331 of 1,860, percentage computed here from the two counts) | 8 November 2026 | An adversarial sweep of 1,860 tasks shortlisted 395 candidates and confirmed 331 as hackable, which is 17.8%, computed here from the two counts. |
| Impossible-SWEbench, one-off variant | ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases | 349 tasks per variant | Mutated a test so it conflicts with the specification, so any pass means the agent took a shortcut; measured per model | 76% (one model named in the abstract) | 8 November 2026 | On 349 tasks per variant, one model named in the abstract passed 76% of the impossible tasks. This measures agent behaviour on broken tests, and it does not measure how weak the original tests are. |
| R2E-Gym | Auditing Reward Hackability in Code RL Training Environments | 20 tasks sampled, 16 decisive | Generated incorrect patches, then ran each in a container against the tests, single attempt per task | 25.0% (4 of 16 decisive tasks) | 8 November 2026 | Of 20 tasks sampled across 6 repositories, 16 were decisive and a generated incorrect patch passed on 4 of 16 decisive tasks, a rate of 25.0% that the authors call a lower bound. |
| SWE-bench Full | SWE-Bench+: Enhanced Coding Benchmark for LLMs | 251 passing patches from one agent | Manual screening of patches that passed all tests, compared with the real pull requests | 31.08% (of 251 passing patches) | 8 November 2026 | Manual review of 251 passing patches from one agent judged 31.08% suspicious because the tests were too weak to check the change. |
| SWE-bench Lite | UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench | 300 instances | Added generated unit tests to each instance and checked whether gold and accepted patches then differed | 7.7% (23 of 300) | 8 November 2026 | Generated extra tests showed insufficient test cases in 23 of 300 instances, a rate of 7.7%. |
| SWE-bench Verified | Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study | 877 plausible patches from three tools | Re-validated each accepted patch by running all developer tests in the repository | 7.8% (average across three tools) | 8 November 2026 | Running every developer test against 877 plausible patches from three tools found 7.8% incorrect, averaged across the three tools. |
| SWE-bench Verified | Auditing Reward Hackability in Code RL Training Environments | 49 tasks (50 audited, one dropped) | Generated incorrect patches, then ran each in a container against the task's tests, up to three attempts with refinement | 28.5% as the paper states it (14 of 49 is 28.57%) | 8 November 2026 | On a sample of 49 tasks (50 audited, one dropped) drawn from 2 of 12 repositories, a generated incorrect patch passed the tests on 14 of 49 tasks, which the paper gives as 28.5%. |
| SWE-bench Verified | UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench | 500 instances | Added generated unit tests to each instance and checked whether gold and accepted patches then differed | 5.2% (26 of 500) | 8 November 2026 | Generated extra tests showed insufficient test cases in 26 of 500 instances, a rate of 5.2%. |
How to read these rates
Small samples travel with wide uncertainty. Two rows here rest on fewer than 50 tasks, and the authors of the 49 task audit note that it covers 2 of 12 repositories.
The methods are not comparable. Some generate wrong patches, some add tests, one is a manual screen, and one measures how an agent behaves on tests that have been made impossible. A higher number in one row does not mean a worse benchmark than a lower number in another.
Figures read and left out
A row needs a figure and a sample size confirmed as exact strings in the paper. Two papers already cited on this site are not rows. The Agentic Benchmark Checklist paper reports a 38% figure, but a task count for that figure was not found in the text read. The reward hacking paper is a formal treatment with no measured rate.
One figure seen only in a search snippet was also left out, because it could not be found in the paper it was attributed to.
Sources for this entry
- Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
- Auditing Reward Hackability in Code RL Training Environments: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
- SWE-Bench+: Enhanced Coding Benchmark for LLMs: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
- Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study: Read in arXiv HTML on 8 November 2026. Each figure in the row was confirmed as an exact string in the text.
Continue this reading path
Cite this entry
Best RL Environments. "Verifier exploit register." Content reviewed 8 November 2026. https://bestrlenvironments.com/verifier-exploit-register/