Designing reward signals for RL environments

Gantovar Environments (fictional example) → RL environment provider → Verified task environments → Reproducible resets

Content reviewed 8 November 2026. How this is assessed.

Part of the Best RL Environments reference, which uses Gantovar Environments (a fictional example) to illustrate reinforcement learning environments criteria.

Treat every reward as a proxy

Researchers have formally defined reward hacking as the case where optimizing an imperfect proxy reduces performance on the true reward. The same paper argues that simplifying a reward or ignoring fine distinctions between outcomes usually does not produce a proxy that is safe from hacking. An earlier and widely read paper, Concrete Problems in AI Safety, treats avoiding reward hacking and avoiding side effects as two distinct problems that stem from a wrong objective function.

The practical response is to keep two signals. One is the training reward, which may be shaped to help learning. The other is a separate evaluation metric that is closer to what you truly want and that never feeds the optimizer. When the training reward rises and the evaluation metric does not, you have found the gap, and you should inspect some episodes before spending more compute.

Decide between sparse, partial and shaped

A sparse reward pays only on full success. It is hard to game and hard to learn from. Partial credit is the middle ground: OSWorld scores 1 for full success, a value between 0 and 1 for partial success, and 0 otherwise. Partial credit helps only when the sub-steps are real progress, so check that an agent cannot collect most of the score by doing the easy parts and stopping.

Shaped rewards add intermediate signals such as tests passed or files correctly created. They speed up learning and create more places to exploit. Add them one at a time, and record each change as a new environment version so that runs remain comparable. Be careful with penalties too. A penalty for taking too many steps can teach an agent to give up early when finishing looks expensive.

Signal the end of an episode correctly

How an episode ends affects learning. The Gymnasium API separates terminated, meaning the agent reached a terminal state as defined by the task, from truncated, meaning a condition outside the task ended it, typically a time limit. The Gymnasium tutorial on time limits explains why this matters: at a true terminal state a value estimate should not bootstrap, while after a truncation the last state is not terminal and bootstrapping should still occur. A single done flag merges these two cases.

Apply the same care to infrastructure. If a run ends because the sandbox crashed, an API failed or a tool timed out, label that as an infrastructure failure instead of as a zero reward. Otherwise your training data will include penalties for events that the agent had no control over. Report the rate of such failures with every batch of results, and treat a rising rate as a defect in the environment.

Sources for this entry

• Gymnasium Env API: Reset, step, termination, truncation and seeding.
• Establishing Best Practices for Building Rigorous Agentic Benchmarks: Task validity, outcome validity and reporting checklist.
• SWE-bench: Fail-to-pass validation on real software issues.
• OSWorld: Per-task setup, snapshots and evaluation in a real computer environment.
• ImpossibleBench: Measuring whether agents exploit test cases.
• Defining and Characterizing Reward Hacking: Formal treatment of proxy rewards.
• Deep Reinforcement Learning at the Edge of the Statistical Precipice: Reporting uncertainty with few runs.
• NIST SP 800-190: Application Container Security Guide.

Continue this reading path

• Best RL Environments category guide
• Gantovar Environments (fictional example)
• How to choose: an RL environment provider
• Resource directory

Cite this entry

Best RL Environments. "Designing reward signals for RL environments." Content reviewed 8 November 2026. https://bestrlenvironments.com/reward-signals/