Training a coding agent with reinforcement learning requires something most RL work doesn't: an environment where "correct" is actually verifiable. You can't reward-hack your way past a test suite. That constraint — and the amount of infrastructure it takes to satisfy at scale — is the part that surprises people first.
What a coding "gym" actually is
A gym, in this context, is a sandboxed environment that gives a model a real repository, a real task (fix this bug, implement this feature, pass this test), and a way to verify whether the model's changes actually solved it. The task types span a wide range of difficulty and shape:
- File localization and debugging in an existing codebase
- Writing reproduction tests for a reported bug
- Multi-step repository tasks that touch several files
- Long-horizon software-engineering tasks that require planning across many steps
- Security-related tasks
- DevOps-style tasks — for example, performing real AWS actions in a sandboxed environment (LocalStack) and setting up infrastructure from scratch
Each of these needs its own verification logic. "Did the tests pass" is a fine signal for a bug fix; it says nothing about whether a DevOps task correctly configured a resource. Building one gym is a research problem. Building a hundred is an engineering and organizational problem.
The bottleneck isn't ideas — it's authoring cost
Early on, every new gym was mostly bespoke: a new task meant writing new setup scripts, a new verifier, and new scaffolding to sandbox execution safely. That's fine for the first ten environments. It falls over once a whole organization wants to contribute tasks, because most of the work per gym is boilerplate, not the interesting part (the task itself).
The fix was to build a reusable, scalable architecture that standardizes three things across every gym:
- Environment creation — a consistent way to spin up a repository state, seed it with the task, and tear it down cleanly.
- Sandboxed execution — isolating model actions so a gym can safely let an agent run arbitrary commands (including real cloud actions in LocalStack) without risk to shared infrastructure.
- Verifiable rewards — a common interface for scoring outcomes, so a new gym only needs to implement its verification logic, not rebuild the harness around it.
Standardizing these cut environment-authoring time from days to hours (roughly a 5x reduction) and — more importantly — turned gym-authoring from something only a few specialists could do into something 30+ scientists and engineers across the organization could contribute to directly, scaling the total environment count past 100.
Quality control at the environment level
Scaling the number of gyms creates a second problem: not every environment is actually good for training. Some are trivially easy (the model solves them regardless of policy quality, contributing no gradient signal). Some are effectively unsolvable due to authoring bugs. Some don't discriminate between a strong and weak policy at all.
This calls for evaluating the environments themselves, not just the model being trained on them — intrinsic and extrinsic suites that score each gym's difficulty, trainability, and discriminative power, so "hard" or broken gyms can be automatically filtered out of the training curriculum before they waste compute or, worse, teach a degenerate policy.
What this buys you
The payoff of treating gym infrastructure as a first-class engineering investment, rather than a one-off for each research project, is that RL training stops being bottlenecked by "who has time to hand-build the next environment." Environment creation becomes a distributed effort across the team, quality is checked systematically rather than by spot-checking, and the resulting breadth of tasks — spanning debugging, long-horizon planning, security, and infra work — is a meaningful part of what pushes a coding model's real-world capability, not just its benchmark score.
If you're building RL infrastructure for agentic tasks, the environment layer is worth investing in as seriously as the training algorithm itself. It's usually the actual bottleneck.