OpenForgeRL brings RL-based training to harness-native AI agents in any environment — no more prompt hacking, no more manual trajectory collection. This is the most impactful paper this week from the Chinese AI scene for anyone building agents.

---

The Problem: Agents Are Trained Like Crippled Cats

Modern AI agents — Claude Code, Codex, OpenClaw, even Hermes Agent — use complex inference harnesses: multi-step loops of reasoning → tool call → observation → reasoning. The problem is that no open RL/SFT stack can handle this natively.

Standard tools (TRL, OpenRLHF, veRL) assume a simple sequence: prompt → response → reward. But an agent makes 5-20 steps, calls APIs, reads files, executes code. Every step changes the environment state. Trying to train this with existing frameworks results in:

• Manual trajectory logging into JSONL
• Endless prompt engineering instead of actual learning
• Or — most often — dropping RL entirely and settling for SFT on static datasets

The authors of OpenForgeRL — Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou — state it plainly: “elaborate inference harnesses make agents hard to train end-to-end with open infrastructure.” And they propose a solution.

How OpenForgeRL Works

The architecture is built on three layers:

1. Environment Abstraction Layer (EAL) — a uniform interface to any environment (terminal, API, browser, game). Every environment exposes the same primitives: reset(), step(action) → observation, reward, done. The agent can’t tell the difference between working in bash and in a game.
2. Harness-Native Policy — an RL policy designed as part of the harness, not a separate component. The model generates not just a response, but an entire sequence: [think] → [tool_call] → [observe] → [think] → ... → [final_answer]. Each step is its own token in the rollout, with its own gradient.
3. Multi-Turn PPO / GRPO — adaptation of PPO and GRPO to multi-step rollouts. The key modification: advantage is computed per-step (with gamma discount), not over the full trajectory. This lets the model learn which individual actions (e.g., “check the file before answering”) are valuable, even if the final answer is wrong.

# Pseudocode: training an agent with OpenForgeRL
env = TerminalEnvironment(cwd="/project")
agent = HarnessPolicy(model="Qwen3.6-8B", harness="codex-style")
trainer = MultiTurnPPO(gamma=0.95, lam=0.95)

for episode in range(1000):
    obs = env.reset()
    trajectory = []
    while not done:
        action = agent.act(obs)          # [think] [tool_call] ...
        obs, reward, done = env.step(action)
        trajectory.append((obs, action, reward))
    trainer.update(trajectory)
    # agent learns: check errors early,
    # don't overuse tools, plan ahead

Results

The authors benchmarked OpenForgeRL on three agent benchmarks:

• SWE-bench Verified (bug fixing in Python repos): +12.3 p.p. over SFT baseline, +5.1 p.p. over vanilla PPO
• WebArena (web navigation): +8.7 p.p. over SFT
• ToolBench (tool calling, 4 environments): +14.2 p.p. over SFT, especially in the “multiple tools in sequence” category

Critically — an 8B model trained with OpenForgeRL matches a 32B model trained with vanilla SFT on SWE-bench. That’s a 4x inference cost reduction for the same quality.

What This Means for Engineers

If you’re building an agent (IDE assistant, CLI agent, web scraper, devops pipeline), OpenForgeRL changes the game:

• No more dataset collection. The agent generates its own trajectories in the environment, gets reward signals (task pass/fail), and learns from its own mistakes.
• Train in any environment. Terminal, API, browser — EAL abstracts the differences.
• RL > prompt engineering. Instead of spending weeks finding the perfect prompt, run 1000 rollouts and let the model discover the optimal strategy.
• Downscaling works. A smaller model with RL beats a larger model without RL — direct cost savings in production.

Limitations

OpenForgeRL isn’t without tradeoffs:

• ~2x more GPU-hours for training than vanilla PPO (longer rollouts)
• Only works for environments with dichotomous rewards (task pass/fail) — not suitable for open-ended tasks like “write a better essay”
• Currently research, not a pip-installable library — no ready integrations with agent frameworks (LangChain, CrewAI)

Paper Link

arXiv:2607.21557v1 — OpenForgeRL: Train Harness-native Agents in Any Environment

Sources

• arXiv:2607.21557v1 — Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou (July 2026)
• China AI Weekly Brief, period July 19-26, 2026 — data from NetEase Tech, arXiv, HuggingFace
• SWE-bench Verified, WebArena, ToolBench — benchmark results cited in the paper

---

This content was generated using artificial intelligence and reviewed by a human editor.

---

This article is also available in Polish (PL).