{"id":91,"date":"2026-07-29T16:46:31","date_gmt":"2026-07-29T15:46:31","guid":{"rendered":"https:\/\/ai.quailoop.com\/index.php\/2026\/07\/29\/openforgerl-finally-you-can-train-ai-agents-with-rl-in-any-environment\/"},"modified":"2026-07-29T16:46:31","modified_gmt":"2026-07-29T15:46:31","slug":"openforgerl-finally-you-can-train-ai-agents-with-rl-in-any-environment","status":"publish","type":"post","link":"https:\/\/ai.quailoop.com\/index.php\/2026\/07\/29\/openforgerl-finally-you-can-train-ai-agents-with-rl-in-any-environment\/","title":{"rendered":"OpenForgeRL \u2014 Finally, You Can Train AI Agents with RL in Any Environment"},"content":{"rendered":"<p><strong>OpenForgeRL brings RL-based training to harness-native AI agents in any environment \u2014 no more prompt hacking, no more manual trajectory collection. This is the most impactful paper this week from the Chinese AI scene for anyone building agents.<\/strong><\/p>\n<hr \/>\n<h2>The Problem: Agents Are Trained Like Crippled Cats<\/h2>\n<p>Modern AI agents \u2014 Claude Code, Codex, OpenClaw, even Hermes Agent \u2014 use complex <em>inference harnesses<\/em>: multi-step loops of reasoning \u2192 tool call \u2192 observation \u2192 reasoning. The problem is that no open RL\/SFT stack can handle this natively.<\/p>\n<p>Standard tools (TRL, OpenRLHF, veRL) assume a simple sequence: prompt \u2192 response \u2192 reward. But an agent makes 5-20 steps, calls APIs, reads files, executes code. Every step changes the environment state. Trying to train this with existing frameworks results in:<\/p>\n<ul>\n<li>Manual trajectory logging into JSONL<\/li>\n<li>Endless prompt engineering instead of actual learning<\/li>\n<li>Or \u2014 most often \u2014 dropping RL entirely and settling for SFT on static datasets<\/li>\n<\/ul>\n<p>The authors of OpenForgeRL \u2014 Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou \u2014 state it plainly: &#8220;elaborate inference harnesses make agents hard to train end-to-end with open infrastructure.&#8221; And they propose a solution.<\/p>\n<h2>How OpenForgeRL Works<\/h2>\n<p>The architecture is built on three layers:<\/p>\n<ol>\n<li><strong>Environment Abstraction Layer (EAL)<\/strong> \u2014 a uniform interface to any environment (terminal, API, browser, game). Every environment exposes the same primitives: <code>reset()<\/code>, <code>step(action) \u2192 observation, reward, done<\/code>. The agent can&#8217;t tell the difference between working in bash and in a game.<\/li>\n<li><strong>Harness-Native Policy<\/strong> \u2014 an RL policy designed as part of the harness, not a separate component. The model generates not just a response, but an entire sequence: <code>[think] \u2192 [tool_call] \u2192 [observe] \u2192 [think] \u2192 ... \u2192 [final_answer]<\/code>. Each step is its own token in the rollout, with its own gradient.<\/li>\n<li><strong>Multi-Turn PPO \/ GRPO<\/strong> \u2014 adaptation of PPO and GRPO to multi-step rollouts. The key modification: advantage is computed per-step (with gamma discount), not over the full trajectory. This lets the model learn which <em>individual<\/em> actions (e.g., &#8220;check the file before answering&#8221;) are valuable, even if the final answer is wrong.<\/li>\n<\/ol>\n<pre><code># Pseudocode: training an agent with OpenForgeRL\nenv = TerminalEnvironment(cwd=\"\/project\")\nagent = HarnessPolicy(model=\"Qwen3.6-8B\", harness=\"codex-style\")\ntrainer = MultiTurnPPO(gamma=0.95, lam=0.95)\n\nfor episode in range(1000):\n    obs = env.reset()\n    trajectory = []\n    while not done:\n        action = agent.act(obs)          # [think] [tool_call] ...\n        obs, reward, done = env.step(action)\n        trajectory.append((obs, action, reward))\n    trainer.update(trajectory)\n    # agent learns: check errors early,\n    # don't overuse tools, plan ahead\n<\/code><\/pre>\n<h2>Results<\/h2>\n<p>The authors benchmarked OpenForgeRL on three agent benchmarks:<\/p>\n<ul>\n<li><strong>SWE-bench Verified<\/strong> (bug fixing in Python repos): +12.3 p.p. over SFT baseline, +5.1 p.p. over vanilla PPO<\/li>\n<li><strong>WebArena<\/strong> (web navigation): +8.7 p.p. over SFT<\/li>\n<li><strong>ToolBench<\/strong> (tool calling, 4 environments): +14.2 p.p. over SFT, especially in the &#8220;multiple tools in sequence&#8221; category<\/li>\n<\/ul>\n<p>Critically \u2014 an 8B model trained with OpenForgeRL matches a 32B model trained with vanilla SFT on SWE-bench. That&#8217;s a 4x inference cost reduction for the same quality.<\/p>\n<h2>What This Means for Engineers<\/h2>\n<p>If you&#8217;re building an agent (IDE assistant, CLI agent, web scraper, devops pipeline), OpenForgeRL changes the game:<\/p>\n<ul>\n<li><strong>No more dataset collection.<\/strong> The agent generates its own trajectories in the environment, gets reward signals (task pass\/fail), and learns from its own mistakes.<\/li>\n<li><strong>Train in any environment.<\/strong> Terminal, API, browser \u2014 EAL abstracts the differences.<\/li>\n<li><strong>RL &gt; prompt engineering.<\/strong> Instead of spending weeks finding the perfect prompt, run 1000 rollouts and let the model discover the optimal strategy.<\/li>\n<li><strong>Downscaling works.<\/strong> A smaller model with RL beats a larger model without RL \u2014 direct cost savings in production.<\/li>\n<\/ul>\n<h2>Limitations<\/h2>\n<p>OpenForgeRL isn&#8217;t without tradeoffs:<\/p>\n<ul>\n<li>~2x more GPU-hours for training than vanilla PPO (longer rollouts)<\/li>\n<li>Only works for environments with <em>dichotomous<\/em> rewards (task pass\/fail) \u2014 not suitable for open-ended tasks like &#8220;write a better essay&#8221;<\/li>\n<li>Currently research, not a pip-installable library \u2014 no ready integrations with agent frameworks (LangChain, CrewAI)<\/li>\n<\/ul>\n<h2>Paper Link<\/h2>\n<p><a href=\"https:\/\/arxiv.org\/abs\/2607.21557v1\">arXiv:2607.21557v1 \u2014 OpenForgeRL: Train Harness-native Agents in Any Environment<\/a><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li>arXiv:2607.21557v1 \u2014 Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou (July 2026)<\/li>\n<li>China AI Weekly Brief, period July 19-26, 2026 \u2014 data from NetEase Tech, arXiv, HuggingFace<\/li>\n<li>SWE-bench Verified, WebArena, ToolBench \u2014 benchmark results cited in the paper<\/li>\n<\/ul>\n<hr \/>\n<p><em>This content was generated using artificial intelligence and reviewed by a human editor.<\/em><\/p>\n<hr \/>\n<p><em>This article is also available in <a href=\"https:\/\/ai.quailoop.com\/index.php\/2026\/07\/29\/pl-openforgerl-wreszcie-mozna-trenowac-agenty-ai-z-rl-w-dowolnym-srodowisku\/\">Polish (PL)<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>OpenForgeRL brings RL-based training to harness-native AI agents in any environment \u2014 no more prompt hacking, no more manual trajectory collection. This is the most impactful paper this week from the Chinese AI scene for anyone building agents. The Problem: Agents Are Trained Like Crippled Cats Modern AI agents \u2014 Claude Code, Codex, OpenClaw, even [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-91","post","type-post","status-publish","format-standard","hentry","category-english"],"_links":{"self":[{"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/posts\/91","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/comments?post=91"}],"version-history":[{"count":0,"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/posts\/91\/revisions"}],"wp:attachment":[{"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/media?parent=91"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/categories?post=91"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai.quailoop.com\/index.php\/wp-json\/wp\/v2\/tags?post=91"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}