Training GUI agents for phones has always hit the same wall: where does the data come from? Real trajectories are hard to collect because sensitive apps and privacy-critical operations are not suitable for mass harvesting, and simulated environments are expensive to scale. Existing GUI world models have their own problems: unstable generation over long rollouts, limited modality coverage, and inconsistent action-transition logic. A team from the Chinese academic scene (Weikai Xu, Shuo Shang, Xiaolin Hu, Bo An and others) proposes a different approach: instead of generating the next screen as an image, predict it as a reachable code update. That is AppDeltaWorld.
Delta instead of full state
AppDeltaWorld is a transition-grounded delta code world model. Instead of rendering the whole screen from scratch, the model predicts the difference between the current and the next state – and it does so as executable HTML, not as an unconstrained image or text description. It works like delta compression: reference plus change instead of full state. That keeps generation stable over long rollouts and ensures predicted states remain reachable in the real app.
Architecture: three stages
1. Transition-grounded retrieval (Level-1)
The model does not generate HTML from nothing. First it converts the current screen x_t into a structured textual state r_t = [f_t, d_t, l_t, q_t, p_t] – functional template, screen description, region layout, slot schema, and page type. This representation identifies the source screen family, and target retrieval is constrained to families reachable by the given action (hence “transition-grounded”). From a memory index holding tuples (retrieval text, Level-1 reference HTML, functional category, cluster), it fetches a reference HTML – a known-good layout, widgets, and DOM organization from that app’s history.
2. Delta generation (Level-2)
The text stage first predicts the next-screen text (s_(t+1)) – that is the target-side semantic query for retrieval. Then a multimodal generator produces Level-2 HTML as a delta from the Level-1 reference: h_(t+1) = G(x_t, m_t, s_(t+1), h_i). The reference supplies reusable app structure; the current screen, the action, and the predicted text decide how that structure should change after the action.
3. Hybrid multimodal rendering
Text handles semantic retrieval and transition grounding, code provides the executable layout, and a diffusion model synthesizes visual assets into image slots. The browser renders the final screenshot. The ablation confirms every component earns its place: removing diffusion drops the overall score from 73.51 to 70.91 (element and layout metrics suffer most), removing RAG drops it to 67.46.
World-model-in-the-loop: SFT data and RL without real apps
Rollouts start from GUI-Owl and OpenMobile seeds, the policy predicts a raw response, a parser splits it into a validity flag and a structured action, and the world model returns the rendered next state plus metadata. Trajectories pass quality filtering and land in the SFT set. One important caveat: world-model data does not work in isolation – the world model still makes errors on fundamental interactions. But added to public data it delivers concrete gains: Total-LL ATP rises from 40.79 to 46.63, Total-HL ATP from 28.17 to 33.05.
Results
- CMGUIBench-500 (Code2World): AppDeltaWorld scores 73.51, ahead of image generators GPT-Image-2 (72.34) and Gemini-3.1-Pro-Image (71.57). Against the Qwen3-8B base (50.53) that is a 45.5% relative gain. On element and layout reconstruction (56.26/57.43) it clearly beats diffusion baselines (GPT-Image-2: 44.00/41.70), though manual evaluation shows GPT-Image-2 aligns elements better.
- AndroidLens: AppDeltaAgent-8B achieves the highest AMS across all language and instruction-granularity splits. Total-LL ATP grows from 34.96 to 46.63 (+33.4%), Total-HL ATP from 23.30 to 33.05 (+41.8%) relative to the Qwen3-VL-8B base.
- MobileGym: SR 14.1 vs 10.2 (+38.2%), PR from 22.0 to 26.4; the biggest gains on L1 (66.2 to 83.8) and L2 (13.4 to 19.5).
- MobileWorld (real online environment): consistent closed-loop gains.
Test-time RL on the world model
The authors evaluate two RL variants where the reward comes from the world model, not from a real app. Self-score RL: the policy samples 8 actions, the world model predicts the next states, and the policy scores action consistency and instruction progress itself. AMS gains: +1.51, +3.25, +3.02 and +5.06 on Feizhu, Meituan, JD and Taobao respectively, and the reward rises from 68 to 76 over the first 60 updates. The second variant, clustering reward RL, needs no LLM judge and no action labels: 64 rollouts, predicted states clustered by similarity (0.60 visual + 0.40 text, threshold 0.82), reward only for a state in the unique largest cluster. A useful signal that with a bit of engineering you can squeeze a reward signal out of the world model itself.
What this means for engineers
- Delta prediction is a solid pattern for world models. Code plus retrieval give you the stability and state reachability that image generation lacks. If you are building a GUI simulator, think “reference + change”, not “render everything”.
- The agent gains do not come from better coordinate localization. Action-type analysis shows the biggest jumps are in action-type decisions (wait: 14.19 to 72.97, type: 77.66 to 89.37) and high-level exploration (swipe: 41.31 to 58.37, stop: 43.73 to 59.32). Even with imperfect element positions, interaction experience from a synthetic environment is valuable for the policy.
- A synthetic environment does not replace public data – it amplifies it. The world model still errors on fundamental interactions, so rollouts alone do not work. The hybrid (public SFT plus filtered rollouts) is what generalizes.
- Test-time RL on a world model adapts a policy to a specific app (Feizhu, Meituan, JD, Taobao) without touching real apps. For production mobile agents that is a direct saving on data collection.
AppDeltaWorld is a good example of where GUI-agent training is heading: cheap, stable synthetic environments instead of expensive simulations, with a world model that understands app structure as code, not pixels. The evaluation dataset is public on Hugging Face (xwk123/AppDeltaWorld-Eval), so you can run it yourself.
*This content was generated using artificial intelligence and reviewed by a human editor.*
Sources
- AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents, arXiv:2608.05891 – https://arxiv.org/abs/2608.05891
- China AI Weekly Brief (job 3d03e2d4fae9), 9.08.2026
- AppDeltaWorld-Eval evaluation dataset – https://huggingface.co/datasets/xwk123/AppDeltaWorld-Eval
This article is also available in Polish (PL).
Leave a Reply
You must be logged in to post a comment.