Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps
The authors propose an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications, splitting the agent into a Planning agent and a UI Grounding model and using an LLM-as-a-Judge for dense step-level plus terminal rewards, optimized with PPO and step-level advantage estimation. They introduce GiGPPO, a neighbor-conditioned value estimation method that improves critic stability using reward signals from visually similar states across concurrent trajectories, backed by infrastructure orchestrating hundreds of concurrent virtual iOS devices. On UINavBench, iOSWorld, and AndroidWorld, their 8B RL agents substantially outperform SFT and terminal-only RL baselines, achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android, and outperform Gemini-3.5-Flash while tying Gemini-3.1-Pro on real iOS applications.