Visual Web Navigation Agents: Beyond the DOM

ABSTRACT

Traditional autonomous web agents rely heavily on parsing underlying website code which often breaks during dynamic updates. We propose a purely visual framework that navigates user interfaces across web and mobile platforms without accessing underlying structures. By combining a multimodal model with a specialized spatial grounding module the agent accurately translates natural language intents into precise pixel actions. Evaluations across major benchmarks show state of the art success rates proving the viability of vision based orchestration.