StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao et al. · arXiv · 2026
Abstract
Computer-use agents are usually improved by making them better at reading screenshots. This paper argues the screenshot is the wrong interface in the first place: it is a lossy picture of program state that code can inspect and change directly. Their harness works in code by default and calls a GUI subagent only for the minority of subgoals that genuinely need clicking. Success rates rise substantially at roughly nine times lower cost per task.
Why it matters
One of the sharper recent arguments in agent design: pick the interface carrying the most information, not the one that most resembles how a person would do it. The cost figure is the part to remember.
https://arxiv.org/abs/2607.22798