Paper 2607.12625
KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 482 stars
01 In brief
Summary
KnowAct-GUIClaw is a personal GUI assistant framework built on the 'Know Deeply, Act Perfectly' paradigm to address OpenClaw's limitations in cross-platform GUI interaction and self-evolution.
It uses a Know-Route-Act-Reflect loop: the host agent decomposes tasks and allocates them (Know), a router classifies tasks and transfers data via a blackboard (Route), a GUI subagent executes with a hybrid action space of GUI primitives, skills, and shortcuts (Act), and reflection distills trajectories into experience memory and self-evolving skills (Reflect).
Experiments on MobileWorld show the Kimi-K2.6-based GUIClaw achieves 64.1% success rate, beating all agentic frameworks and closed-source models like Seed-2.0-Pro and GPT-5.5.
Memory and skills transfer across base models, improving Qwen3.5-35B by 8.5% and Qwen3.5-397B by 16.2%.
On AndroidDaily, it scores 78.61% (resolved) and 64.89% (all).
Cross-platform checks show 76.2% on HarmonyOS and 70.0% on Windows.
The framework reduces GUI steps and tokens via skill reuse, and the host resolves eligible subtasks directly to cut costs.
02 From the paper
Abstract
OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.