The Year in AI Papers
The year/Independent research

Paper 2605.13527

MMSkills: Towards Multimodal Skills for General Visual Agents

Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

Published
May 2026
Research lab
Independent
Citations
3
GitHub
29 stars

01 In brief

Summary

MMSkills is a framework for representing, generating, and using reusable multimodal procedural knowledge for visual agents.

Each MMSkill package combines a textual procedure, runtime state cards (encoding when-to-use conditions, visible cues, verification cues), and multi-view keyframes (full-frame, focus crop, before/after) to support state-conditioned decision making.

A trajectory-to-skill Generator transforms public, non-evaluation trajectories into these packages via workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing.

At inference, a branch-loaded agent selects relevant state cards and keyframes in a temporary branch, aligns them with the live observation, and returns structured guidance (applicability, subgoal, plan, do-not-do, verify) to the main agent, avoiding context overload and over-anchoring.

Experiments on OSWorld, macOSWorld, VAB-Minecraft, and Super Mario Bros show consistent improvements over no-skill and text-only baselines across frontier and smaller models (e.g., Qwen3-VL-8B-Instruct OSWorld success rises from 10.78% to 25.40%).

Ablations confirm that both state cards and visual evidence are complementary, and branch loading with view selection outperforms direct loading.

MMSkills also reduce action load and repetitive behavior, indicating more grounded, state-aware execution.

Limitations include dependence on source-trajectory coverage, potential generation errors, and extra inference cost from branch loading.

02 From the paper

Abstract

Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.