Paper 2601.15197
LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 14
- GitHub
- 75 stars
01 In brief
Summary
LangForce is a framework for Vision-Language-Action (VLA) models that addresses the 'vision shortcut' pathology, where models ignore language instructions due to dataset bias.
In goal-driven datasets, language is predictable from vision, causing conditional mutual information between instructions and actions to vanish (Information Collapse).
LangForce uses a dual-branch architecture with learnable Latent Action Queries to estimate a vision-only prior p(a|v) and a language-conditioned posterior π(a|v,ℓ), optimizing the conditional Pointwise Mutual Information (PMI) via a Log-Likelihood Ratio objective.
This penalizes vision shortcuts and rewards actions that explain language.
Experiments show an 11.3% improvement on SimplerEnv (66.5% vs 55.2% baseline), state-of-the-art results on RoboCasa (52.6%), and gains on LIBERO Goal (99.4%).
Real-world tests show 80.8% success on vegetable pick-and-place vs 59.2% baseline.
The method preserves general language capabilities and incurs no inference overhead.
02 From the paper
Abstract
Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints and fail in out-of-distribution (OOD) settings. To address this, we propose LangForce, a novel framework that enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $π(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Without requiring new data, LangForce significantly improves generalization. Extensive experiments across on SimplerEnv and RoboCasa demonstrate substantial gains, including an 11.3% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of our approach to robustly ground language in action.