Paper 2603.28301
LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 42 stars
01 In brief
Summary
The paper introduces LIBERO-Para, a benchmark for evaluating paraphrase robustness in Vision-Language-Action (VLA) models, and PRIDE, a metric for fine-grained robustness analysis.
LIBERO-Para varies action expressions and object references independently, yielding 43 paraphrase types.
Across seven VLA configurations (0.6B–7.5B), consistent performance drops of 22–52 percentage points are observed under paraphrasing, primarily driven by object-level lexical variation.
80–96% of failures are planning-level trajectory divergences, indicating task identification errors.
PRIDE combines keyword and structural similarity with task success to weight paraphrase difficulty, revealing that binary success rates overestimate robustness, especially for models like VLA-Adapter (22% overestimation).
The findings suggest current VLA models rely on surface-level matching rather than semantic grounding, and improving object grounding is key to robustness.
02 From the paper
Abstract
Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para