Paper 2601.16211
Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 9 stars
01 In brief
Summary
This paper addresses object-driven shortcuts in Zero-Shot Compositional Action Recognition (ZS-CAR), where models predict verbs by relying on object class labels rather than temporal evidence.
The authors identify two root causes: sparse and skewed compositional supervision, and asymmetric learning difficulty (objects are easier to recognize than verbs).
They introduce diagnostic metrics (FSP, FCP, Compositional Gap) to quantify shortcut reliance and show that existing models overfit to co-occurrence priors.
To mitigate this, they propose RCORE with two components: Co-occurrence Prior Regularization (CPR), which synthesizes unseen compositions and penalizes frequent hard negatives, and Temporal Order Regularization for Composition (TORC), which enforces temporal-order sensitivity.
Experiments on Sth-com and a new EK100-com dataset show RCORE reduces shortcut diagnostics and improves unseen compositional generalization across CLIP and InternVideo2 backbones, achieving positive compositional gaps on unseen compositions.
02 From the paper
Abstract
Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.