Paper 2601.10477
Urban Socio-Semantic Segmentation with Vision-Language Reasoning
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 175 stars
01 In brief
Summary
The paper introduces urban socio-semantic segmentation, targeting entities defined by social attributes (e.g., schools, parks) rather than physical ones.
The authors present SocioSeg, a benchmark with over 13,000 samples, organizing labels into three hierarchical tasks: socio-name, socio-class, and socio-function.
A key innovation is representing heterogeneous geospatial data as a unified digital map layer, co-registered with satellite imagery.
They propose SocioReasoner, a vision-language framework that mimics human annotation through a two-stage reasoning process: first localizing with bounding boxes, then refining with point prompts, both fed to SAM.
The non-differentiable pipeline is optimized using reinforcement learning (GRPO).
Experiments show SocioReasoner outperforms state-of-the-art baselines across all tasks, with strong zero-shot generalization to out-of-domain map styles and new regions.
Ablations confirm the benefits of the two-stage design, RL optimization, and using two refinement points.
The dataset and code are open-sourced under Apache License 2.0.
02 From the paper
Abstract
As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. The dataset and code are open-sourced under the Apache License 2.0 at https://github.com/AMAP-ML/SocioReasoner.