ICLR 2026

Urban Socio-Semantic Segmentation with Vision-Language Reasoning

Yu Wang1,2 Yi Wang2 Rui Dai2 Yujie Wang1 Kaikui Liu2 Xiangxiang Chu2* Yansheng Li1*
1Wuhan University 2Amap, Alibaba Group
*Corresponding authors
Paper (PDF) arXiv Code Colab Demo

Urban Socio-Semantic Reasoning in 1080P: SocioReasoner bridges digital street maps and high-resolution Earth observation with vision-language reasoning. Gliding the interactive divider reveals satellite imagery, on which the vision-language reasoning policy illuminates the precise socio-semantic segmentation mask matching textual POIs and commercial functional zones.

Abstract

As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach’s gains over state-of-the-art models and strong zero-shot generalization.

Method Overview

SocioReasoner Framework Architecture

The Overall Architecture of SocioReasoner: Given high-resolution satellite imagery, an aligned OpenStreetMap digital block map, and natural language instructions, the Vision-Language Model (VLM) first produces coarse bounding boxes to localize candidate functional regions. These boxes guide the Segment Anything Model (SAM) to generate an initial coarse mask. The synthesized predictions are iteratively re-rendered and overlaid onto the visual inputs, enabling the VLM to reflect, verify, and emit refined point-and-box prompts for final high-fidelity mask synthesis. The multi-stage prompt generation policy is trained end-to-end via Group Relative Policy Optimization (GRPO).

Quantitative Results

Method Socio-name cIoU Socio-name F1 Socio-class cIoU Socio-class F1 Socio-function cIoU All cIoU All gIoU All F1
UNet 10.9 8.0 12.6 11.2 11.1 11.7 10.7 10.0
Segformer 22.0 18.1 22.4 19.5 21.4 22.1 20.5 18.7
SegEarth-OV 3.3 0.0 3.8 0.0 4.2 3.7 3.7 0.0
RSRefSeg 27.1 30.9 30.7 35.3 28.7 29.0 28.3 32.8
SAM-R1 25.6 37.2 22.3 32.1 17.7 22.5 23.7 32.4
Seg-R1 46.0 50.4 40.4 45.2 34.5 41.0 45.0 45.2
RemoteReasoner 46.6 56.1 42.9 53.9 38.0 43.2 47.7 53.3
VisionReasoner 48.5 58.4 44.4 55.5 36.3 44.0 48.5 54.3
SocioReasoner (Ours) 52.6 64.6 47.6 60.1 40.6 47.9 52.8 59.7
* Comparison on the SocioSeg test split across hierarchical socio-semantic categories. SocioReasoner achieves state-of-the-art performance across all semantic tasks and metric evaluations.

Qualitative Demos

Wuhan University (Educational Entity Grounding)
Wangjing SOHO (Commercial Complex Reasoning)

Citation

@inproceedings{wang2026socioreasoner,
  title={Urban Socio-Semantic Segmentation with Vision-Language Reasoning},
  author={Wang, Yu and Wang, Yi and Dai, Rui and Wang, Yujie and Liu, Kaikui and Chu, Xiangxiang and Li, Yansheng},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2026}
}