The Year in AI Papers
The year/Independent research

Paper 2511.13647

Part-X-MLLM: Part-aware 3D Multimodal Large Language Model

Chunshi Wang, Junliang Ye, Yunhan Yang, Yang Li, Zizhuo Lin, Jun Zhu, Zhuo Chen, Yawei Luo, Chunchao Guo

Published
Nov 2025
Research lab
Independent
Citations
5
GitHub
119 stars

01 In brief

Summary

Part-X-MLLM is a native 3D multimodal large language model that unifies diverse 3D tasks by formulating them as programs in a structured, executable grammar.

Given an RGB point cloud and a natural language prompt, the model autoregressively generates a token sequence encoding part-level bounding boxes, semantic descriptions, and edit commands.

This structured output serves as a versatile interface to drive downstream geometry-aware modules for part-based generation and editing.

The model uses a dual-encoder architecture to disentangle structure (XYZ+normals) from appearance (RGB), and is instruction-tuned on a large-scale, part-centric dataset.

Experiments show state-of-the-art performance in grounded Q&A, compositional generation, and localized editing through one unified interface.

The model also supports semantic granularity control via clustering of part bounding boxes.

The authors introduce UniPart-Bench, a 30k-entry benchmark spanning 11 task families, to evaluate plan quality and downstream performance.

02 From the paper

Abstract

We introduce Part-X-MLLM, a native 3D multimodal large language model that unifies diverse 3D tasks by formulating them as programs in a structured, executable grammar. Given an RGB point cloud and a natural language prompt, our model autoregressively generates a single, coherent token sequence encoding part-level bounding boxes, semantic descriptions, and edit commands. This structured output serves as a versatile interface to drive downstream geometry-aware modules for part-based generation and editing. By decoupling the symbolic planning from the geometric synthesis, our approach allows any compatible geometry engine to be controlled through a single, language-native frontend. We pre-train a dual-encoder architecture to disentangle structure from semantics and instruction-tune the model on a large-scale, part-centric dataset. Experiments demonstrate that our model excels at producing high-quality, structured plans, enabling state-of-the-art performance in grounded Q\&A, compositional generation, and localized editing through one unified interface. Project page: https://chunshi.wang/Part-X-MLLM/