Paper 2510.06679
DreamOmni2: Multimodal Instruction-based Editing and Generation
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 34
- GitHub
- 2K stars
01 In brief
Summary
DreamOmni2 introduces two new tasks: multimodal instruction-based editing and generation, which accept both text and image instructions and support concrete objects as well as abstract attributes (e.g., texture, pose, style).
The authors propose a three-stage data synthesis pipeline: (1) a feature mixing scheme in attention to generate paired images sharing attributes, (2) using an extraction model and an instruction-based editing model to create editing training data, and (3) applying the extraction model to create generation training data.
The framework includes index encoding and position encoding shift to handle multiple reference images, plus joint training of a VLM (Qwen2.5-VL) with the generation/editing model (Flux Kontext with LoRA) to better understand complex instructions.
A new benchmark with real images (205 editing and 114 generation cases) is introduced.
Experiments show DreamOmni2 outperforms open-source models and is competitive with commercial models like GPT-4o and Nano Banana in both editing and generation, with human evaluations favoring DreamOmni2.
02 From the paper
Abstract
Recent advancements in instruction-based image editing and subject-driven generation have garnered significant attention, yet both tasks still face limitations in meeting practical user needs. Instruction-based editing relies solely on language instructions, which often fail to capture specific editing details, making reference images necessary. Meanwhile, subject-driven generation is limited to combining concrete objects or people, overlooking broader, abstract concepts. To address these challenges, we propose two novel tasks: multimodal instruction-based editing and generation. These tasks support both text and image instructions and extend the scope to include both concrete and abstract concepts, greatly enhancing their practical applications. We introduce DreamOmni2, tackling two primary challenges: data creation and model framework design. Our data synthesis pipeline consists of three steps: (1) using a feature mixing method to create extraction data for both abstract and concrete concepts, (2) generating multimodal instruction-based editing training data using the editing and extraction models, and (3) further applying the extraction model to create training data for multimodal instruction-based editing. For the framework, to handle multi-image input, we propose an index encoding and position encoding shift scheme, which helps the model distinguish images and avoid pixel confusion. Additionally, we introduce joint training with the VLM and our generation/editing model to better process complex instructions. In addition, we have proposed comprehensive benchmarks for these two new tasks to drive their development. Experiments show that DreamOmni2 has achieved impressive results. Models and codes will be released.