CS Paper Radar

OverLay++: Dense-Overlap Layout-to-Image Generation Dataset

Shivansh Aggarwal, Shresth Grover, Divyansh Srivastava, Haiyang Xu, Bingnan Li, Xiang Zhang, Ethan J. Armand, Chuan Li, Jianwen Xie, Zhuowen Tu

cs.CV ·submitted 2026-10-06 20:16 UTC · v1
top venuepositioncode/data

Abstract

Layout-to-Image generation has made substantial progress in spatial and object-level control. However, existing methods still struggle with complex scenes containing many overlapping and interacting objects. We argue that training data is a particular bottleneck: existing datasets lack examples with dense, complex object interactions. To address this gap, we introduce OverLay++, a large-scale Layout-to-Image dataset with structurally complex scenes. OverLay++ contains approximately 500K images with an average of 6.6 objects per image, exceeding existing datasets by 1.67 times in annotation density. Beyond annotation density, OverLay++ provides rich semantic detail with object captions over six times longer than in current datasets. Our dataset generation pipeline is simple and produces dense, overlapping object annotations with rich per-object captions. Across multiple benchmarks, state-of-the-art Layout-to-Image methods trained on the OverLay++ dataset show consistent improvement and faster convergence, demonstrating the importance of dense, overlap-aware, and caption-rich supervision for controllable image generation.

Comment
Accepted at NeurIPS 2026, Evaluations & Datasets Track. Project website: https://mlpc-ucsd.github.io/OverLayPP . Dataset: https://huggingface.co/datasets/mlpcucsd/OverLayPP

Closest papers

Nearest neighbours by meaning across the whole index.

Advancing Video-Text Pretraining with Multi-View Captions

cs.CV ·2026-09-28 ·Fida M. Thoker, Renaud Vandeghen, Karen Sanchez, Marc Van Droogenbroeck, Bernard Ghanem
code/data similarity 0.72
Abstract

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/

0.26

UniMore: Fine-Grained Supervision for Unified Understanding and Generation

cs.MM cs.CV ·2026-11-06 ·Haoyue Bai, Yinyi Luo, Yicheng He, Qingsong Wen, Sharon Li +1 more
similarity 0.71
Abstract

Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image--text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image--text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter details, conditioned on the full image. The two routes are optimized independently, and their learned updates are combined into a single unified model without changing its inference procedure. Experiments show gains across multiple understanding and generation benchmarks.

0.13

Training-free Conditional Object Composition for Multi-object Image Generation with Diffusion Models

cs.CV ·2026-10-31 ·Hsuan-Fu Wang, Hsuan-Tien Lin
publishedcode/data similarity 0.71
Abstract · Accepted for oral presentation at TAAI 2026

Multi-object image generation in text-to-image diffusion models remains challenging, as existing approaches struggle to reliably generate all objects specified in a prompt. Compositional approaches that generate objects independently often fail to capture the interactions among them, whereas joint generation methods tend to suffer from inter-object interference. In this work, we propose a training-free framework, termed conditional object composition, that relaxes joint generation methods by taking one step back toward compositional approaches. Our framework decomposes the generation of $N$ objects into two stages: it first generates a subset of $N-1$ objects and then composes the remaining object conditioned on the generated content. To enable effective composition, we adopt an inpainting-based formulation with layout guidance to preserve existing regions and encourage spatially coherent placement. To address the challenge of selecting which objects to generate separately, we introduce an attention-based scoring mechanism based on cross-attention maps and further explore the use of off-the-shelf object detectors. We evaluate our method on multi-object image generation using the widely adopted MS COCO dataset. Experimental results show that our approach achieves competitive or improved performance compared to existing methods, particularly as the number of objects increases. Further studies highlight the importance of the proposed design choices. The source code is available at https://github.com/ShampooWang/condComp_imageGen/tree/main

0.32

PoseAdapter: Dual-Stream 2.5D Controllable Image Generation for Complex Multi-Object Scenes

cs.CV ·2026-08-16 ·Yufeng Chi, Huimin Ma, Fan Gao, Zhice Niu, Keqin Li +1 more
top venuedatasetcode/data similarity 0.69
Abstract · 10 pages, 5 figures

While Text-to-Image (T2I) diffusion models have achieved remarkable success, precise spatial and orientational control in multi-object scenes remains a persistent challenge. Existing methods either rely on computationally expensive dense 3D maps or suffer from severe attribute leakage and "cut-and-paste" artifacts. To address these limitations, we propose PoseAdapter, a lightweight framework for high-fidelity 2.5D controllable image generation. Instead of dense spatial maps, it establishes precise spatial-angular anchors using an efficient condition layout: individual object captions, 2D bounding boxes, and 3D angles. To resolve the generative trade-off between strict instance isolation and global coherence, we introduce a Context-Aware Dual-Stream Representation. By injecting local object tokens and relation-enriched scene tokens into the visual stream of modern MM-DiT architectures via parallel masked and unmasked pathways, PoseAdapter eliminates attribute leakage while preserving natural inter-object relationships and scene-level coherence. To support this paradigm, we construct OrientLayout, a high-quality dataset featuring standardized 2.5D annotations and instance-level decoupled semantics. Extensive experiments demonstrate that PoseAdapter outperforms state-of-the-art baselines in spatial accuracy, orientational precision, and multi-object visual fidelity. Code and dataset will be available at https://github.com/cyf23/PoseAdapter.

0.60

Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data

cs.CV ·2026-11-18 ·Liu He, Yizhi Song, Ruiqi Xu, Baris Sarper Tezcan, Xiao Zeng +7 more
publishedbenchmark similarity 0.69
Abstract · Accepted to Machine Vision and Applications 2027

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

0.25

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

cs.CV ·2026-10-14 ·Tengju Ru, Weitong Lian, Zecong Tang, Zhiheng Liu, Lingyi Meng +4 more
community pick similarity 0.68
Abstract

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact \(G\) tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00\%, 17.94\%, and 8.62\% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

0.26

Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

cs.CV ·2026-10-12 ·Ziqiao Peng, Yutao Cui, Wang Lin, Zhong-Yu Li, Qingwen Gu +10 more
similarity 0.68
Abstract · 29 pages, 11 figures

Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.

0.22

Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation

cs.CV cs.AI ·2026-10-14 ·Yuheng Chen, Yufan Chen, Zhuojun Cai, Ruiping Liu, Junwei Zheng +6 more
top venuebenchmarkdatasetcode/datahot direction similarity 0.68
Abstract · Accepted by NeurIPS 2026

Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD

0.51