CS Paper Radar

When Optimization Meets Human Intent: Human-Centered Diffusion Model Optimization

Renye Yan, Jikang Cheng

cs.CV ·submitted 2026-11-02 13:50 UTC · v1

Abstract

Diffusion models are evolving from general tools into personalized systems that adapt to user needs through fine-tuning and guidance. However, diffusion optimization mainly targets measurable objectives such as aesthetics, rarely examining whether they align with human intent. Human needs are multidimensional, contextual, and dynamic, whereas existing methods approximate them with fixed metrics. Because these metrics capture human needs only partially and may amplify deviations from true intent during optimization, higher scores need not yield results better matching human expectations. To understand how human needs enter and influence optimization, we systematically analyze how human intent is expressed, translated into optimizable objectives, applied to model behavior or generation, and evaluated through outputs. We further discuss trade-offs affecting human-intent alignment in personalized image generation and gaps between automated metrics and real user judgments. This human-centered synthesis reorganizes diffusion optimization and provides a systematic reference for personalized generation systems designed for human preferences.

Closest papers

Nearest neighbours by meaning across the whole index.

Personalized Image Generation with Reasoning and Reflection

cs.CV cs.AI ·2026-09-30 ·Bo Ni, Ngoc N. Tran, Qinwen Ge, Franck Dernoncourt, Seunghyun Yoon +8 more
benchmark similarity 0.66
Abstract

Personalized image generation has remained narrowly focused on conditional synthesis from curated visual exemplars, rather than capturing who a user is. In practice, however, a user's personal context is much richer, comprising reviews, posts, images, captions, and metadata accumulated over time. A truly personalized generator should leverage this history to produce images aligned with the user's lifestyle and aesthetic preferences. To this end, we introduce the first unified benchmark for personalized image generation from user histories. The benchmark comprises two complementary tasks and a multi-axis evaluation protocol that assesses target fidelity, visual quality, user distinguishability, semantic alignment with the user's history, and task-specific utility. Grounded in real-world e-commerce and social media settings, the benchmark includes: (1) Personalized Scene Generation, which places a given object in a scene that reflects a user's preferences and lifestyle, motivated by personalized product presentation; and (2) Personalized Creative Generation, which generates a novel image on a specified topic that is faithful to a user's aesthetic and visual identity, motivated by social media content creation. We further propose PEARL, which couples a multimodal reasoner with a frozen image generator in an interleaved reason-reflect loop optimized with differential data reward. Across both tasks, PEARL outperforms strong baselines, achieving an average improvement of 15% across personalization metrics.

0.23

Personalize at Test Time: Learning User Preferences for Image Generation

cs.CV cs.AI ·2026-10-06 ·Jiamu Bai, Jiaming Hu, Yanhong Wu, Zellux Wang
similarity 0.66
Abstract

Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is accurately modeling diverse user preferences from limited feedback. Existing approaches often rely on labor-intensive manual preference annotations or vision-language models (VLM) to extract preference information from user interaction histories, introducing substantial annotation or computational costs that limit scalability. We propose an approach that learns personalized reward models directly from users' historical image preference pairs. First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions and train an evaluator to score images along these dimensions. We then represent each user's preferences as a linear combination of the shared dimension scores, estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley-Terry model. This formulation reduces per-user adaptation to optimizing a low-dimensional weight vector, simplifying optimization and enabling data-efficient personalization from sparse feedback. The learned personalized rewards guide image generation at inference time while keeping the diffusion model frozen. Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy and improves the alignment of generated images with individual user preferences.

0.31

From Latent Influence to Language: Diffusion-Oriented Content Generation via Audience-Susceptible Features

cs.SI ·2026-08-20 ·Jiaying Lei, Shengqi Dang, Runqian Bai, Ziqing Qian, Nan Cao
similarity 0.64
Abstract

The rapid growth of multimodal user-generated content on social media has made information diffusion a critical factor for advertisers and brand marketers. However, manually tailoring content to resonate with specific audiences is labor-intensive and heuristic-driven. While recent generative models offer promising capabilities for automatic content generation, existing approaches for diffusion-oriented content generation still struggle to effectively translate numeric diffusion influence signals into actionable guidance that captures latent audience susceptibility and accounts for heterogeneous audience interests. To address these challenges, we propose DOCG-AS, a three-stage framework for diffusion-oriented content generation. It first performs implicit feature optimization on the realistic content manifold to discover an optimal propagation feature vector. Then, it explicitly decodes this vector using a learnable decoder into interpretable audience-susceptible features described in natural language, providing guidance for content generation. Finally, it leverages multiple sets of audience-susceptible features obtained from different optimization initializations to rewrite the user's input into the final multimodal content. Experiments demonstrate that DOCG-AS consistently outperforms state-of-the-art baselines in terms of predicted diffusion influence.

0.21

CropReward: A Human-Preference Benchmark and Reward Model for Aesthetic Image Cropping

cs.CV ·2026-10-25 ·Xiangfei Sheng, Xiaofeng Pan, Leida Li, Qian Zhang, Biao Wang +2 more
benchmark similarity 0.63
Abstract · 14 pages

Aesthetic image cropping (AIC) aims to improve image composition and aesthetic quality by removing extraneous or distracting content. Over the past decade, AIC methods have evolved from specialized vision models to multimodal large language models(MLLMs). Reliable evaluation that reflects human preferences is essential for benchmarking AIC methods and guiding model optimization. However, the current AIC evaluation relies largely on geometry-based metrics such as intersection over union (IoU), as well as image aesthetic assessment(IAA) models. Although inconsistencies with human preferences have long been observed, the robustness and reliability of these evaluation criteria remain underexplored. To bridge this gap, we first construct CropBench, a large-scale benchmark comprising 106,995 human preference pairs for 1,019 images, with each candidate set containing the original image and candidate crops produced by 14 AIC methods. A systematic study on CropBench reveals that the existing criteria, ranging from IoU to off-the-shelf IAA assessment models, only exhibit limited alignment with human preferences. Building upon CropBench, we further propose CropReward, an MLLM-based reward model that evaluates a candidate crop by comparing with the original uncropped image and outputs a score indicating explicit aesthetic improvement or degradation relative to the original. In addition, we employ CropReward to construct CropReward-300k, the largest AIC dataset to date, and further develop CropAlign, a preference-aligned AIC model trained solely with CropReward supervision, without human-annotated crop boxes. Experimental results demonstrate that CropReward aligns with human preferences significantly better than AIC existing criteria. CropAlign also surpasses the state-of-the-art AIC methods, demonstrating the advantage of CropReward in guiding AIC model optimization.

0.25

PAG-Agent: Zero-Shot Personalized Music Album Generation via Aesthetic Profile Distillation

cs.MM cs.SD ·2026-10-10 ·Yifeng Mao, Ziye Wang, Yutong Shen, Feiran Liu, Wenjuan Han
similarity 0.62
Abstract · 5 pages, 2 figures

Large Music Models (LMMs) can synthesize high-fidelity audio and support increasingly flexible prompt control, yet modeling the subjective preferences of an individual listener remains underexplored. We present PAG-Agent, a zero-shot multimodal framework that uses an explicit Personal Aesthetic Profile to bridge user preference inference and music generation control. A multimodal LLM distills favorite-song audio and contextual user information into this structured representation, which a planning LLM translates into generation instructions for existing music synthesizers. We further introduce PA-Eval, which evaluates personalization through the Aesthetic Consistency Score (ACS) for target-user alignment and the Aesthetic Specificity Index (ASI) for cross-user specificity. Experiments with Personal-100 on multiple MLLM and music-generation backbones show consistent improvements over generic and direct conditioning. A 50-participant follow-up study further shows a correlation $r=0.64$ between ACS and subjective preference.

0.29

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment

cs.CV ·2026-10-04 ·Junzhou Xie, Haozhong Xiong, Xunyun Tian, Kaile Du, Tianchen Yu +6 more
benchmark similarity 0.61
Abstract

Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent. These scores support data filtering, candidate selection, and preference modeling in image-generation pipelines. Existing methods typically predict a single aesthetic score or use general-purpose MLLMs without conditioning on photographic intent. This omission matters because the same blur, pose, lighting, or framing choice may serve one photographic intent but undermine another. These models thus learn context-agnostic aesthetic priors and yield inconsistent, inaccurate, misleading judgments for portraits with distinct photographic objectives. We introduce PortraitAes-Bench, an 11K-scale benchmark that decomposes this task into intent-conditioned subjudgments. Expert-authored rubrics define nine photographic intents, six first-level dimensions, and 22 secondary criteria. They support a structured pipeline for intent routing, specialist assessment, verification, and score fusion. Following this structure, we train PortraitAes with multi-task supervision. We then improve score comparability through Gaussian score calibration and within-dimension cross-image ranking. On the standard benchmark, PortraitAes achieves a Pearson correlation of 0.924 and a Spearman rank correlation of 0.934. On the hard-case set, its Pearson correlation is 0.829 and its Spearman rank correlation is 0.795. Across both sets, PortraitAes outperforms the evaluated general-purpose MLLMs and specialized aesthetic baselines.

0.22

Beyond Explicit Intent: Discovering and Shaping Artistic Intent through Human-AI Co-Creation

cs.AI cs.CVcs.HC ·2026-10-15 ·Renye Yan, Jialun Zhong, Jikang Cheng, Yi Gao, Yanzeng Li +1 more
similarity 0.61
Abstract · Human-AI Co-Creation, Reinforcement Learning

Generative AI is becoming an important medium for artistic creation, yet existing interactions often assume that creators can specify intentions in advance and accurately communicate them through text prompts. Artistic intentions, however, may be vague, latent, or recognizable only after seeing concrete visual possibilities. We propose a human-AI co-creation paradigm beyond explicit intent expression. By expanding model exploration, we expose artists to diverse visual possibilities, allowing them to identify previously unarticulated preferences through observation, comparison, and selection. We then integrate artists' curatorial judgments into reinforcement learning, assigning different learning strengths based on preferences over generation trajectories, so recognized directions are more strongly reinforced and the model gradually adapts to individual aesthetics. This creates a continuous "explore-identify-shape-re-explore" loop in which artistic intent is progressively discovered, externalized, and shaped. Through three user studies, we examine latent-preference discovery, personalized preference alignment, and the externalization of vague, difficult-to-express creative intentions.

0.37

DiffAuction: An Auction-Driven Framework for Generative Image Advertising

cs.CV ·2026-10-26 ·Yijie Su, Ruiyang Si, Tonghan Wang
similarity 0.61
Abstract

Diffusion models enable multiple advertisers to compete for exposure within a shared generated image. Yet existing approaches are computationally costly when generating images under varying target-share configurations, and it remains unclear how the resulting outcomes should be evaluated and priced. To address these gaps, we propose DiffAuction, a framework integrating controllable generation, click-intent evaluation, and auction design. First, our training-free Residual Share Calibration (RSC) decomposes joint-attention updates into product-specific residuals and rescales them according to target shares. We further introduce Share-Conditioned Prompting (SCP) to adapt the viewpoint to each desired allocation. Second, we develop an evaluator that uses a VLM to extract visual evidence and an LLM to assess Click-Through Intent (CTI), a proxy for click propensity, along four dimensions: Product Prominence, Brand Recognizability, Scene Integration, and Contextual Value Communication. We validate its assessments against judgments from 10 trained annotators on 1000 images. Third, we formulate this advertising setting as an auction over jointly feasible generation configurations. Each configuration has a public expected CTI vector, while advertisers hold configuration-specific private values per unit of CTI that map click intent into commercial value. We adopt a learnable affine maximizer auction (AMA), which jointly selects a shared generation configuration and determines advertiser-specific payments to maximize expected platform revenue. The resulting allocation and payment rules guarantee dominant-strategy incentive compatibility (DSIC) and individual rationality (IR). Experiments show that our generation method strengthens the monotonic relationship between target shares and CTI relative to baselines, while the learned AMA achieves higher platform revenue in the resulting auction setting.

0.25