CS Paper Radar

Measuring Preferred Representation in AI-generated Images with Community Scorers

Melanie F. Pradier, Camilla Longden, Anja Thieme, Rita F. Marques, Nari Johnson, Sidhika Balachandar, Xinnuo Xu, Linda Y. Wen, Martin Grayson, Katja Hofmann, Cecily Morrison, Daniela Massiceti

cs.LG ·submitted 2026-11-11 23:02 UTC · v1
dataset

Abstract

AI-generated images often misrepresent historically marginalised communities, with real-world consequences beyond the images themselves. However, conventional metrics fail to measure how communities want to be represented in AI-generated images. To address this, we introduce the concept of preferred representation. Building on previous work, we propose a measurement instrument that operationalises this concept through a Community Scorer - a model that evaluates a specific community's preferred representation in AI-generated images. We interrogate the validity of Community Scorers on three diverse disability communities that span the Global North and South. The results show that Community Scorers align significantly better with community ratings compared to other widely used text-to-image metrics. We then demonstrate the usefulness of these scorers in a downstream content filtering task. We release the Community Preference Dataset to enable others to optimise how Community Scorers can be scaled for culturally specific evaluation of AI-generated visual media.

Closest papers

Nearest neighbours by meaning across the whole index.

Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities

cs.CL cs.HC ·2026-08-29 ·Millicent Ochieng, Felermino D. M. A. Ali, Elizabeth A. Ankrah, Najeeb Gambo Abdulhamid, Migisha Boyd +6 more
top venue similarity 0.64
Abstract · Accepted to Findings of EMNLP 2026

In this paper, we examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectations of the communities they represent. We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities, combining quantitative annotations with qualitative focus group discussions. We find that cultural alignment depends not simply on recognizable cultural markers, but on how those markers fit social, linguistic, procedural, and visual context. From these evaluations, we develop a taxonomy of cultural alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment. We additionally evaluate five multimodal LLM judges to examine whether automated evaluation can approximate community-grounded judgments at scale. Judge reliability and score calibration vary substantially across communities, with no single judge performing consistently across all five settings. These findings motivate community-calibrated evaluation pipelines in which automated judges are validated against community judgments to determine where they can be trusted and where human review remains necessary.

0.44

Perceive, Refine, Reason: A Calibrated Pipeline for Measuring Indicators in Strategic Visual Communication on Social Media

cs.CV cs.CY ·2026-09-13 ·Weihong Qi, Chen Ling
similarity 0.63
Abstract

Visual content shapes audience perception and opinion on social media, and computational social science increasingly relies on automated tools to analyze images at scale. Yet a measurement gap persists: existing tools rely on predefined categories or produce only coarse image-level labels, while measuring which specific objects appear in an image, how prominently, and where in the frame remains difficult at scale. We introduce Perceive, Refine, Reason (PRR), a calibrated pipeline that turns flexible vision-language detectors into auditable measurement instruments for social-scientific research. PRR combines natural-language category prompts with pixel-level spatial refinement via the Segment Anything Model (SAM) and a multimodal LLM arbitration layer whose reasoning chains externalize domain knowledge and lower the expertise threshold for human-in-the-loop validation. A complementary three-tier auditability framework applies quantification learning to profile per-category reliability, support task-aligned configuration, and statistically correct prevalence estimates. Across four vision-language detectors and nine sociological categories, the pipeline yields substantial precision gains over zero-shot baselines, including a 43.3-point improvement for the strongest backbone. Applying PRR to 103,920 Facebook images from U.S. legislators during the 2024 election cycle and linking detections to DW-NOMINATE ideology scores, we find that more conservative legislators display U.S. flags as larger visual elements, with a weaker tendency toward peripheral placement, a spatial pattern invisible to binary detection. PRR provides computational social scientists with a model-agnostic toolkit for accessible, spatially-grounded, and correctable visual measurement.

0.11

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

cs.AI ·2026-09-14 ·Haonan Jiang, Guojian Zhan, Jiancong Xie, Shijun Wan, Dongiia Zhao +3 more
similarity 0.62
Abstract · 30 pages, 13 figures

Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users on Xiaohongshu, a mainstream Chinese image-sharing platform, continue to turn to other people for help with everyday visual questions. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 9 frontier VLMs, the highest short-answer accuracy is 52.8\%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0\%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.

0.21

Personalized Image Generation with Reasoning and Reflection

cs.CV cs.AI ·2026-09-30 ·Bo Ni, Ngoc N. Tran, Qinwen Ge, Franck Dernoncourt, Seunghyun Yoon +8 more
benchmark similarity 0.61
Abstract

Personalized image generation has remained narrowly focused on conditional synthesis from curated visual exemplars, rather than capturing who a user is. In practice, however, a user's personal context is much richer, comprising reviews, posts, images, captions, and metadata accumulated over time. A truly personalized generator should leverage this history to produce images aligned with the user's lifestyle and aesthetic preferences. To this end, we introduce the first unified benchmark for personalized image generation from user histories. The benchmark comprises two complementary tasks and a multi-axis evaluation protocol that assesses target fidelity, visual quality, user distinguishability, semantic alignment with the user's history, and task-specific utility. Grounded in real-world e-commerce and social media settings, the benchmark includes: (1) Personalized Scene Generation, which places a given object in a scene that reflects a user's preferences and lifestyle, motivated by personalized product presentation; and (2) Personalized Creative Generation, which generates a novel image on a specified topic that is faithful to a user's aesthetic and visual identity, motivated by social media content creation. We further propose PEARL, which couples a multimodal reasoner with a frozen image generator in an interleaved reason-reflect loop optimized with differential data reward. Across both tasks, PEARL outperforms strong baselines, achieving an average improvement of 15% across personalization metrics.

0.23

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

cs.AI cs.CV ·2026-09-03 ·Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein, Afsana Mimi +3 more
benchmarkcode/data similarity 0.60
Abstract

Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \textbf{\ours}, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for \textit{Semantic Alignment}, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.

0.55

The 2026 World Cup, Anime Edition: Production, Engagement, and Circulation of Generative Memetic Sports Content

cs.SI ·2026-10-10 ·Azneef Chowdhury, Ho-Chun Herbert Chang
similarity 0.60
Abstract · 15 pages, 16 figures, including appendix Project Website: wcanime-mu.vercel.app

Generative AI gives audiences new ways to portray real sporting figures through familiar fictional worlds. Expanded opportunities to create, however, need not broaden cultural representation or distribute attention evenly. We examine this tension through the supply, demand, and circulation of anime style football videos on Instagram. Using 2,021 posts and associated video and account data, we combine caption entity extraction, human-reviewed visual identity clustering, engagement analysis, and content-reuse networks. Representation is uneven across athletes, while Blue Lock, Naruto, and One Piece account for 60.1% of explicit franchise mentions. Among posts referencing generative tools, rankings by total engagement differ from rankings by average engagement per video, distinguishing aggregate visibility from response to individual posts. Reuse is extensive: 772 of 1,548 evaluated videos belong to 124 detected groups, including 71 spanning multiple known accounts. Shared material connects a large account component alongside smaller groups, and later observed uploads receive lower median normalized engagement than first observed appearances. These findings characterize generative cultural crossover as a practice organized around concentrated cultural resources, uneven attention, and extensive recirculation. More broadly, they distinguish opportunities for visual authorship from the processes through which particular representations become widely visible.

0.38

Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment

cs.CV ·2026-09-29 ·Yu Zhao, Jiarui Wang, Huiyu Duan, Ye Zhao, Jutao Tang +3 more
benchmark similarity 0.59
Abstract

With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts. While large multimodal models (LMMs) are increasingly adopted as evaluators, existing benchmarks typically study semantic understanding, quality perception, and authenticity identification in isolation, while largely neglecting responsibility detection. This leaves a gap in unified and comprehensive validation. To bridge this gap, we introduce SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility. SQUARE-Bench introduces a granular taxonomy of 38 sub-dimensions to evaluate nearly 10K AI-generated images sampled from 22 diverse models, ranging from legacy to state-of-the-art generators, complemented by over 3K real-world images. The images are annotated with curated question-answering pairs. Extensive experiments on 23 LMMs reveal that top proprietary models, such as Gemini-3-Pro, already outperform the individual human expert baseline. However, the performance gap between models remains significant, exhibiting notable disparities in fine-grained inference and domain-specific robustness. Beyond benchmarking, we conduct a proof-of-concept study of LMM-guided iterative editing, in which dimension-specific LMMs provide diagnostic feedback to fixed image editors. The resulting guided system yields selective improvements in semantics, authenticity, and responsibility, while exhibiting a consistent visual-quality trade-off. SQUARE-Bench can serve as both a diagnostic tool for characterizing LMM evaluator capabilities and studying their use in T2I generation refinement. The benchmark and dataset will be released upon publication.

0.24

Undertone: Personalizing Frozen VLM Aesthetic Judges from Few Ratings

cs.CV cs.CLcs.LG ·2026-09-22 ·Amit Jadhav, Shaurya Beriwala, Beomjin Kim
benchmark similarity 0.59
Abstract · v2: substantially revised and retitled; supersedes v1 ("Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language…

Image quality assessment predicts a mean opinion score (MOS), but for image aesthetics the mean hides the individual, so personalized assessment must start from the few images a new user rates. Fitting each person from scratch fails with ten ratings; networks fine-tuned per person carry over poorly to new kinds of images. We present Undertone-P, which learns from other raters a Gaussian prior over per-person regression weights on read-outs of frozen vision-language model (VLM) judges: the probability-weighted rating and the hidden state at the score token. Since people differ from the crowd in ways other raters reveal, a few ratings need only choose among those ways: a new user costs one closed-form solve, and no network is trained. On Ryu and Yanaka's PARA benchmark, the prior alone reaches a Spearman correlation of .640, above the best published 10- and 100-rating results (.581 and .620); 10 and 100 of a person's ratings raise it to .648 and .676. These ratings add most where tastes depart from the crowd: with 100 ratings, +.067 for the raters who agree least with it and +.206 on LAPIS artworks, against +.037 overall. Without labels, the judges' expected scores predict PARA's MOS at .835 (best published model we ran: .694). PARA-trained networks still lead under PARA's original protocol, likely through a stronger end-to-end MOS model.

0.36