CS Paper Radar

Design Creativity Bench: Measuring creativity in LLM-Generated UI

Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey

cs.HC cs.AI ·submitted 2026-10-08 09:09 UTC · v2
benchmark

Abstract

As leading LLMs improve on capability evaluations, their limitations in producing creative outputs on design tasks remain insufficiently characterised. Our work introduces Design Creativity Bench, a benchmark that evaluates diversity and appropriateness in UI designs. It measures distinctiveness among models on the same prompt (originality), how much a model's designs change between two prompts for the same UI goal in different product domains (creative range), and the share of a brief's acceptance criteria each design meets (appropriateness). Originality is 0.592 for same-prompt design pairs from different models (95% CI [0.582, 0.602]), far below the 0.764 for same-prompt human-model pairs (95% CI [0.751, 0.778]). Creative range is 0.581 across models (95% CI [0.567, 0.597]), against 0.902 for human designs (95% CI [0.884, 0.919]). Appropriateness is above 90% for every model in our test set, and the best model reaches 99.2%, slightly above the 98.0% for human designs. Our work shows that the default output of LLMs, though generally appropriate, is substantially more repetitive than the human baseline. This calls for strong measures to address the issue. A live leaderboard with newer models is maintained at https://kombai.com/benchmark/.

Comment
v2: added link to live leaderboard (https://kombai.com/benchmark/); clarified that appropriateness results apply to the evaluated models. Results unchanged. 29 pages (11 pages main text), 9 figures, 10 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Breaking Homogeneity: Diversifying Persona Sets for Creative LLM Outputs

cs.CL cs.AIstat.ML ·2026-09-24 ·Sang Bin Moon, Nicole Cho, Daniel Borrajo, Sumitra Ganesh, Abolfazl Hashemi
similarity 0.66
Abstract

Language models often produce homogeneous responses to open-ended tasks; such homogeneity can spawn groupthink-the convergence of ideas toward a singular and potentially suboptimal decision. We formulate persona diversification as a set-level conditioning problem and study two orthogonal design choices: selecting versus generating personas, and space-filling versus frontier-seeking diversity. We instantiate this design space with four methods spanning coverage and dispersion subset selections, uniform-coverage sampling, and evolutionary persona generation. Evaluations on the Alternative Uses Task (AUT), Infinity-Chat, and Divergent Association Task (DAT) show the benefits of the proposed methods across tasks and creativity objectives. On AUT, evolutionary persona generation increases response diversity by 78.8%, originality by 26.1%, flexibility by 49.5%, and holistic creativity by 13.9% over task-only prompting, while maintaining 98.5% validity; on Infinity-Chat, it nearly doubles persona-induced response separation relative to random personas. Moreover, evolutionary personas compose with creativity-optimized prompting, further increasing its response diversity by 18.6% and creativity by 6.3%. These results establish persona-set geometry as a task-agnostic mechanism for eliciting divergent LLM outputs, and support persona diversification as a reusable complement to prompt optimization.

0.19

Are LLMs becoming similarly creative? Evidence from three years of models

cs.CL cs.AIcs.CY ·2026-08-19 ·Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger
highly cited similarity 0.66
Abstract · 12 pages, 4 figures

Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.

0.61

DesignTaste: Benchmarking AI-Generator Visual Design Against Award-Wining Works

cs.CV cs.AI ·2026-11-12 ·Zhou Chen, JiaXiang Li, DanYing Pan, Ye Li, TianRui Yang +12 more
benchmarkcode/data similarity 0.62
Abstract · 31 pages, 12 figures, including appendices. Code and data: https://github.com/ArtArk-AI/DesignTaste

Text-to-image benchmarks commonly assess text-image alignment or overall preference. Whether models can produce good designs when a brief leaves the appearance open remains less well understood. We introduce DesignTaste, a visual design benchmark that compares generated images with award-winning works. From these works and their publicly available context, we derive 403 design briefs: 100 poster tasks from 100 Beste Plakate 2025 and 303 tasks spanning 20 categories from ADC 2026. Six leading commercial text-to-image models each generate an image from the same brief. A single model judge scores each output and the corresponding award-winning work independently on five evaluation dimensions. On each dataset, only ArtArk achieves a higher mean overall score than the award-winning works. All six models score lower in Creative Originality on both datasets. These results show that producing finished, visually appealing images does not imply that a model can develop good design solutions.

0.41

Evaluating LLM Creativity through Long-Tail Performance

cs.CL ·2026-10-09 ·Yichen Wang, Tenghao Huang, Vishakh Padmakumar, Mina Lee
benchmarkcode/data similarity 0.62
Abstract · 26 pages, including references and appendices. Code and data: https://github.com/YichenZW/longtail-creativity

Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text. We ground this proposition in the novelty-quality product formulation and evaluate it empirically across 15 benchmarks and 26 open-weight models, using n-gram absence from the pretraining data as a novelty proxy. Our results show that creativity benchmarks lie further in the tail regions than standard benchmarks. Moreover, on both creativity and standard benchmarks, model rankings from higher-tail subsets of each benchmark align more closely with human creativity judgments in LM-Arena. Building on these findings, we introduce q@Tail($α$), a lower-cost creativity diagnostic that measures mean task quality on the tail fraction of an existing benchmark. Applying it to creativity-promoting methods reveals substantial room to improve LLM creativity through stronger tail performance.

0.38

Enabling Creative Exploration for Vibe Design Agents

cs.AI ·2026-09-14 ·Yifan Zhang, Nghi D. Q. Bui, Georgios Evangelopoulos, Arnaud Benard
benchmark similarity 0.61
Abstract · 24 pages; revised exposition and evaluation, updated figures, and expanded appendix

Vibe design agents translate natural-language requests into interfaces, yet repeated generations often converge on similar visual directions. We present an inference architecture that makes design exploration an explicit stage before implementation. A pre-pass proposes structured design specifications with typicality weights, an external policy samples a direction, and the generator implements it under fixed decoding settings. Candidate elicitation draws on Verbalized Sampling. Applied to UI themes and visual-asset prompts, this architecture provides direct control over aesthetic exploration without changing code-generation temperature. Across 168 benchmark prompts, theme sampling expands observed selection coverage from one option to nearly three and increases rendered visual variation. For visual-asset prompts, the strongest observed LLM-judge win/loss ratio is 1.32, with only a small change in whole-screen similarity. Localized aesthetic choices can thus influence judged preference even when the overall screen remains similar. We further evaluate joint theme and asset sampling in a randomized online study conducted over nearly two months, covering 3.39 million tasks and 87,416 labeled conversations. These studies show how explicit design specifications support exploration of alternative UI concepts and help us evaluate their effects on rendered interfaces, judged preference, and user interaction.

0.26

AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces

cs.AI ·2026-09-28 ·Zhijie Deng, Ling Li, Junhao Ji, Siwei Lyu, Zhipeng Xu +5 more
similarity 0.61
Abstract

Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages. However, under professional design scrutiny, their behavior can differ substantially from that of human designers. In professional design practice, designers rely on a systematic set of aesthetic principles that consistently guide judgment, diagnosis, repair, and creation. A coherent aesthetic capability should therefore connect aesthetic judgment with design actions. Existing evaluations, however, typically assess these abilities in isolation, making it difficult to determine whether task-level success reflects a shared aesthetic understanding or merely fragmented task-specific competence. To address this gap, we introduce AUV-Bench, developed in collaboration with professional UI designers around 1,395 executable web interfaces and four tasks: aesthetic scoring, diagnosis, repair, and text-to-UI generation. The tasks share a pool of UIs and aesthetic principles, with diagnosis and repair further aligned on 660 controlled-degradation instances to enable instance-level analysis of judgment and action. Evaluation of 12 models reveals a capability imbalance: models show moderate agreement with professional designers in holistic aesthetic scoring, yet exact diagnosis-chain success peaks at only 24.7%. On the aligned diagnosis-repair cases, correct judgments and successful repairs do not consistently coincide, exposing a Judgment-Action Gap between identifying aesthetic problems and successfully acting on them. In open-ended generation, even leading models achieve only moderate aesthetic quality under human-calibrated evaluation. Overall, current models exhibit partial aesthetic competence, but still lack the fine-grained understanding and judgment-action coherence required for reliable UI design.

0.13

GUIDE: Designer-in-the-loop Authoring of Conformant Generative User Interfaces

cs.HC ·2026-09-18 ·Hyewon Lee, Ziying Wang, Aiden Moy, Saran Nagubandi, Jason Wu
hot direction similarity 0.60
Abstract · 24 pages, 13 figures, 5 tables

Generative User Interfaces (GenUIs) enable applications to generate interfaces on demand from user needs and context. Like conventional UIs, they must still reflect designers' intent and conform to requirements such as brand identity. Unlike conventional UIs, designers cannot directly specify or see every interface a GenUI may produce, making design intent harder to enforce. We introduce GUIDE (GenUI Development Environment), a system that lets designers continuously inspect and refine GenUI behavior as they create and edit interfaces. GUIDE uses designers' modifications and interactions to adapt GenUIs through prompt optimization and a novel adaptive conformance scoring model. We validate GUIDE's scoring model and system. The scoring model matched or outperformed proprietary LLM baselines on held-out comparisons of real and synthetic application screens. In a study with 12 UI/UX practitioners, participants found GUIDE effective and usable and significantly preferred aligned outputs over a strong baseline using exemplars and a model-generated design.md.

0.34

Does AI Save Time on Product Design? A Randomized Controlled Experiment of AI Prompt-to-Design Workflows

cs.HC cs.AI ·2026-09-22 ·Remy Stewart, Olabode Anise, Andrew Hogan, Augustus Griffin
similarity 0.58
Abstract

AI tools for digital product design now offer prompt-to-design capabilities, allowing designers and their non-designer colleagues to create prototypes through conversational workflows with large language models (LLMs). While these tools promise time savings, experimental evidence in product design remains limited compared with evidence from software engineering. We conducted a randomized controlled trial with 50 product designers and 50 product managers to evaluate prospective time savings from leveraging Figma Make in design work. Participants attempted three standardized design tasks with or without access to Figma Make. Among participants who completed the study tasks, access to Figma Make was associated with approximately 20% shorter completion times, with larger gains among product managers. Our findings suggest that prompt-to-design tools may enable product managers to further contribute to design work, while the benefits for professional designers may be task dependent.

0.19