CS Paper Radar

Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions

Difang Huang, Shumin Yu, Fadong Chen, Weihua Zhou

cs.AI econ.GN ·submitted 2026-10-10 15:02 UTC · v1
position

Abstract

Generative AI makes options cheap and shifts the bottleneck of creative work from generating options to screening them. We argue that AI's value depends on whether its options arrive before or after the team's screen. In our model, early options compete for a place in the portfolio; late options arrive after that competition and are kept unscreened. In a preregistered experiment, we randomly assigned 148 three-person teams to five schedules of access to the same language model during a product-design task with timed search and selection stages. Moving the same 15 minutes of access from search to selection lowered expert-rated creativity by 0.62 standard deviations, although teams used the model at similar rates. Platform logs locate the difference at the screen. Early access enlarged the pool of ideas fivefold, and teams pruned it. Late-access teams reached the screen with the same small pool as teams without AI and then searched again, devoting 29% of their selection-stage queries to generating new options, compared with 11% for teams with access throughout. Late access added breadth, not quality: these teams submitted 73% more proposals than did teams without AI. Language-model judges detected the benefit of access but not the cost of late timing.

Closest papers

Nearest neighbours by meaning across the whole index.

We Ran a Research Sprint With AI. The AI Worked. The Research Didn't

cs.AI cs.LG ·2026-11-02 ·Anush Sankaran, Arijit Banerjee
position similarity 0.64
Abstract

Generative AI tools for research are widely deployed, yet individual productivity gains do not guarantee better team research. We argue that how AI use is structured matters alongside access. Personal AI tools may amplify two failures under deadline pressure: a consumption bottleneck, in which generation outpaces team-visible evaluation, and AI-induced research silos, in which private human-AI loops displace shared decision-making. We motivate this position with a mechanism-generating pilot: a 100-minute research sprint with twelve researchers in three independently operating teams, within a seventeen-person session. Both patterns recurred across teams and evidence sources. Of 33 logged AI interactions, one was accepted outright and 27 lacked a recorded disposition; missing records alone do not establish absent evaluation. A parallel autonomous pipeline produced a complete draft without in-session human evaluation, illustrating production capacity rather than comparative quality. We propose Human-AI-Human collaboration: an interaction topology in which AI connects researchers' contexts and supports inspectable collective evaluation and decision-making. The pilot does not isolate AI-specific effects from time pressure or workflow practices. We close with seven modular, falsifiable design hypotheses, prioritizing shared context and team-visible evaluation checkpoints for controlled follow-up.

0.23

Does AI Save Time on Product Design? A Randomized Controlled Experiment of AI Prompt-to-Design Workflows

cs.HC cs.AI ·2026-09-22 ·Remy Stewart, Olabode Anise, Andrew Hogan, Augustus Griffin
similarity 0.64
Abstract

AI tools for digital product design now offer prompt-to-design capabilities, allowing designers and their non-designer colleagues to create prototypes through conversational workflows with large language models (LLMs). While these tools promise time savings, experimental evidence in product design remains limited compared with evidence from software engineering. We conducted a randomized controlled trial with 50 product designers and 50 product managers to evaluate prospective time savings from leveraging Figma Make in design work. Participants attempted three standardized design tasks with or without access to Figma Make. Among participants who completed the study tasks, access to Figma Make was associated with approximately 20% shorter completion times, with larger gains among product managers. Our findings suggest that prompt-to-design tools may enable product managers to further contribute to design work, while the benefits for professional designers may be task dependent.

0.19

AI Tools Adoption across the Double Diamond Workflow: Phase, Mode, and Barriers in Designer Practice

cs.HC ·2026-09-28 ·Sepideh Tajarmakan, Khashayar Hojjati Emami
similarity 0.62
Abstract

Designers are adopting AI faster than the tools built for them can keep up. General-purpose conversational AI led AI use among designers in three of the four phases of design work, ahead of tools built for design. We surveyed 443 designers from at least 43 countries about AI use in each phase of the Double Diamond: Discover, Define, Develop and Deliver. Tools were classified by output and function. Of these designers, 79.7% confirmed AI use in at least one phase. Adopters used AI in a mean of 2.89 phases, and only 35.1% used it in all four. They added phases in workflow order, from Discover onward. Conversational tools led in Discover, Define and Deliver. AI-native image generation led in Develop, where AI-native tools peaked at 57.6% of tool picks. Embedded AI features in existing design software peaked later, in Deliver (37.5%). Fewer designers used AI in later phases, from 318 in Discover to 182 in Deliver. Those who did named a similar number of tools per phase. Low perceived usefulness was the only barrier that separated non-adopters (40.5%) from adopters (16.4%) after correction for multiple tests. Concern about accuracy was similar in both groups.

0.09

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

cs.CL cs.AI ·2026-10-29 ·Art Kanke
code/data similarity 0.58
Abstract · 27 pages, 1 figure, 9 tables. Code and data: https://github.com/ArtKanke/rTaste

Do research agents have taste, the judgment of which experiments are worth running before their outcomes are known? We give six frontier models seven recent NLP papers. Each model, working as a research agent alone or as a team, proposes minimum viable experiments (MVEs), forecasts whether each will pass a threshold it sets and ranks them. A separate executor then attempts every MVE, including those ranked last, and three model auditors judge the results. Of 415 MVEs, 194 are judged sound, valuable and new, but fewer than half of these (73) are also judged beyond the seed paper and non obvious. The agents' first choices match the auditors' most valued experiment more often than a random pick, but are not shown to improve on choosing the experiment written first. Their forecasts beat a base rate but are not measurably more accurate than those of a fresh instance of the same model given only the paper and registrations. Teams cost more without a clear overall improvement in proposal quality. The agents produce many worthwhile experiments, but the evidence that their rankings improve the choice of which to run remains limited. The evaluation harness and a dataset of all 415 MVEs, with their forecasts and audited outcomes, are available at https://github.com/ArtKanke/rTaste.

0.47

PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

cs.AI cs.CLcs.HC ·2026-09-28 ·Shane K. A. Dalumura Hettige, Jonas Oppenlaender
benchmark similarity 0.58
Abstract · 25 pages, 7 figures, 11 tables

Figural divergent thinking is the ability to develop a given shape fragment into an original drawing. In humans, this ability is assessed with incomplete-drawing tasks. We introduce PainterBench, a benchmark that ports the incomplete-drawing task to the agentic setting. The agent draws on a canvas through tool calls and observes the result after every turn. The canvas includes a starting shape which cannot be erased, and the agent's goal is to incorporate this shape into the most original drawing it can produce. The task is open-ended, and the agent itself decides when the drawing is finished. The benchmark tests incremental visual planning over a short horizon and the transfer of creative ability from pretraining to multi-turn tool use. We evaluate 14 multimodal language models from small to frontier scale. Across the primary study and six sensitivity analyses, we collect 2,700 drawings and crowdsource creativity and recognizability ratings for every drawing and for 300 human reference drawings. We also present ViDrA-adapted, an automated scorer that predicts human creativity ratings of agent drawings (r = 0.85 on random held-out test split). Figural divergent thinking varies widely across the 14 models, and GPT-6 Astra produces the most creative drawings. Relative to the human drawings, the agent drawings score higher in creativity but lower in recognizability. We release the final drawings, per-round canvas snapshots, tool call traces, stimulus bank, benchmark harness, crowdsourced ratings (N = 72,000), and ViDrA checkpoint.

0.25

Exploratory creative pre-ranking: Hierarchical empirical-Bayes Thompson sampling for real-time creative optimization

cs.LG cs.GT ·2026-10-26 ·Jiale Han, Ajith Moparthi, Dan Xie, Xiaowu Dai
similarity 0.58
Abstract

Dynamic creative optimization decides which creative of an ad to render for each request. Recent work places that choice before ad ranking, but relies on deterministic point predictors with no calibrated uncertainty. AI-generated content has made creatives nearly free, while feedback per creative remains sparse and selection faces a tight latency budget. We introduce a creative pre-ranker that scores every recalled creative and shortlists each campaign's pool before the per-user ranker runs. Creatives sit in a hierarchy of audience segment, advertised item, campaign and creative, so a cold creative falls back on what its parent levels have earned, and how much it borrows from its parents is set in closed form from historical logs, with one parameter per level and nothing tuned. To stay within the latency budget, the pre-ranker predicts for an audience segment rather than an individual user, from per-attribute counters. It selects by a single Thompson draw per creative, balancing exploration and exploitation so that a new creative still has a chance to be learned. Replaying Roblox Sponsored-ads traffic over campaigns with competing creatives, the shortlist lifts predicted play-through rate by 8.14% against 2.80% for an equally sized random shortlist, forwards more than 40% fewer creatives than the full pool, and adds only 0.2% to serving time. After an A/B test on live traffic, the method now serves in production at Roblox.

0.30

Design Creativity Bench: Measuring creativity in LLM-Generated UI

cs.HC cs.AI ·2026-10-08 ·Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey
benchmark similarity 0.58
Abstract · v2: added link to live leaderboard (https://kombai.com/benchmark/); clarified that appropriateness results apply to the evaluated models. Re…

As leading LLMs improve on capability evaluations, their limitations in producing creative outputs on design tasks remain insufficiently characterised. Our work introduces Design Creativity Bench, a benchmark that evaluates diversity and appropriateness in UI designs. It measures distinctiveness among models on the same prompt (originality), how much a model's designs change between two prompts for the same UI goal in different product domains (creative range), and the share of a brief's acceptance criteria each design meets (appropriateness). Originality is 0.592 for same-prompt design pairs from different models (95% CI [0.582, 0.602]), far below the 0.764 for same-prompt human-model pairs (95% CI [0.751, 0.778]). Creative range is 0.581 across models (95% CI [0.567, 0.597]), against 0.902 for human designs (95% CI [0.884, 0.919]). Appropriateness is above 90% for every model in our test set, and the best model reaches 99.2%, slightly above the 98.0% for human designs. Our work shows that the default output of LLMs, though generally appropriate, is substantially more repetitive than the human baseline. This calls for strong measures to address the issue. A live leaderboard with newer models is maintained at https://kombai.com/benchmark/.

0.35

Who Thinks First? Designing Productive Friction with Engage-to-Unlock GenAI

cs.HC ·2026-10-01 ·Xiaotian Su, Laura Rimell, Jiazheng Li, Amal Rannen-Triki, Ulrich Paquet +4 more
similarity 0.57
Abstract · 31 pages, 7 figures, 3 tables

Generative AI can support writing, but frictionless access may cause cognitive offloading before users develop their own ideas. We introduce Engage-to-Unlock, a productive-friction mechanism that unlocks generative capabilities after users meaningfully engage with the task. In a controlled experiment (N = 398), participants completed a writing task under one of four conditions: Human-Only, Standard Chatbot, Engage-to-Unlock, or Time-Matched Unlock, which matched unlock timing to Engage-to-Unlock participants but independent of users' engagement, then evaluated passages for evidence and inferential errors. Results show that Engage-to-Unlock redistributed effort across tasks: participants spent more time writing and less time evaluating, without increasing overall task duration. They also submitted more prompts than in other AI-assisted conditions and showed the highest accuracy-per-time evaluation efficiency across conditions. These findings suggest that designing GenAI access to encourage early human engagement may provide a productive form of friction, while retaining active AI use and efficient downstream evaluation.

0.07