CS Paper Radar

Thinking Outside the Bots: Human Ideas in AI's Semantic Blind Spots Are Not Inferior

Kibum Moon, Dan R. Johnson, Kostadin Kushlev, Benjamin Lira, Indre Viskontas, James C. Kaufman, Roger E. Beaty, Angela L. Duckworth, Adam E. Green

cs.CL cs.AIcs.CY ·submitted 2026-10-12 19:31 UTC · v1

Abstract

AI-generated outputs are frequently perceived as highly creative, but also tend to be more similar to each other (homogenized) than human ideas. Does the human heterogeneity that extends beyond AI reflect valuable thinking that AI misses? Alternatively, is greater human heterogeneity driven by lower-quality ideas and less effective thinking that AI avoids? We developed Expansion of Divergent Generative Expression (EDGE) to map AI's semantic "blind spots" by assessing how strongly/weakly individual human ideas were represented in a task-matched AI semantic comparison set. Specifically, we compared human-generated vs. AI-generated ideas across three open-ended tasks: college admissions essays, alternative uses for everyday objects, and interpersonal advice. Across all three tasks, human responses extended into semantic regions sparsely covered by AI outputs. These distinctively human responses showed no meaningful deficits in creativity, quality, or effectiveness, were associated with equal-or-better academic performance, and with credibly superior ratings and achievement for some outcomes. These ideas were thus not merely poor-performing variation; they performed as well as or better than ideas that overlapped more strongly with AI ideas. More distinctive human ideas tended to receive higher creativity ratings, but this relationship was weak or reversed for AI ideas. Taken together, these results suggest that differentiation among human-generated ideas does not reflect poor thinking that AI can safely omit, and instead may afford both individual and population-level advantages.

Closest papers

Nearest neighbours by meaning across the whole index.

AI-AI co-creation outperforms human pairs in creative tasks

cs.CY ·2026-08-10 ·Yingyue Luna Luan, Luning Sun, Yeun Joon Kim, Jindong Wang, Xing Xie
position similarity 0.62
Abstract · 19 pages, 3 figures

Prior research often finds that AI creativity is limited: single systems rarely outperform humans, and human-AI collaboration does not exceed human output. We argue these conclusions underestimate AI's potential because most studies do not allow iterative, multi-agent exchanges that mirror the social processes underpinning human creativity. We conducted an experiment comparing four conditions: (i) AI-AI co-creation with complementary generator-evaluator roles, (ii) AI-AI co-creation with identical roles, (iii) single-AI creation, and (iv) human-human co-creation. Across three open-ended tasks, 1,212 ideas were rated by trained judges on creativity, novelty, and usefulness. Both AI-AI co-creation conditions consistently outperformed single-AI creation and human pairs on creativity and novelty. Usefulness varied by task: complementary roles yielded the most useful solutions in the broadest and most socially complex task, suggesting role differentiation is advantageous when problems require both imaginative ideation and practical refinement. Human pairs performed worst, consistent with production losses in group creativity. These findings indicate that structured, iterative multi-agent AI co-creation can exceed single-AI and human-human ideation.

0.10

Confident, Not Wiser: The Dunning-Kruger Effect in Human-AI Interaction

cs.HC ·2026-09-25 ·Daniela Fernandes, Michelle Rausch, Agnes Mercedes Kloft, Daniel Buschek, Robin Welsch
similarity 0.62
Abstract · 37 pages, 10 figures, 12 tables

AI assistance can improve performance without improving self-assessment. We report a study (N=366) comparing Human alone and Human+AI performance on reasoning tasks, for which the AI model is benchmarked on the same items. Participants estimated global and block performance and rated confidence in their answers. Human+AI achieved higher scores, but self-estimates tracked performance weakly. Average overestimation was similar across groups, covering individual errors. Across tasks, confidence distinguished correct from incorrect answers less accurately in the Human+AI group, while within-task differences remained uncertain. The Dunning-Kruger pattern was found in both groups, with a larger observed contrast in Human+AI. Controls for score noise reduced but did not eliminate the pattern, with the controlled group difference remaining inconclusive. An extended computational account describes global and block estimates. Our findings distinguish performance augmentation from metacognitive augmentation and motivate interfaces that support verification, communicate task-specific AI model performance, and help users evaluate the quality of their joint work rather than produce answers.

0.14

Process-Constituted Intelligence: A Shared Criterion for Humans and Machines

cs.AI cs.ET ·2026-08-17 ·Michael J. Richardson, Manuel Varlet, Ayeh Alhasan, Cassandra Crone, M. Paula Diaz Monfort +4 more
hot direction similarity 0.62
Abstract · v2: revised throughout for clarity and consistency; supplementary cross-references corrected

Intelligence is constituted by \textit{process} (iterative activity through which output emerges), not in the output itself. Generative AI (GenAI) is trained on \textit{traces} (textual and visual residues of human cognitive processes), reproducing samples from a distribution of those traces. Its outputs resemble reasoning, problem-solving, and creativity, yet the activity that produces such outputs in humans remains largely absent. Current GenAI is, therefore, weakly equivalent to the cognition it imitates, matching outputs while process stays absent or opaque. The cognitive sciences have long distinguished between weak and strong equivalence. Here, we define \textit{strong} equivalence across seven process features, assessable against human and machine cognition. Our process-based account addresses a symmetric risk: GenAI tools that outsource a person's generative processes may leave critical capacities unbuilt. We specify design principles for GenAI that instantiate more process and preserve rather than erode human judgment and creativity, and outline process audits that make strong equivalence testable.

0.29

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

cs.CL ·2026-11-02 ·Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar +1 more
benchmarkcommunity pick similarity 0.61
Abstract

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.

0.28

Are LLMs becoming similarly creative? Evidence from three years of models

cs.CL cs.AIcs.CY ·2026-08-19 ·Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger
highly cited similarity 0.60
Abstract · 12 pages, 4 figures

Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.

0.61

The Limits of Automatic Evaluation of Creativity in Large Language Models

cs.CL cs.AIcs.CY ·2026-08-24 ·Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
similarity 0.60
Abstract

Large Language Models (LLMs) are increasingly capable of generating text that challenges human performance in domains requiring creativity, yet evaluating creativity in LLM-generated content remains a significant challenge. Here, we investigate whether current automatic evaluation methods can reliably capture human judgments of creativity. We collect human evaluations of human- and AI-generated short stories from the WritingPrompts dataset across 11 dimensions of creativity, and compare these judgments with automated objective metrics and LLM-as-a-Judge evaluations. Our experiments reveal substantial misalignment between automatic evaluations and human assessments. In particular, LLM-based judges exhibit a systematic preference for AI-generated stories, consistently favoring their stylistic characteristics over the unpredictability and other qualities of human-authored texts. Furthermore, correlation analyses show that widely used automatic metrics exhibit near-zero alignment with human judgments across both human- and AI-generated stories, suggesting that they fail to capture important dimensions of creativity. These findings highlight fundamental limitations in current approaches to the automatic evaluation of creative text and underscore the difficulty of reducing the multidimensional and subjective nature of creativity to computational metrics.

0.38

Beyond Better Ideas: How Human-AI Collaboration Shapes Scientific Ideas and Human Perceptions

cs.HC ·2026-10-22 ·Qian Chen, Saatvik Pradhan, Janice Ahn, Berk Atil, Angkai Li +5 more
similarity 0.59
Abstract · 23 pages, 10 figures, 9 tables

As AI increasingly becomes a collaborator in scientific research, its impact may extend beyond improving ideas to reshaping how they are evaluated, attributed, and perceived by humans. Yet existing research has largely examined these dimensions in isolation. We present a unified empirical study of Human-AI scientific ideation, investigating two complementary perspectives: i) how AI collaboration shapes scientific ideas and their evaluation, and ii) how firsthand collaboration reshapes human perceptions of AI. Through a shared experimental setting involving Human-AI and Human-Human research teams, human and LLM evaluators, collaboration-provenance analysis, and participant surveys, we uncover several counterintuitive findings. First, AI collaboration shows no significant advantage in human-evaluated idea quality, while iterative revision consistently improves quality. Second, AI judges can approximate human scores yet disagree on idea rankings; notably, an AI judge favors AI-assisted proposals without a corresponding advantage in human evaluations. Third, collaboration provenance remains only partially detectable, revealing substantial overlap between Human-AI and Human-Human scientific outputs. Finally, firsthand AI collaboration is associated with greater perceived usefulness but smaller efficiency gains than expected, suggesting a nuanced recalibration of human expectations. Overall, our findings challenge the notion that AI's value in scientific ideation lies primarily in producing better ideas, revealing instead its broader influence on scientific creation, evaluation, attribution, and human perceptions.

0.10

Cross-Modal Associations in Creativity: Evaluating Word-Image Pairing in Humans and Large Language Models

cs.HC ·2026-08-23 ·Eva Illion, Linxuan Wang, Liane Gabora
published similarity 0.59
Abstract · 6 pages

Given differences in the associative memory architectures of humans and AI, it is not clear whether AI generates and interprets visual-linguistic associations the way humans do. To investigate this, human participants and two large language models (LLMs), ChatGPT and Gemini, were asked to generate novel words and corresponding images that form meaningfully linked dyads. The LLMs and a second group of human participants were presented with shuffled sets of these words and images and asked to rematch them in ways they felt were most meaningful. Humans performed similarly regardless of whether the pairs were human or LLM-generated, while LLMs were better at matching LLM-generated pairs. Overall, LLMs outperformed humans at recovering the intended pairings. These findings suggest that despite architectural differences, the way in which AI generates and detects associations between language and visual stimuli is not only like that of hu-mans, but they may be superior at recognizing cross-domain associations that humans find meaningful.

0.32