CS Paper Radar

Language Models are Lazy Poets: Creative Mode Collapse at Scale

Jacob A. Matthews, Sil Hamilton

cs.CL ·submitted 2026-10-16 20:45 UTC · v1

Abstract

While aligned large language models (LLMs) can successfully complete many creative tasks, their outputs tend to be highly self-similar. Though solutions to this problem of "mode collapse" have been proposed, comprehensive large-scale surveys of the phenomenon in specific textual genres are still lacking. We address this gap through an extensive treatment of LLM mode collapse in poetry. Across 15 poetic forms, 248 subject matter tags, and 16 open-weight models, we generate over 1.1 million poems. To our knowledge, this constitutes the largest corpus of LLM-generated poetry to date. At million-poem scale, we find that LLM poetic vocabularies and rhyming abilities are substantial, but severely underutilized. In other words, LLMs are "lazy" poets: despite possessing large poetic inventories, LLMs over-rely on a small number of words, opening phrases, and rhymes. Further, we find that lexical homogenization differs across models and poetic forms, noting observed rhyming vocabulary is correlated with model size.

Closest papers

Nearest neighbours by meaning across the whole index.

Poetry 'resurrection' from LLMs

cs.CL ·2026-11-12 ·Vincent Cellucci, Peter van der Putten
publishedsurveybenchmark similarity 0.73
Abstract · 8 pages, 8 figures, to be published in ISEA 2026 conference proceedings

In an era where linguistic creativity is challenged by algorithmic authorship, we pose a provocative question: can artificial intelligence (AI) write an infinite poem? We set up two experiments to investigate the quantitative and qualitative output produced by AI utilizing couplets that evidence a poetic relationship between end words, the only variation in a strict anaphoric, litany structure. Our results show that while AI can generate interesting pairings, it still falls short of human-level creativity and understanding of poetic relationships. We offer commentary and a brief survey of the field of computational poetry experiments, tracing back authorial intention and distinctions of 'natural' and 'artificial' poetry, as well as justifying why couplet form is suited to testing AI's poetic output. Experiment 1 relies on manual prompting and co-creation to extend the original poem, which is developed into an animated webpage installation and interactive artwork. Experiment 2 reliably achieves automated output reaching the 1000 benchmark via an alternative method of using an LLM in combination with semantic similarity, of which Method 2 demonstrates a higher capacity for semantic and quantitative poetic output. These experiments highlight the limitations of current LLMs' generation capacity at the word level but also suggest that poetry will also respond and revive by incorporating AI workflows.

0.23

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

cs.CL ·2026-08-29 ·Keunhyeung Park, Seunguk Yu, YoungBin Kim
top venuehot direction similarity 0.68
Abstract · Accepted to Findings of EMNLP 2026

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

0.55

Single-Sample Decoding of a Two-Layer Sound-Word Canvas Model for Thai Klon Pad

cs.AI ·2026-10-10 ·Worasait Suwannik
similarity 0.66
Abstract

We present a generator of Thai klon pad verse, one stanza from a short topic, and evaluate it against human verse on held-out topics. The generator is a masked language model initialised from WangchanBERTa over a fixed canvas with two layers: a sound layer of pronounced syllables, where rhyme, tone and syllable count are enforced as hard masks, and a word layer of whole words, so that no word is cut into written pieces. Decoding commits one cell at a time and samples at temperature 0.4; one sample is drawn per topic, with no selection, reranking or repair. On 78 held-out topics never used in development, judged by three independent runs of a frozen LLM judge in the same pools as the human stanzas from which the topics were written, 70.5% of the generated stanzas pass form, lexical cleanness of all four lines and topic alignment together, against 76.9% for the human stanzas (difference -6.4 points, 95% CI [-19.2, +6.4]). Cleanness is level with human (85.9% vs. 87.2%), and the stanzas pass form and cleanness together slightly more often than the human ones (+3.8 points, [-6.4, +14.1]); the difference lies in alignment (84.6% vs. 91.0%), on topics that were written from the human stanzas. The low temperature matters: at higher temperatures the same model passes clearly fewer of the same topics, because lexical errors lie in the model's lower-probability choices; sharpening the sampling distribution removes most of them, at a cost in vocabulary (7.5% rare words against 16.5% in the human stanzas) and diversity. The pre-registered target, all-three within 5 points of human, is missed by 1.4 points. KlonSuphap-LM, the only public klon pad generator we know of, passes none of the 78 held-out topics when judged in separate pools together with the method's and the human stanzas (form 19.2%, cleanness 3.8%).

0.21

Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

cs.CL ·2026-10-12 ·Ankit Bhattacharjee, Indrasabarna Hari, Krityapriya Bhaumik, Arka Das, Piyush Sarangi
similarity 0.64
Abstract

While language models provide powerful mechanisms for evaluating textual predictability, it remains unclear whether the relative extents of LM surprisal remain stable across models. This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources. We consider two stylistic environments: a corpus of contemporary poetry and a corpus of academic research papers. We demonstrate that mean and variance of cross-entropy maintain robust cross-model rank correlation ($ρ> 0.90$) across both regimes. To test whether this agreement merely reflects generic differences in textual complexity, we compare it against six parameter-free compression baselines. The extent of agreement among language models is found to be substantially higher than that among model-baselines, for most of the measures. Geometric analysis of the structurally constrained academic regime reveals that higher-order moments are sensitive to specific models, although strong agreement persists. The findings suggest that relative extents of textual unpredictability may be independent of the language model they are computed against.

0.27

HaikuS2S: A Cascaded System For Responding In Verse

cs.CL cs.AIcs.SD ·2026-09-20 ·Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe
published similarity 0.63
Abstract · Accepted to SLT 2026, Demo Track. 5 pages, 5 figures

Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior work on prosody transfer improves expressiveness, and fine-tuned poetry TTS (text-to-speech) systems capture verse intonation. However, these models do not model haiku's 5-7-5 syllable structure or line-ending pauses. We present a cascaded system, HaikuS2S, combining ASR (automatic speech recognition), LLM (large language model)-generated haiku, and TTS fine-tuning on both prose and custom haiku datasets. Our evaluation focuses on emotion similarity, speech quality, and prosody alignment. In our experiments, we see that our prosody and tonal alignment improve significantly with our fine-tuned systems, particularly the one trained on both general poetry and haiku. We also see that we maintain similar emotion similarity scores across all systems.

0.32

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

cs.CL ·2026-08-22 ·Tian Lan, Shanshan Wang, Zehua Duo, Jiang Li, Guanglai Gao +2 more
benchmark similarity 0.63
Abstract · 20 pages, 2 figures

Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks, yet their ability to understand literary texts, particularly modern Chinese poetry, remains largely unexplored. The unique literary characteristics of modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension. Unlike conventional texts that convey clear information, the unique "poetic logic" of modern Chinese poetry requires a holistic reasoning approach that goes beyond superficial semantic analysis to be understood. However, current evaluation paradigms largely ignore this critical dimension. To address this gap, we propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry. We define poetic logic as four tasks across three levels, namely stanza, line, and imagery, and systematically evaluate and analyze six mainstream LLMs based on Peony. We evaluate these models under both non-thinking and thinking configurations. The experimental results reveal the limitations of current LLMs in understanding the poetic logic of modern Chinese poetry and validate the effectiveness and necessity of Peony. Our data and code will be available.

0.30

Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?

cs.CL cs.AI ·2026-09-23 ·AbdulRahman A. Morsy, Aya Zirikly
similarity 0.62
Abstract · 14 pages

Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored. Prior work has focused largely on modern language varieties and poetry, while classical prose traditions such as maqama remain largely unstudied. The maqama is a classical literary genre characterized by rhymed prose (saj), dense rhetorical ornamentation, and episodic narrative structure, making it a challenging testbed for evaluating whether LLMs can move beyond surface fluency toward deeper literary competence. In this paper, we present the first controlled evaluation study of maqama generation with LLMs, comparing five models under zero-shot, few-shot, and rule-based prompting, and evaluating outputs through both human annotation and an LLM-as-a-judge framework across dimensions such as rhetorical richness, saj density, structural coherence, and stylistic authenticity. Our results show that prompting strategy plays a strong role in stylistic quality: few-shot prompting most consistently improves saj density, while its effects on rhetoric and coherence vary by model, with the strongest models (GPT-4o and GPT-5.4-mini) benefiting most from rule-based prompting on these dimensions, though zero-shot prompting yields the highest aggregate scores across all five models. We further observe systematic differences between models in stylistic alignment with Arabic maqama conventions, and corroborate our findings with a second independent LLM judge, paired statistical significance testing, and non-LLM proxy measures of saj.

0.18

Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku

cs.CL cs.AI ·2026-09-14 ·Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese
published similarity 0.61
Abstract · 18 pages, 10 figures. Accepted as a full paper at the 2026 Joint 14th International Conference on Soft Computing and Intelligent Systems and…

This paper investigates the generation and human evaluation of Japanese haiku by contemporary Large Language Models (LLMs), focusing on authorship perception and aesthetic judgment within a constrained poetic form. Using a few-shot prompting strategy, Japanese haiku were generated across a heterogeneous set of large language models, including open- and closed-source systems, medium-scale and large-scale architectures, models with native or adapted Japanese support, and multilingual proprietary models. These AI-generated haiku were combined with human-written ones and presented in a questionnaire distributed to students at Japanese universities in Tokyo. The survey assessed whether respondents could distinguish between AI-generated and human-written haiku and which cues informed their judgments. Recognition accuracy varied across models. GPT-5, Gemini 2.5, and StableLM-7B performed at approximately chance level (approx 0.50), whereas LLM-JP, Gemma-2B, and LLaMA-2 showed moderate detectability (approx 0.59-0.67). However, recognition was strongly item-dependent. Ratings of fluency, coherence, poeticness, and related aesthetic dimensions predicted perceived humanness but not correct classification, indicating an attribution bias linked to aesthetic evaluation and revealing a dissociation between aesthetic evaluation and true authorship detection. The extended analysis additionally examines generation-constraint adherence, participant-level characteristics, and exploratory LLM-based evaluations of haiku authorship. Overall, the findings suggest that as LLMs improve, surface-level creative plausibility may reduce reliable human discrimination within constrained poetic settings.

0.35