CS Paper Radar

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Keunhyeung Park, Seunguk Yu, YoungBin Kim

cs.CL ·submitted 2026-08-29 01:45 UTC · v1
top venuehot direction

Abstract

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

Comment
Accepted to Findings of EMNLP 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Language Models are Lazy Poets: Creative Mode Collapse at Scale

cs.CL ·2026-10-16 ·Jacob A. Matthews, Sil Hamilton
similarity 0.68
Abstract

While aligned large language models (LLMs) can successfully complete many creative tasks, their outputs tend to be highly self-similar. Though solutions to this problem of "mode collapse" have been proposed, comprehensive large-scale surveys of the phenomenon in specific textual genres are still lacking. We address this gap through an extensive treatment of LLM mode collapse in poetry. Across 15 poetic forms, 248 subject matter tags, and 16 open-weight models, we generate over 1.1 million poems. To our knowledge, this constitutes the largest corpus of LLM-generated poetry to date. At million-poem scale, we find that LLM poetic vocabularies and rhyming abilities are substantial, but severely underutilized. In other words, LLMs are "lazy" poets: despite possessing large poetic inventories, LLMs over-rely on a small number of words, opening phrases, and rhymes. Further, we find that lexical homogenization differs across models and poetic forms, noting observed rhyming vocabulary is correlated with model size.

0.19

Single-Sample Decoding of a Two-Layer Sound-Word Canvas Model for Thai Klon Pad

cs.AI ·2026-10-10 ·Worasait Suwannik
similarity 0.64
Abstract

We present a generator of Thai klon pad verse, one stanza from a short topic, and evaluate it against human verse on held-out topics. The generator is a masked language model initialised from WangchanBERTa over a fixed canvas with two layers: a sound layer of pronounced syllables, where rhyme, tone and syllable count are enforced as hard masks, and a word layer of whole words, so that no word is cut into written pieces. Decoding commits one cell at a time and samples at temperature 0.4; one sample is drawn per topic, with no selection, reranking or repair. On 78 held-out topics never used in development, judged by three independent runs of a frozen LLM judge in the same pools as the human stanzas from which the topics were written, 70.5% of the generated stanzas pass form, lexical cleanness of all four lines and topic alignment together, against 76.9% for the human stanzas (difference -6.4 points, 95% CI [-19.2, +6.4]). Cleanness is level with human (85.9% vs. 87.2%), and the stanzas pass form and cleanness together slightly more often than the human ones (+3.8 points, [-6.4, +14.1]); the difference lies in alignment (84.6% vs. 91.0%), on topics that were written from the human stanzas. The low temperature matters: at higher temperatures the same model passes clearly fewer of the same topics, because lexical errors lie in the model's lower-probability choices; sharpening the sampling distribution removes most of them, at a cost in vocabulary (7.5% rare words against 16.5% in the human stanzas) and diversity. The pre-registered target, all-three within 5 points of human, is missed by 1.4 points. KlonSuphap-LM, the only public klon pad generator we know of, passes none of the 78 held-out topics when judged in separate pools together with the method's and the human stanzas (form 19.2%, cleanness 3.8%).

0.21

Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku

cs.CL cs.AI ·2026-09-14 ·Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese
published similarity 0.62
Abstract · 18 pages, 10 figures. Accepted as a full paper at the 2026 Joint 14th International Conference on Soft Computing and Intelligent Systems and…

This paper investigates the generation and human evaluation of Japanese haiku by contemporary Large Language Models (LLMs), focusing on authorship perception and aesthetic judgment within a constrained poetic form. Using a few-shot prompting strategy, Japanese haiku were generated across a heterogeneous set of large language models, including open- and closed-source systems, medium-scale and large-scale architectures, models with native or adapted Japanese support, and multilingual proprietary models. These AI-generated haiku were combined with human-written ones and presented in a questionnaire distributed to students at Japanese universities in Tokyo. The survey assessed whether respondents could distinguish between AI-generated and human-written haiku and which cues informed their judgments. Recognition accuracy varied across models. GPT-5, Gemini 2.5, and StableLM-7B performed at approximately chance level (approx 0.50), whereas LLM-JP, Gemma-2B, and LLaMA-2 showed moderate detectability (approx 0.59-0.67). However, recognition was strongly item-dependent. Ratings of fluency, coherence, poeticness, and related aesthetic dimensions predicted perceived humanness but not correct classification, indicating an attribution bias linked to aesthetic evaluation and revealing a dissociation between aesthetic evaluation and true authorship detection. The extended analysis additionally examines generation-constraint adherence, participant-level characteristics, and exploratory LLM-based evaluations of haiku authorship. Overall, the findings suggest that as LLMs improve, surface-level creative plausibility may reduce reliable human discrimination within constrained poetic settings.

0.35

HaikuS2S: A Cascaded System For Responding In Verse

cs.CL cs.AIcs.SD ·2026-09-20 ·Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe
published similarity 0.62
Abstract · Accepted to SLT 2026, Demo Track. 5 pages, 5 figures

Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior work on prosody transfer improves expressiveness, and fine-tuned poetry TTS (text-to-speech) systems capture verse intonation. However, these models do not model haiku's 5-7-5 syllable structure or line-ending pauses. We present a cascaded system, HaikuS2S, combining ASR (automatic speech recognition), LLM (large language model)-generated haiku, and TTS fine-tuning on both prose and custom haiku datasets. Our evaluation focuses on emotion similarity, speech quality, and prosody alignment. In our experiments, we see that our prosody and tonal alignment improve significantly with our fine-tuned systems, particularly the one trained on both general poetry and haiku. We also see that we maintain similar emotion similarity scores across all systems.

0.32

Poetry 'resurrection' from LLMs

cs.CL ·2026-11-12 ·Vincent Cellucci, Peter van der Putten
publishedsurveybenchmark similarity 0.61
Abstract · 8 pages, 8 figures, to be published in ISEA 2026 conference proceedings

In an era where linguistic creativity is challenged by algorithmic authorship, we pose a provocative question: can artificial intelligence (AI) write an infinite poem? We set up two experiments to investigate the quantitative and qualitative output produced by AI utilizing couplets that evidence a poetic relationship between end words, the only variation in a strict anaphoric, litany structure. Our results show that while AI can generate interesting pairings, it still falls short of human-level creativity and understanding of poetic relationships. We offer commentary and a brief survey of the field of computational poetry experiments, tracing back authorial intention and distinctions of 'natural' and 'artificial' poetry, as well as justifying why couplet form is suited to testing AI's poetic output. Experiment 1 relies on manual prompting and co-creation to extend the original poem, which is developed into an animated webpage installation and interactive artwork. Experiment 2 reliably achieves automated output reaching the 1000 benchmark via an alternative method of using an LLM in combination with semantic similarity, of which Method 2 demonstrates a higher capacity for semantic and quantitative poetic output. These experiments highlight the limitations of current LLMs' generation capacity at the word level but also suggest that poetry will also respond and revive by incorporating AI workflows.

0.23

Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis

cs.CL cs.AI ·2026-09-28 ·Haiyue Yuan, Jie Guo, Weidong Qiu, Zheng Huang, Ruizhe Li +1 more
benchmarkcode/data similarity 0.61
Abstract · Preprint

Automated detection of LLM-generated texts (LGTs) is critical, yet dedicated detectors often struggle to generalize across domains and models. While general-purpose LLMs offer flexible zero-shot authorship classification with explanatory rationale, their detection behavior, especially regarding self-detection versus cross-detection across model generations, remains poorly understood. We systematically evaluate 15 LLMs spanning three model generations as both generators and detectors. Using a benchmark of 1,000 human-written texts and 15,000 LGTs (1,000 per model), we collected over 233,000 binary classifications alongside natural-language explanations. Our results reveal that detection efficacy is primarily driven by detector capability rather than generator provenance, although outputs from newer generators remain notably harder to detect. Crucially, statistical comparisons show no systematic advantage or disadvantage for self-detection across models. Error analysis further exposes generational bias shifts: first-generation detectors under-detect LGTs (high false-negative rates), second-generation detectors over-flag human texts (high false-positive rates), and the latest models achieve balanced trade-offs. Finally, we highlight significant inconsistencies in how different LLMs apply textual cues to justify their decisions. Code: https://github.com/hyyuan/detect-llm-generated-texts.

0.28

Shaer: Controlled Arabic Poetry Generation with Meter Subform and Semantic Conditioning

cs.CL ·2026-10-07 ·Ahmad Abbas, Tamara Fakih, Nour Fakih, Ammar Mohanna
publishedcode/data similarity 0.61
Abstract · 22 pages, 7 figures. Code: https://github.com/AhmaddAbbass/Shaer ; models and datasets: https://huggingface.co/Shaer-AI

Classical Arabic poetry generation requires simultaneously satisfying semantic, linguistic, and fine-grained prosodic constraints. Existing systems typically control broad poetic attributes but do not jointly model semantic intent, meter subform, and poem length. We present Shaer, a controllable Classical Arabic poetry generation framework jointly conditioned on natural-language descriptions, meter subforms, and target hemistich counts. To support this task, we construct an enriched corpus of 116,032 classical Arabic poems derived from Ashaar, containing normalized meter-subform labels and automatically generated, validated semantic descriptions. We then adapt Yehia-7B using QLoRA-based supervised fine-tuning with a completion-only objective. Our evaluation combines automatic assessment of base-meter conformity, requested-subform adherence, and length control with three LLM judges, blinded human evaluation, and memorization analysis. Shaer achieves 95.17% base-meter accuracy, 91.75% poem-level meter-subform accuracy, and 83.40% exact count accuracy. Relative to its untuned foundation model, these results represent gains of 68.68, 57.77, and 38.93 percentage points, respectively; Shaer also attains the highest base-meter accuracy among all evaluated systems. Multi-LLM evaluation and a blinded human assessment of top-ranked outputs further indicate competitive semantic and literary quality. Finally, analysis of all 3,481 test generations finds no exact copies from the training corpus or paired source poems. Code, models, and datasets are publicly available.

0.40

AI Writers Have a Consistent Stylometric Footprint, but AI Editors Do Not

cs.CL ·2026-08-28 ·Zhengyang Shan, Yukyung Lee, Sophie Hao
top venue similarity 0.60
Abstract · EMNLP Main 2026

Text generated by large language models (LLMs) has been shown to be stylometrically distinct from human-written text (Andre et al., 2023; Shah et al., 2023; Opara, 2024; Soto et al., 2024; Li and Zhang, 2025; Selvioglu et al., 2025). But LLMs are increasingly used not only to generate text but also to edit human writing, and it is unclear whether the two leave the same trace. We show that AI generation leaves a consistent "stylometric footprint": a small subset of features, primarily entropy and lexical diversity, consistently separates AI-generated text from human writing across 8 LLMs and 5 domains, while the remaining features depend heavily on the domain and generator. AI editing, however, does not reproduce the same footprint. Relative to their human- written sources, AI-edited texts show only a small increase in lexical diversity and a decrease in entropy, rather than the joint increase that characterizes AI generation. Lexical density, which contributes little to generation, instead becomes the dominant editing-associated signal. Stylometric features therefore separate AI-edited text from AI-generated text but are substantially less effective at separating it from human-written text. Our results suggest that "AI text" is not a single phenomenon: generation and editing leave qualitatively different stylometric traces and should be studied separately.

0.39