CS Paper Radar

SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives

Tianyu Wang, Nianjun Zhou

cs.CL cs.AI ·submitted 2026-09-06 13:57 UTC · v1

Abstract

Assessing the literary quality of narratives requires evaluating interpretive dimensions (cultural representation, emotional depth, and philosophical engagement) that existing NLG metrics cannot measure. We introduce SAGE, a six-layer evaluation framework that separates rule-based assessment of observable textual properties from LLM-based evaluation of interpretive qualities drawn from cultural theory, affect theory, and existentialist philosophy. Each interpretive layer is assessed through multi-round iterative LLM evaluation with independent cross-validation, achieving measurement-grade reliability (98.8% convergence, >94% inter-rater agreement) stable across evaluator models. Validated on 600 evaluations across 100 short stories, our central finding is a systematic capability boundary: emotional-psychological representation approaches human levels, while cultural critique and philosophical depth exhibit approximately double the gap. LLM-generated narratives score below even commercial genre fiction on all three layers. We interpret this as a boundary between pattern-reproducible literary capacities learnable from training corpora and stance-requiring ones demanding cultural positioning and philosophical engagement that pattern matching alone cannot provide.

Comment
12 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Evaluating Whether LLMs Can Reliably Connect the DOTs?

cs.CL ·2026-09-29 ·Eftekhar Hossain, John Salvador, Santu Karmaker
benchmarkcode/data similarity 0.64
Abstract

Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct missing spans within a broader storyline, commonly referred to as text infilling, while preserving consistency with both the local context and the global storyline. Despite using text infilling as a pre-training objective in many Large Language Models (LLMs), their actual performance on real-world narrative infilling remains underexplored. In this paper, we address this gap by introducing a multi-domain benchmark of ~9.2K instances for narrative infilling, constructed by masking one to three sentences across four narrative types: encyclopedic text, commonsense stories, news articles, and visual narratives. Using this benchmark, we evaluate 20 instruction-tuned open-source LLMs ranging from 1.5B to 70B parameters across varying levels of instruction specificity and reasoning guidance. Outputs are assessed using standard automatic metrics and a qualitative framework covering five narrative dimensions. Results show that model scale does not reliably predict infilling quality: Gemma-2-2B achieves the highest qualitative score (4.02/5), outperforming models over ten times larger, including DeepSeek-Qwen-32B (3.77/5, 6.6%) and LLaMA-3.3-70B (3.71/5, 8.3%). We further find that explicit reasoning offers limited benefits as chain-of-thought reasoning yields only a marginal improvement (+0.6%). Additionally, short narratives and domain characteristics emerge as stronger predictors of task difficulty than infill position alone for narrative infilling in current LLMs.

0.27

The Limits of Automatic Evaluation of Creativity in Large Language Models

cs.CL cs.AIcs.CY ·2026-08-24 ·Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
similarity 0.64
Abstract

Large Language Models (LLMs) are increasingly capable of generating text that challenges human performance in domains requiring creativity, yet evaluating creativity in LLM-generated content remains a significant challenge. Here, we investigate whether current automatic evaluation methods can reliably capture human judgments of creativity. We collect human evaluations of human- and AI-generated short stories from the WritingPrompts dataset across 11 dimensions of creativity, and compare these judgments with automated objective metrics and LLM-as-a-Judge evaluations. Our experiments reveal substantial misalignment between automatic evaluations and human assessments. In particular, LLM-based judges exhibit a systematic preference for AI-generated stories, consistently favoring their stylistic characteristics over the unpredictability and other qualities of human-authored texts. Furthermore, correlation analyses show that widely used automatic metrics exhibit near-zero alignment with human judgments across both human- and AI-generated stories, suggesting that they fail to capture important dimensions of creativity. These findings highlight fundamental limitations in current approaches to the automatic evaluation of creative text and underscore the difficulty of reducing the multidimensional and subjective nature of creativity to computational metrics.

0.37

Literary Axioms: A Conceptual Framework for Literary Creation, Interpretation, and Evaluation

cs.CY ·2026-09-22 ·Qiang Liu, Chunyi Zhao
similarity 0.64
Abstract

How can the conceptual commitments of a literary work connect its creation, interpretation, and evaluation? We propose Literary Axioms, a conceptual framework in which a work develops a configuration of claims about experience and commitments to literary form. A literary axiom is a provisional organizing premise whose scope and textual realization can be examined. The framework develops the proposal by Qiang Liu that writers select and combine such premises, realize them through characters, events, language, and form, and make them available for reconstruction by readers. We distinguish an open axiom space, a selected configuration, its relational organization, and situated reader reconstructions. This yields an account of coherence as organized compatibility or conflict, originality as a change in claims, relations, or realization, and interpretive richness as a plurality of textually grounded reconstructions. Five dimensions connect this account to literary evaluation: coherence, contextual reach, distinctiveness, realization fidelity, and interpretive richness. We specify boundary cases and derive five empirical predictions with observations that would count against them. A concrete implementation contains 1455 Chinese language records, 1464 mappings to 149 works, and 472 typed relationships. A descriptive audit establishes the structural integrity of the artifact and identifies gaps in reuse, context coding, and provenance. The paper contributes a theory based framework, an inspectable implementation, and a research program. Its predictions await reader and creative writing studies.

0.51

CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text

cs.CL ·2026-08-31 ·Mohammad Reza Modarres, Armin Tourajmehr, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar
top venue similarity 0.63
Abstract · Accepted to Findings of EMNLP 2026

Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered. We examine how reliably LLMs evaluate short literary text in Persian, a low-resource language, across multiple evaluation strategies and prompt formulations. We find that LLM-human agreement varies substantially across dimensions: alignment is stronger for structured TTCT-derived properties such as Originality, Fluency, and Elaboration, but considerably weaker for more subjective dimensions, particularly Emotion and Attractiveness. Judgments are also sensitive to prompt formulation, while few-shot prompting, ensembling, and multi-agent debate provide no consistent improvement. Motivated by this dimension-dependent behavior, we investigate whether structured creativity dimensions can instead be approximated using simple, interpretable proxy metrics. We introduce CLIN, which evaluates three TTCT-derived dimensions separately using topic-aware novelty for Originality, contextual lexical clustering for Fluency, and lexical diversity for Elaboration. These proxies achieve human alignment comparable to or better than the strongest zero-shot LLM judge in our setting while requiring substantially lower evaluation cost.

0.39

Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?

cs.CL cs.AI ·2026-09-23 ·AbdulRahman A. Morsy, Aya Zirikly
similarity 0.62
Abstract · 14 pages

Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored. Prior work has focused largely on modern language varieties and poetry, while classical prose traditions such as maqama remain largely unstudied. The maqama is a classical literary genre characterized by rhymed prose (saj), dense rhetorical ornamentation, and episodic narrative structure, making it a challenging testbed for evaluating whether LLMs can move beyond surface fluency toward deeper literary competence. In this paper, we present the first controlled evaluation study of maqama generation with LLMs, comparing five models under zero-shot, few-shot, and rule-based prompting, and evaluating outputs through both human annotation and an LLM-as-a-judge framework across dimensions such as rhetorical richness, saj density, structural coherence, and stylistic authenticity. Our results show that prompting strategy plays a strong role in stylistic quality: few-shot prompting most consistently improves saj density, while its effects on rhetoric and coherence vary by model, with the strongest models (GPT-4o and GPT-5.4-mini) benefiting most from rule-based prompting on these dimensions, though zero-shot prompting yields the highest aggregate scores across all five models. We further observe systematic differences between models in stylistic alignment with Arabic maqama conventions, and corroborate our findings with a second independent LLM judge, paired statistical significance testing, and non-LLM proxy measures of saj.

0.18

From annotation to reasoning: Culture in language models

cs.CL ·2026-09-25 ·Daniel Hershcovich, Alexander Conroy, Jens Bjerring-Hansen
benchmarkposition similarity 0.61
Abstract · 8 pages, 1 table; perspective paper

How should we evaluate language models when more than one interpretation can be right? Cultural benchmarks often test factual knowledge, agreement with survey responses, or recognition of a predefined meaning. These tasks leave open whether a model can explain how a cultural reference works in a particular text, support a reading with evidence, or revise it after criticism. This is a question of interpretive depth, complementary to the breadth of cultural coverage. We argue that literary interpretation offers a useful setting for studying these capabilities. We focus on cultural referencing and reuse: how texts invoke, repeat, and transform earlier expressions across historical and linguistic contexts. Our central claim is that literary scholars can disagree about an interpretation while recognizing the quality of its support. We propose linking evidence-centered benchmarks, evaluation that preserves scholarly disagreement, and model-development experiments on literary data, contextual resources, and scholarly feedback. Danish literature provides a concrete starting point, with implications for other languages and domains. The aim is to develop alternative evaluation strategies that go beyond conventional benchmark metrics and guide model development toward cultural robustness in AI systems.

0.24

Scaling Creative Writing Beyond Story-Centric Data with Attribute-Guided Genre Expansion

cs.CL ·2026-08-14 ·Hwan Chang, Yongil Kim, Heuiyeen Yeen, Yireun Kim, Jinsik Lee +1 more
top venue similarity 0.61
Abstract · CIKM 2026

High-quality creative writing data for large language models (LLMs) remains dominated by story-centric data, limiting models' ability to follow the structural and functional conventions of diverse creative formats. We propose an attribute-guided genre expansion framework for scaling creative writing data beyond story generation. By separating thematic breadth from genre-form control, our framework leverages human-authored story prompts as diverse creative seeds, while utilizing manually curated genre attributes to enforce distinct structural, stylistic, and formatting conventions. We combine these to prompt strong LLMs for genre-faithful query-response pairs, which are then quality-filtered. Applying this framework, we construct the Multi-Genre Collection, a 50K-example corpus spanning 13 creative genres, including story, rap, lyrics, scripts, game design, character design, and other creative formats. Experiments across out-of-distribution writing benchmarks and held-out genre diagnostics demonstrate that models fine-tuned on our data consistently surpass not only base models and writing-specialized baselines, but also models trained on existing writing corpora. Genre-count ablations further indicate that controlled genre expansion, rather than story-centric scaling alone, is a key driver of robust creative writing capability.

0.29

Incipit: Axiom-Grounded Scaffolding for Human-AI Literary Creation

cs.CY ·2026-09-25 ·Qiang Liu, Chunyi Zhao
similarity 0.61
Abstract

Large language models can produce fluent prose from short prompts, but direct prompt-to-text interaction gives writers limited access to the assumptions that shape a long narrative. We present Incipit, an implemented research prototype that introduces an explicit planning layer between a writer's intent and generated prose. This layer is grounded in literary axioms, defined as curated and reusable propositions about human experience and narrative craft. The prototype connects a knowledge base of 1455 axioms and 472 typed relationships with a five-round direction dialogue, a retrieval-and-selection pipeline, and a three-level blueprint covering creative premises, story beats and character arcs, and chapter outlines. Writers can inspect and edit these structures before using them as context for scene generation. Additional modules support real-event abstraction and five-dimensional diagnostic feedback. We describe the system design rationale, data flow, implementation boundaries, and a worked design example. As no controlled user study or independently rated output study has yet been completed, we do not claim that the system improves literary quality. Instead, we outline a future preregistered evaluation designed to distinguish the contribution of axiom grounding from that of hierarchical planning. The paper contributes a concrete architecture for using literary knowledge as an inspectable coordination object in human-AI writing.

0.21