CS Paper Radar

Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment

Yasuaki Sumita, Tomoharu Iwata, Toshiyuki Tanaka

cs.CL cs.LG ·submitted 2026-10-16 06:24 UTC · v1

Abstract

Alignment is known to change the output characteristics of large language models (LLMs). Base models tend to generate more diverse outputs, whereas aligned models tend to produce higher-quality outputs. The appropriate balance between the two depends on the task. Existing methods either require costly fine-tuning or, like temperature sampling, mainly improve lexical diversity rather than semantic diversity. We introduce a depth-aware FrankenMerging approach for controlling the diversity--quality trade-off between an aligned model and its base model without additional training. This approach combines the lower layers of an aligned model with the higher layers of its base model based on the hypothesis that lexical diversity is more closely associated with lower layers and semantic diversity with higher layers. Experiments showed that varying the number of layers taken from each model adjusts both diversity and quality. In most cases, the merged models exhibited higher semantic diversity than the aligned model, and at some split positions, they showed higher lexical diversity than both original models. They also demonstrated a better diversity--quality trade-off than temperature sampling in most settings. A supporting ablation, in which we reversed the merging order, yielded results largely consistent with the hypothesis.

Comment
13 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?

cs.CL ·2026-10-12 ·Xaver Raphael Davey, Shubhra Mishra, Anton Adelöw, Siyao Liu
benchmark similarity 0.76
Abstract

Text generated by Large Language Models (LLMs) often remains limited in diversity compared to that produced by humans. Recent advances demonstrate the presence of inter- and intra-model collapse in open-ended generation of English text, raising concerns about the long-term homogenization of LLM, and more importantly, human thought. However, our understanding of whether this homogenization extends to languages beyond English is limited. In this work, we investigate the role of language on output diversity. Specifically, we aim to understand whether multilingual LLM-generated text is also homogeneous, and whether this homogeneity matches what we observe in humans. Across 23 open-weight models and 13 languages, we find that the artificial hivemind effect is pervasive but uneven in its extent. The intra-model effect holds in three different embedding spaces and under a lexical metric that uses no embedding model, so it is not an artifact of how diversity is measured. The size of the homogenization gap is associated with a model's competence in a language, measured on a held-out translation benchmark, robustly across embedding spaces. Thus, we demonstrate a performance-diversity tradeoff in multilingual LLMs.

0.26

Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models

cs.LG cs.MA ·2026-09-03 ·Ross Tieman, Evan Markou
benchmarkposition similarity 0.69
Abstract · 31 pages, 13 figures

Diversity is a widely observed factor in the resilient function of collective systems, yet the type of diversity that matters depends on the properties and failure modes of the system. This distinction is important for systems composed of multiple language models. Different models may be treated as independent components even when their behaviour and failures remain strongly correlated. Assessments of language-model populations using semantic similarity demonstrate limited semantic diversity, but this captures only differences in the meaning of observed outputs. We argue that a more fundamental notion of model diversity is generative-process diversity, the differences between processes capable of generating the observed outputs. Drawing from Algorithmic Information Theory, we use Normalised Compression Distance between raw model outputs, residualised against a permutation control, as a measure of inferred generative-process diversity. Across 38 language models, this measure identifies population structure missed by semantic similarity and predicts cross-task variation in chance-corrected correlated failure among model pairs across ten disjoint benchmark families, beyond semantic similarity and model-pair capability. The cross-benchmark partial rank association is $-0.216$ with a 95% interval of $[-0.309,-0.122]$, and the estimate is negative on all ten benchmarks. These results indicate that increased generative-process diversity is associated with reduced correlated failure in model pairs that is not attributable to semantic similarity or capability. Inferred generative-process diversity offers a novel and practical approach for investigating diversity of multi-model systems in safety-relevant contexts.

0.24

Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs

cs.SI cs.CL ·2026-09-01 ·Jingshen Zhang, Shaoyang Xu, Wenxuan Zhang
top venuehot direction similarity 0.68
Abstract · Accepted at EMNLP 2026 (Findings)

Cultural fine-tuning has become the de facto paradigm for building culture-aware large language models (LLMs), yet existing optimization exclusively for alignment scores provides an incomplete portrait of cultural fidelity by systematically obscuring inherent cultural diversity. This unidimensional evaluation lens prompts a fundamental question: do models genuinely perceive distinct cultural nuances, or do they merely memorize dominant cultural values? To address this, we propose a synergistic evaluation framework that jointly formalizes cultural alignment and diversity. Through extensive benchmarking of six mainstream LLMs on the World Values Survey, this framework uncovers a systematic and critical trade-off: the pursuit of cultural alignment consistently incurs an acute expense of diversity, leading to severe "cultural flattening." Investigating this behavioral shift, we demonstrate that these superficial alignment gains stem from models artificially anchoring to dominant majorities, converging onto a monolithic response pattern that wipes out the heterogeneous distributions inherent to human groups. Crucially, our mechanistic analysis suggests that this diversity collapse is not merely a behavioral anomaly but more likely a structural consequence of the low-rank bias inherent in neural network optimization. Therefore, our findings expose the limitations of current post-training paradigms and call for a shift toward alignment objectives that preserve cross-cultural pluralism.

0.41

PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation

cs.LG ·2026-10-23 ·Qixin Xu, Zihan Zhu, Xiaojiang Liu, Bohan Lyu
top venuebenchmark similarity 0.68
Abstract

Large language models (LLMs) exhibit mode collapse, where their output distribution concentrates on familiar semantic modes. When diversity disappears, repeated sampling becomes semantic repetition rather than discovery, making rare but stronger solutions unlikely to surface and leaving self-evolution and continual learning difficult to sustain. Most remedies intervene only at inference time, shaping sam- ples without making diversity a model capability. We propose PRISM, a training-time framework that internalizes diversity within the model. PRISM combines (i) submodular curation that pairs each prompt with diverse-yet-correct multi-teacher solutions, and (ii) stochastic embedding modulation, which turns special-token perturbations into implicit signals for sampling distinct outputs. A theoretical analysis establishes the stability of this mechanism. On WritingPrompts and BookMIA, PRISM moves the quality-diversity frontier outward, achieving the best generation quality in all model-benchmark settings while maintaining the strongest or near-strongest semantic diversity across two open-weight model families. It also improves sampling efficiency on math and code benchmarks. Beyond inference, PRISM yields a stronger foundation for reinforcement learning, sustaining diverse rollouts and mitigating the diversity collapse commonly observed during RL training.

0.57

Five Architectures, One Shared Space Cross-Model Representational Alignment Without Weight Merging

cs.CL ·2026-10-24 ·Oladimeji Anthonio, Dimeji Olawuyi, Oloruntoba Ajayi, Temilola Aderemi
similarity 0.66
Abstract

Combining language models is usually done by averaging weights, which works only when the models share a common starting point. We ask whether five frozen, independently trained models with different internal designs (four transformers from four organizations and one state-space model) can be related through their activations alone, with no weight merging. Using permutation tests, an independent replication probe set, depth sweeps, pooling ablations and confound checks, we find statistically robust correspondence between how the five models organize the same text. It is strongest at each model's final layer and weaker for the state-space model, whose weakest pairing is still 35-43 times chance. We use this to build a shared space with Generalized Procrustes Analysis, in which no model is a fixed reference. At the final layer, mean held-out retrieval accuracy over the twenty ordered model pairs is 0.91 to 0.93 on two probe sets (chance 0.03), beating pairwise linear maps (0.82 to 0.85), but the space is worse than those maps in the middle layers. It transfers to ordinary English of unfamiliar grammatical form and fails on mathematical and logical notation. We then test whether the space supports a router that picks among the five models on gradeable tasks. Across four task types we find no case where routing beats always using the strongest model, and we report this as a negative result. A correctness signal that looked strong turned out to be category membership in disguise. All experiments ran on one laptop without a GPU.

0.16

Do Large Language Models Capture the Diversity in their Training Data?

cs.CL cs.AIcs.LG ·2026-09-02 ·Youqi Wu, Farzan Farnia
theory similarity 0.66
Abstract

Large language models are trained to model conditional distributions over text, yet it remains inadequately understood whether they capture the full diversity of plausible outputs present in their training data. We study this question through an information-theoretic lens by comparing the conditional entropy of model-generated outputs with that of the corresponding training data. Given paired input-output samples, we use conditional entropy and its matrix-based analogue based on von Neumann entropy to measure output variability beyond what is explained by the conditioning input, without requiring multiple reference outputs for the same prompt. Across LLM families with publicly available training data, including OLMo, Pythia, and GPT-Neo, we consistently find that model-generated outputs exhibit lower conditional entropy than their training data, across different model scales, sequence lengths, and decoding strategies. We observe a similar conditional diversity gap beyond language modeling, including class-conditioned ImageNet generators and text-conditioned models trained on MS-COCO. To address this gap, we propose a post-hoc correction mechanism that generates multiple outputs for each input and reweights them through a matrix-entropy projection, increasing conditional diversity while remaining close to the original model distribution. We prove the concavity of the matrix-based conditional entropy functional, which makes the resulting entropy-constrained projection a convex optimization problem, and develop a scalable mirror-descent algorithm for its implementation. Our results reveal a systematic conditional diversity gap between modern generative models and their training data, and provide an information-theoretic framework for measuring and mitigating this gap.

0.27

Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

cs.AI ·2026-09-15 ·Kirill Skobelev, Eric Fithian, X. Y. Han
similarity 0.66
Abstract

Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse, or its opposite, occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently from the same fixed prompt coincide (collide), or their expected similarity under a kernel. We derive a bias-variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback-Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: small transformers on synthetic languages, four LLMs fine-tuned on human surveys, and these LLMs fine-tuned on CodeNet, a dataset of human code solutions. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution.

0.38

Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models

cs.CL ·2026-08-11 ·Alexandrine Fortier, Hazel Chen, Peter West
top venueposition similarity 0.65
Abstract

The lack of diversity in LM content is widely attributed to the alignment process, but how and where exactly in the pipeline this collapse begins is unknown. We argue that output homogeneity is likely learned during the pretraining phase, and only revealed or magnified during the alignment process. Specifically, we find that semantic convergence is observed from the first alignment stage--the instruction-tuning phase (SFT)--suggesting that homogeneity might already exist in the pre-alignment model. To investigate this, we conduct controlled SFT experiments examining how training data influences output convergence on specific input/output pairs. We find that convergence can be revealed and amplified, but not introduced by the SFT data, supporting its role as a catalyst rather than a cause. To further test whether homogeneity originates before alignment, we measure convergence in base models. We find that instruct-like collapse can be induced through prompting alone, even without alignment. Taken together, our results suggest that semantic convergence may arise naturally from the objectives underlying LM training, making it difficult to mitigate through post-alignment interventions alone.

0.70