CS Paper Radar

Concept Representations in Vector Space: Toward an Interdisciplinary Study of Meaning Variation

Simon Josef Durstewitz

cs.CL cs.CY ·submitted 2026-10-18 11:12 UTC · v1
code/data

Abstract

An increasing number of disciplines recognize the explanatory potential of meaning-bearing mental structures through which individuals interpret the world and guide their behavior. Concept representations constitute a core class of these structures, emerging from individuals' experiences with concept referents and encoding and organizing their understanding of them. Recognizing a private level of meaning, where concepts are not objectively defined by rules in the world but constitute subjective structures in individuals' minds, raises a fundamental question: do different individuals actually mean the same thing when they invoke the same concept? As latent structures, concept representations are difficult to access and model empirically. The present study introduces a measurement framework that makes such representations empirically accessible through textual traces elicited under standardized conditions. Using transformer-based sentence embeddings, the framework maps standardized responses elicited for concepts into a high-dimensional shared vector space, opening concept representations to the systematic analysis of interindividual variation. It develops three analytical avenues: (1) comparing representational sharedness across concepts, (2) identifying variables that correlate with representational variation, and (3) examining how focal actors or subpopulations deviate from a reference population. Each avenue is illustrated through an empirical proof-of-concept drawing on a large-scale free-association dataset. Together, the demonstrations show how researchers from across disciplines can move from a research question and data source to analysis and interpretation. An accompanying GitHub repository provides a ready-to-use implementation of the framework for existing and custom datasets.

Comment
20 pages, 5 figures. Companion paper to arXiv:2610.15496. Code: https://github.com/simon-josef/conrep

Closest papers

Nearest neighbours by meaning across the whole index.

Concept Representations in Vector Space: Quantifying Sharedness and Idiosyncrasy of Meaning

cs.CL ·2026-10-11 ·Simon Josef Durstewitz, Denis Buehler
code/data similarity 0.81
Abstract · 22 pages, 6 figures. Preprint under review at Open Mind: Discoveries in Cognitive Science. Data, code and pre-registration: https://osf.io/d…

Concepts are the basic units through which we navigate the world and communicate meaning. But do you and I really mean the same thing when we think or speak about a concept? The present study develops a measurement framework for modeling and analyzing interindividual variation in concept representations through sentence embeddings, dense vector representations mapping entire utterances into a shared geometric space. The framework is developed in a theoretical argument and validated, demonstrated, and subjected to robustness checks in a pre-registered study. Short personal definitions, elicited from $N = 125$ participants, are encoded as sentence embeddings and submitted pairwise to a crossed random effects model yielding two concept-level quantities: sharedness ($S_b$), the expected dissimilarity between two randomly drawn individuals, and idiosyncrasy ($I_b$), the person-specific share of total dyadic variance. Two validity tests support the approach: the embedding space fully recovers concept and domain organization from participant definitions even with the target word excluded, and a parameter-free classifier achieves 0.630 accuracy in a within-participant extensional prediction task against a chance baseline of 0.500 and a consistency ceiling of 0.893. The framework finds theoretically interpretable variation in $S_b$ and establishes $I_b$ as a reliable source of dyadic variance across concepts. Results replicate across three encoders varying in architecture and capacity, providing a reproducible and scalable instrument for the empirical study of meaning variation.

0.37

From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness

cs.CL cs.AIcs.LG ·2026-09-19 ·Qianli Wang, Yilong Wang, Dennis Wei, Jingyi Sun, Simon Ostermann +3 more
similarity 0.63
Abstract · In submission

Chain-of-thought (CoT) can sound plausible yet be unfaithful to the model's underlying reasoning. Most prior work probes CoT faithfulness through input--output behavior or input attributions, leaving internal computation largely underexplored. We instead cast faithfulness as internal concept grounding: Does a large language model's (LLM) CoT reasoning engage the same internal concepts that support the LLM's direct prediction, and do the shared concepts causally drive its answer? Encoding a prediction pass and a CoT pass with a single shared sparse autoencoder (SAE), a reliable approximator of the latent concepts LLMs use, makes their internal concepts directly comparable. We introduce three correlational metrics of concept-level alignment and a causal metric, $Δp$, which ablates the shared concepts and measures the drop in answer probability. Across five LLMs and four datasets, concept alignment is generally high, as indicated by the correlational metrics; yet these only identify which concepts are shared, not how much they causally contribute. $Δp$ fills this gap: causal faithfulness varies substantially with model depth, peaking at mid-to-late layers rather than the final ones, and model scale reshapes the layer-wise profile. Moreover, causally important shared concepts are not always verbalized in the CoT. These dissociations suggest that faithfulness cannot be reliably assessed from surface-level or representational correspondence alone; assessing it requires causal tests of whether the internal concepts underlying a CoT actually drive the model's prediction.

0.44

Inductive Learning of Concept Embeddings from Cross-Linguistic Colexification Patterns with Textual Attributes

cs.CL ·2026-10-15 ·Arne Rubehn
similarity 0.62
Abstract

The advent of distributional semantics as a framework for computational semantics marked a paradigm shift in the nature of representations, by which meanings are represented as dense vectors commonly known as word embeddings. Computational approaches to historical or typological language comparison, on the other hand, still mostly rely on discrete representations, while the potential of embedding-based methods remains largely untapped in this context. Recent studies have proposed concept embeddings as a language-independent, onomasiological counterpart to word embeddings, based on networks of cross-linguistic colexification patterns. While the resulting concept embeddings reflect meaningful semantic properties and relations, they suffer from a lack of scalability, since they rely on transductive embedding approaches which only learn embeddings for nodes present in the training data. In this study, we present an inductive framework for learning concept embeddings, combining the structural properties of full and partial colexification networks with semantic information from short concept descriptions. This framework allows the embedding of unseen concepts based on a short definition. We show that the resulting concept embeddings correlate with various cross-linguistic measures of similarity or relatedness, while generalizing well to new concepts that were absent from the training data.

0.16

Meaning Contracts: Local Semantic Geometry in Large Language Models

cs.CL cs.AI ·2026-10-22 ·Andrés Daniel Godoy Ortiz
code/data similarity 0.62
Abstract · 24 pages, 8 figures. Code and data: https://github.com/adgodoyo/Meaning_Contracts

This paper studies the local geometry of semantic meaning in large language models. Current representation analyses ask whether meanings are close in embedding space, or whether linear probes separate semantic classes, but they do not characterize the within-concept local geometry: what is invariant across realizations of a single semantic factor, what varies, and what changes when meaning is refined. This paper characterizes that geometry through two controlled probes evaluated on four architecturally diverse decoder-only LLMs (Gemma 4, Llama 3.2, Qwen 2.5, Pythia). A cross-lingual probe uses language as a perturbation to separate exact meaning matches, same-factor realizations, nearby concepts, and out-of-domain contrasts. A refinement probe separates specificity-bearing semantic narrowing from high-probability attributive elaboration and adversarial controls; both probes are validated by blind human judgments. Four findings recur across all models: under mean pooling, once dominant variance directions are removed, sentence pairs order by semantic relatedness (same factor, then same domain, then different domain) without overlap over a band of layers; semantic factors organize around an invariant core surrounded by structured realization-specific variation; refinement produces directed displacement into tighter neighborhoods, including against controls built on the same base noun; and sentence-level and token-local readouts expose complementary geometry (volume contraction at the sentence level, sharper anisotropic endpoints at token-local readout). We interpret these results as evidence that LLM semantics are organized as local anisotropic neighborhoods in which meaning contracts under refinement.

0.41

Revisiting LLM Homogeneity Through a Multidimensional Lens

cs.CL cs.AI ·2026-11-01 ·Ziliang Qiu, Renfen Hu, Yi Ding, Ted Underwood
similarity 0.62
Abstract · 26 pages, 9 figures

Concerns about LLM homogeneity are typically quantified with embedding similarity, which reduces two texts to a single score. We revisit homogeneity with 25 interpretable metrics spanning five dimensions: semantic, lexical, syntactic, structural, and stylistic & presentational. A human study shows that perceived similarity depends on both content (what responses say) and form (how they say it), yet embedding similarity correlates less with human meaning judgments than sentence-level matching ($ρ=0.417$ vs. $0.606$) and not significantly with form judgments ($ρ=0.140$). Across 18 contemporary LLMs, models say similar things in different words: they share global semantics and broad linguistic profiles but remain distinct in lexical characteristics, structural patterns, and interactional conventions. Across releases from 2024 to 2026, models become more like themselves but not more like one another: among significantly changing metrics, intra-model distances decrease for 17 of 22, and inter-model distances increase for 14 of 21. Across two post-training pipelines, model-specific conventions emerge during SFT and are selectively reshaped by preference optimization. These results suggest that homogeneity is multidimensional and evolving, and that embedding similarity alone can overstate cross-model homogeneity while concealing diverging intra- and inter-model trends.

0.13

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis

cs.CL ·2026-10-10 ·Samhita Bollepally, Takashi Yamauchi, Manju Muralee, Motunrayo Adebimpe
similarity 0.61
Abstract · & pages, 6 figures, 2 tables

Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning. A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input. The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer. The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.

0.17

Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models

cs.AI ·2026-09-15 ·Keqing Zhang, Jingyu Chen, Yufan Liu, Yongqiang Zhu, Nai Ding +4 more
similarity 0.59
Abstract · Preprint. 9 authors

As Large Language Models (LLMs) increasingly handle complex subjective tasks, aligning their intentions and behaviors with human values has become a critical scientific challenge. However, current efforts are confounded by a striking behavioral paradox: they fluctuate unpredictably under minor wording changes ("swing"), yet stubbornly ignore explicit instructions to correct ingrained biases ("rigidity"). Resolving this duality is critical for reliable AI alignment. To systematically understand and safely steer these latent subjective preferences, our study is structured around three fundamental questions. First, do LLMs possess an intrinsic value system? By projecting responses from 106 LLMs (150,000 queries per model) and 95,000 human survey profiles into a shared sociological space, we empirically confirm that they do. However, they do not mirror human diversity, instead crystallizing into a highly concentrated, idealized value core. Second, how can these values be quantified? We propose the Prior-Environment-Cognition (PEC) framework. This model mathematically defines value expression as the joint outcome of inherent dispositions like parameter weights (Prior), external contexts such as user prompts (Environment), and internal reasoning processes like Chain-of-Thought (Cognition). Finally, how can LLMs' values be aligned toward a desired target? Using PEC diagnostics, we establish an adaptive "Alignment Prescription". Rather than blindly applying resource-intensive training, this method identifies the minimum effective intervention needed for each dimension, ranging from zero-cost prompts to targeted parameter updates. Extensive empirical validation confirms that our approach successfully verifies the presence of LLM values, accurately quantifies their shifts, and achieves more efficient and precise steering than conventional blind training, all without degrading general capabilities.

0.09

Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations

cs.LG cs.AI ·2026-09-29 ·Hanwei Zhang, Tianma Hu, Gaojie Jin, Xu Cheng, Ronghui Mu
top venueposition similarity 0.59
Abstract · accepted by NeurIPS 2026

Concept Bottleneck Models (CBMs) are designed to provide interpretable intermediate representations, yet how such bottlenecks affect robustness remains unclear, with existing studies reporting mixed and sometimes contradictory findings. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability robustness trade off that depends critically on the perturbation regime and task structure. Together, our results show that interpretability and robustness are distinct objectives: interpretable intermediate representations do not uniformly improve robustness, but instead redistribute sensitivity across perturbation spaces and model families.

0.47