CS Paper Radar

Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

Jiaqi Deng

cs.CL ·submitted 2026-09-23 15:36 UTC · v1

Abstract

Meaning identity (whether two sentences say the same thing after wording changes) is treated in retrieval and RAG as a geometric fact about independently encoded sentence vectors. We show that, for frozen off-the-shelf encoders and language models, it is not: identity is computed when both sentences share one forward pass, and is not a property of the embedding geometry those systems ship. On overlap-matched PAWS-X, purpose-built encoders (BGE, E5, GTE, MiniLM, E5-Mistral-7B) reach English confirm AUC only 0.55-0.65 (dense peak 0.70). Independently encoded last-token states of Llama 3, Mistral, and Qwen do no better; late fusion of the two vectors stays near chance. The same probe on a joint forward pass reaches 0.90-0.96 from 1.5B to 32B, collapses under partner shuffle, is mid-depth, saturates near 0.94 by 3B, and appears more weakly in GPT-2 XL (0.76). The gap holds beyond Llama-style models on other causal LMs, bidirectional encoders (DeBERTa, RoBERTa), and encoder-decoders (Flan-T5, T5, BART). Fixed or linear readers over frozen independent encodings never unlock identity; nonlinear pair readers recover part of it only on the full 49k-pair PAWS train split (0.68-0.87). Off-the-shelf rerankers split: BGE-reranker-large reaches 0.94, while MS-MARCO and Jina stay at 0.55-0.64. Independently trained families compute the same relation and a 1.5B joint reader can distill it from unlabelled teacher scores, while no linear function of the teachers own independent vectors can. Bi-encoders can be fine-tuned to fit PAWS (0.87-0.93), but transfer and STS-B suffer. Cosine compares wording neighbourhoods; identity is a cheap computed operator, not a property of either sentence vector.

Comment
12 pages, 2 figures. Code to be released

Closest papers

Nearest neighbours by meaning across the whole index.

Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

cs.CL ·2026-08-31 ·Donghoon Han, SungHyun Moon, Aidyn Zhakatayev, Junghun Cha, SeungJae Lee
top venue similarity 0.65
Abstract

Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

0.50

Do Multilingual Encoders Produce Language-Consistent Semantic IDs?

cs.IR cs.CL ·2026-10-01 ·Abhinav Bohra, Anuj Bohra
similarity 0.62
Abstract · 7 pages, 8 tables. Accepted as a short paper at WiNLP 2026, co-located with EMNLP 2026

Semantic IDs (SIDs) compress item embeddings into discrete code sequences used in generative retrieval. We ask whether a multilingual encoder is sufficient for different-language renderings of the same product to receive language-consistent SIDs. Using Amazon ESCI listings rendered in English, Spanish, and Japanese, we test whether translations remain close to their English source, whether residual quantization is unusually sensitive to translation-induced movement, and whether multilingual or language-balanced quantizer fitting improves SID agreement. Multilingual E5 places translations measurably apart: under an English-heavy fit, a Japanese translation preserves the first SID code of its English counterpart in only 7.7% of cases, compared with 89.0% for an English rewording. Distance-matched product-directed controls produce nearly the same full-SID mismatch as translation, providing no evidence that the quantizer selectively amplifies language directions. Balancing the fitting mixture makes codebook use more uniform but further reduces cross-lingual prefix agreement: Spanish first-code consistency falls from 28.3% to 6.6%, while an English-only fit preserves it for 67.6% of Spanish translations. These results show that multilingual exposure and balanced codebook use alone do not guarantee language-consistent SIDs.

0.34

UIBK-Embedding: Effective Text Retrieval by Unifying Dense and Late-Interaction Retrieval

cs.IR ·2026-10-19 ·Abdelrahman Abdallah, Mohammed Ali, Adam Jatowt
code/data similarity 0.61
Abstract

Dense bi-encoders and late-interaction retrievers are usually treated as alternatives, yet they fail on different queries: one vector captures what a text is about, token-level matching finds its exact entities and phrases. Using both today means two models, two indexes and a score-fusion step. We present UIBK-Embedding-v1, a 179M-parameter English encoder that produces both representations in one pass and scores them with one unmodified MaxSim. The dense embedding is written into the list of token vectors as three extra vectors, and a one-dimensional tag keeps the two kinds of vectors from matching each other, so that plain MaxSim equals the token-level MaxSim plus a weighted dense cosine. The model therefore runs in a standard late-interaction index without changes; we test it with PLAID. A shared 18-layer trunk with a separate 4-layer top per head, and a cross-encoder distilled into the fused score, turn the two heads into one retriever. On BEIR, UIBK-Embedding-v1 reaches 59.79 nDCG@10. This is 2.57 points above LateOn, the strongest model of its size, and 2.10 points above it when both models read documents of the same length; UIBK-Embedding-v1 is ahead on all 15 datasets. On a decontaminated version of BEIR it stays ahead on average (61.55 against 60.36) and on 8 of 12 datasets. Among the 156 models with complete BEIR results in the MTEB results repository, only models with at least 1.5B parameters score higher. Code, model weights, and a technical blog post are available at https://github.com/DataScienceUIBK/uibk-embedding , https://huggingface.co/DataScience-UIBK/uibk-embedding-v1 , and https://huggingface.co/blog/abdoelsayed/uibk-embedding-v1 .

0.33

Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders

cs.CL cs.LG ·2026-10-23 ·Mahdi Naser Moghadasi
code/data similarity 0.61
Abstract · 6 pages, 2 figures, 3 tables. Code: https://github.com/mahdinaser/sent2vec-sise

We report a systematic dissociation in modern text embedders: sentiment is almost perfectly decodable from their embeddings, yet almost entirely absent from their metric. Across eleven frozen encoders spanning 2021 to 2026, including 2026-generation models and the 4B-parameter LLM embedder Qwen3-Embedding-4B, a linear probe recovers sentence polarity at up to 91.3% accuracy while cosine similarity separates polarity by a margin of only 0.003-0.064. Qwen3-Embedding-4B, the strongest probe in the panel, reaches 91.3% probe accuracy but only 64.3 retrieval MAP. Because retrieval, clustering, deduplication, and retrieval-augmented generation operate on the metric rather than on a trained probe, these systems are effectively blind to information their encoder demonstrably contains. A task-conditioning control suggests this reflects the pretraining objective rather than a prompting artifact: conditioning Jina-v5 on its classification task rather than its retrieval task recovers only about half of the MAP gain that injection achieves on the same weights (63.2 to 73.8, versus 85.1), and leaves the alignment gap essentially unchanged. Sentiment-Injected Sentence Embeddings (SISE) train a small (0.13M-0.69M-parameter) supervised-contrastive head on a frozen encoder in seconds on a CPU, using the labelled training set. The head improves every encoder released since 2022 on full-data probe accuracy (up to +5.8 points), retrieval MAP (up to +26.0) and polarity margin (7-61x). This is a specialization, not a free gain: the compact 128-d head loses 38-41 Spearman points on STS-B, while a fused variant that concatenates it with the base embedding loses 2.4-12.0. With the head, a 0.6B encoder at 128 dimensions matches a 4B encoder given the same head (91.8 vs. 90.3 MAP). We release a CPU-reproducible pipeline and a reusable diagnostic, the Attribute Alignment Gap.

0.44

Exact semantic readout from compressed vector representations

cs.CL cs.LGcs.LO ·2026-09-16 ·Daniel Quigley
similarity 0.61
Abstract · 24 pages; 31 references; 13 figures; 2 tables

We characterize when compressed vector representations admit exact linear or affine readouts of a finite lexicon's truth conditions: one fixed map per predicate, sending each entity vector to the corresponding truth vector. A necessary and sufficient row-space condition determines existence; the augmented truth matrix has rank r, giving minimum dimension r in the linear case, and r-1 in the affine. Exact readouts return values in a shared truth basis on which Boolean connectives act unchanged; separability alone requires an intervening threshold. For binary relations, exact bilinear readout of identity or strict total order requires linearly independent entity vectors. Experiments with GloVe and word2vec distinguish exact affine recovery, linear separability, and held-out prediction: most predicates are strictly separable, but none admits an exact affine readout from the pretrained embeddings. Supervised transductive training attains exact affine recovery to numerical precision at every tested dimension meeting the bound. At the embeddings' original dimension, geometries constrained to exact linear recovery retain 98-99 percent of the pretrained variance on the feature norms, and 80-83 percent on the WordNet lexicon.

0.31

Representation Alignment from a Few Paired Data via Distilled Optimal Transport

cs.AI ·2026-10-08 ·Julien Kordes, Reuben Dorent
similarity 0.60
Abstract · Preprint

Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only $N \sim 100$--$500$ pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the $N$ paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.

0.38

BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring

cs.CV ·2026-09-20 ·Liuyang Song, Yi Zhang, Zhongyi Deng, Daqian Yang, Hongbo Zhang
position similarity 0.60
Abstract

Global vision--language similarities compress an image and a caption into one vector, preserving semantics but not which word corresponds to which region or how those regions are arranged; a model can recognize every word and object yet prefer a compositionally incorrect caption. We argue that a frozen encoder retains this association structure, so the problem is to read it rather than to rebuild it beside the pretrained similarity. We introduce BindCLIP, a pairwise scorer built on one latent object: a balanced token--patch--depth optimal-transport coupling that places both candidate captions and several visual depths in a single plan. Semantic, entity, order, and spatial evidence are read as energies of this state, and exchanging the candidates permutes the plan, making the score exactly antisymmetric. A geometric refinement inside the coupling contracts moves that the candidates and the visual depths do not support. No task label, parser, relation inventory, or detector is used. One checkpoint and one inference path improve the official What'sUp, ARO, and SugarCrepe benchmarks over frozen global CLIP, with the strongest transfer on the relation splits. Controls rule out patch access and caption-length shortcuts, and an inference-time lesion localizes spatial arrangement to the coupling.

0.20

StaRSE: Compact Russian Sentence Embeddings with a Sign-Coded Static Encoder

cs.CL cs.IR ·2026-11-10 ·Boris Malashenko, Ivan Jarsky, Egor Soldatov, Vladislav Luzhnov, Sviatoslav Stumpf +4 more
top venue similarity 0.60
Abstract · Accepted to ReNeuIR @ SIGIR 2026. 16 pages

Transformer sentence encoders provide strong retrieval quality, but their inference cost becomes a bottleneck when embeddings must be produced on CPUs or refreshed frequently. We introduce StaRSE, a compact Russian static sentence encoder that removes Transformer inference at deployment time: token directions are stored as sign bits, token norms are stored separately, and sentences are encoded by mean pooling over norm-scaled sign directions. StaRSE is initialized from ruBERT-base, fine-tuned on Russian adjacent-sentence and task-shaped pairs, and post-trained with a contrastive stage that directly optimizes the sign-projected token table. On RuMTEB, StaRSE reaches a task mean of $51.16$, $7.98$ points above the strongest included static baseline. The packaged model occupies $11.3$ MiB and has a mean CPU latency of $0.08$ ms per sentence, compared with an estimated $55.7$ MiB 16-bit parameter footprint and $1.43$ ms mean CPU latency for the closest-quality contextual encoder. Ablations examine adjacent-sentence supervision, token norms, and direct sign-projected post-training.

0.32