CS Paper Radar

Not All Confusion Is Equal: A Source-Aware Uncertainty Diagnosis for Fine-Grained Aircraft Detection

Hai Huang, Helmut Mayer

cs.CV cs.LG ·submitted 2026-09-24 15:17 UTC · v1
positionunusual

Abstract

Fine-grained object detectors are commonly evaluated with confusion matrices, which show where the model is confused but not why, nor whether the confusion can be reduced. We argue that confusion can be attributed to distinct, separable sources, each quantitatively measurable, turning a passive measurement into actionable guidance. We present $A^2E^2$, a diagnostic tool that decomposes the sources of confusion along two axes, $\{$aleatoric, epistemic$\} \times \{$within-class, between-class$\}$, giving a $2\times2$ taxonomy that enumerates the source types. Each quadrant is measured by its own quantity, computed in one of three places (input geometry, output-space disagreement, and the bias-parameter posterior), so the two epistemic sources are separated by construction rather than by an empirical correlation. On fine-grained aircraft detection, the four quadrants become four named sources with their own remedy verdict: affinity (geometric similarity, irreducible from size alone), heterogeneity (geometrically heterogeneous sub-variants, pointing to re-labeling rather than more data), contested (an insufficiently trained but learnable boundary, improvable), and collapsed (a class starved of data, reducible). After attributing the confusion to a specific reducible source, we apply a targeted intervention and verify experimentally that it reduces the diagnosed source specifically while leaving the irreducible sources unchanged. $A^2E^2$ thus turns confusion measurement into a concrete, validatable and actionable "diagnosis" in which the same off-diagonal mass can carry opposite causes and opposite remedies. We also state this framework's limits, including which sources are only partially identifiable on this specific dataset and why.

Comment
23 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Where Is Uncertainty Observed? A Mechanistic Understanding of Uncertainty Quantification in LLMs

cs.CL cs.AIcs.LG ·2026-11-11 ·Xiaoyu Liang, Virginia Ceccatelli, Aliai Eusebi, Tobias Braun, Nils Loose +2 more
survey similarity 0.62
Abstract

Knowing when a large language model's (LLM's) response can be trusted is critical for its safe use. Uncertainty quantification (UQ) can help LLMs recognize if their responses may be unreliable. Existing taxonomies organize UQ methods by uncertainty source or estimation mechanism. They leave unresolved the question of where uncertainty is observed in the model's computation. The answer matters because the observation point determines whether an estimator has access to enough information to recover a given target, and how well it can recover it. Drawing on mechanistic interpretability, we define four mechanistic observation lenses: the structural lens observes component updates, the representational lens observes hidden states, the distributional lens observes token probabilities, and the semantical lens observes distributions over meanings. We relate these lenses through the residual stream and order them by the information they carry, which gives a principle for selecting UQ methods for different targets. We survey existing methods through the four lenses. Furthermore, we distinguish evaluation targets, criteria, and metrics to clarify different UQ evaluation protocols. Finally, we identify open problems and research directions to connect uncertainty estimation to the internal computation and downstream use of LLMs.

0.19

NeSySpec: Neuro-Symbolic Technical Intelligence for Auditing Engineering Specifications with an Application to Aircraft

cs.AI cs.CVcs.SC ·2026-10-26 ·Pier Mattia Basciano, Gianvito Antenori, Cristiano Andolfi, Giovanni Farina, Salvatore Monteleone
hot direction similarity 0.57
Abstract

Engineers who audit a declared specification of an engineered system need to know which physical constraints it respects, which it violates, which field was altered and how to correct it. NeSySpec is an integrated and interpretable neuro-symbolic audit framework, built on Logic Tensor Networks, that answers these questions in one pipeline. Its axioms encode the physical relations of the domain as differentiable graded predicates whose tolerances are learned from a fleet of real records; from the same degrees, it diagnoses the violated constraint, identifies the altered field by abduction, proposes a minimum-change correction and ties image evidence to the parts it describes, abstaining exactly where evidence is missing, and returns all of it in one verification report. We validate NeSySpec on 107 aircraft types, with 16 test aircraft of 9 families held out of training, on specification records paired with ground photographs and satellite imagery and on synthetic errors: generated corruptions, edits labeled from the edit alone and deliberate falsifications. NeSySpec combines all the assessed audit outputs in one pipeline, which no learned, rule-based, solver-based, neuro-symbolic or language-model comparator does; on the test aircraft none is ahead of it in the macro-F1 of its constraint verdicts, in naming the violated constraint and the altered field or in the repair error, and it is competitive in record and image-pair detection. Its diagnosis names the violated constraint in 94.2% of corrupted test records, against 79.5% for classical Fellegi-Holt diagnosis, and on edits labeled from the edit alone, without retraining, abduction finds the altered field in 59.3% of records, against 30.6% for minimum-cardinality localization with random tie- breaking. Only the knowledge base, the visual predicates, the corruption generator and the data are specific to the domain.

0.33

From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs

cs.AI ·2026-09-03 ·Omer Nahum, Niv Nayman, Jonathan Fhima, Alon Zolfi, Jeremy Levy +2 more
top venueposition similarity 0.56
Abstract

A key challenge in reliable LLM deployment is recognizing when uncertainty reflects irreducible variability in the task rather than limitations in the model's knowledge. In language tasks, a central source of such aleatoric uncertainty is input ambiguity or underspecification, where multiple interpretations remain plausible. Existing decomposition methods estimate aleatoric uncertainty by generating multiple clarifications of the input, querying the model for an answer under each clarification, and comparing the resulting answers. We argue that answers are not necessary for identifying ambiguity: they are often redundant, add avoidable cost, and can mislead through epistemic leakage. We support this claim theoretically, and propose a clarification-only approach that estimates this ambiguity-induced component directly from the space of plausible interpretations, without answers to the clarified inputs. Using ambiguity detection as an operational evaluation across three benchmarks, this direct approach improves AUROC (63.34 vs. 60.85), reduces computational cost by 4-26x in output tokens and 2.2-3.5x in API calls, and yields estimates with substantially lower correlation with epistemic uncertainty. Overall, our results suggest that ambiguity-induced aleatoric uncertainty is better estimated from the interpretation space than from the response space.

0.36

Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks

cs.AI ·2026-09-28 ·Manya Singh, Arjun Pakrashi
similarity 0.56
Abstract

Human-model alignment is critical for trustworthy AI-assisted decision-making systems. Yet, most work evaluates model predictions against single ground-truth labels, overlooking that humans themselves often disagree on labels, a signal of genuine ambiguity. We investigate whether models struggle on the same instances that humans find difficult. We measure this on two vision datasets (FER+ and CIFAR-10H) where multiple human annotations per image capture human disagreement patterns. We evaluate eight pretrained models across three architectures (ResNet, EfficientNet, MobileNetV3) in two parts: first, whether model uncertainty (softmax confidence, entropy) correlates with human disagreement, and second, whether predictive multiplicity measures (inter-model disagreement, Jensen-Shannon divergence) do. We find that it does not: alignment is weak in both dimensions. At the discrete label level, 50.4% of CIFAR-10H images and 33.5% of FER+ images receive multiple valid classifications from humans, while the models converge on only one. These instances represent a critical failure case where humans perceive ambiguity and would request expert review, yet models decide confidently. At the continuous score level, single-model uncertainty correlates weakly with human disagreement ($ρ= 0.24--0.55$), and predictive multiplicity provides only modest improvement. Widely-used uncertainty quantification methods do not reliably identify instances humans find ambiguous. Model uncertainty should not be treated as a trustworthy signal by default for decision-making in high-stakes scenarios.

0.17

When Unpaired Sets Support Shared-Corruption Calibration: Moment Geometry and Two-Sample Precision

cs.AI stat.MEmath.ST ·2026-08-11 ·Shuheng Cao, Zhenhao Zhang, Ruiqi Chen, Renjie Cao, Siyu Zhang +3 more
similarity 0.55
Abstract

Collections of diverse observations often share one acquisition, processing, geometric, or channel corruption, while only an unpaired clean reference set is available. For a prescribed low-dimensional correction shared across observations, the observed and clean reference sets support inference only through the response of fixed moments. We formulate this problem as two-sample moment calibration and report a rank-aware information state combining local rank, scaled moment sensitivity, source-separated covariance, and a moment compatibility residual. Full rank gives local moment identifiability, whereas kernel directions remain unresolved to first order. A unified linearization separates observed-set and reference-set uncertainty. Under covariance weighting, the weakest scaled singular value determines worst-direction asymptotic amplification. For an orientation-preserving planar-similarity correction shared across observations, ensemble centroids and a nonzero third-order complex moment yield closed-form global population identification of translation, rotation, and isotropic scale under matched-population and no-clipping assumptions. Controlled validation tests the predicted $N^{-1}$ and $σ_{\min}^{-2}$ laws, Gaussian efficiency, and interval coverage. Bounded applications report color corrected-output quality, channel magnitude-response calibration, and a separate paired geometric de-beautification result. The framework therefore reports missing or weak information instead of treating every fitted correction as identified.

0.04

H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification

cs.LG cs.CV ·2026-08-18 ·Yuanye Liu, Xiahai Zhuang
similarity 0.55
Abstract

Fine-grained recognition often involves hierarchical label spaces, where a model may be confident about a coarse semantic concept while remaining uncertain among its descendant classes. Such structured ambiguity requires uncertainty representations that capture both fine-grained classes and intermediate concepts. However, existing tools each capture only half of it: flat evidential classifiers quantify total ignorance with a single vacuity on the leaf frame, and hierarchical classifiers propagate point probabilities with no notion of evidence. Hyper-opinions would unify the two, but their general form is exponential in the label count, and existing hyper-evidential networks either require composite labels to be supplied in the training data or read them off an unstructured weight pattern, with no principled notion of which composites deserve mass. We observe that the taxonomy itself is the missing hyperdomain. Its subtrees and leaf singletons form a linear-size focal family, and one local Dirichlet opinion per branching node induces every composite mass in closed form. The resulting model, H$^2$EDL, can be interpreted in two complementary ways using the same set of parameters. From a prediction perspective, it functions as a hierarchical classifier that preserves consistency across different levels of the label tree. From a probabilistic perspective, it defines a valid tree-structured hyper-opinion, where the mass assigned to each node represents the belief that reaches that node but does not provide sufficient confidence to further specialize into its descendants. On FGVC-Aircraft and DERM12345, H$^2$EDL reduces calibration error by approximately half compared with cross-entropy baselines, with the improvement becoming more pronounced at deeper hierarchy levels and under larger training budgets.

0.25

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

cs.CL cs.LG ·2026-08-12 ·Florian Braun
benchmarkcode/data similarity 0.54
Abstract · 23 pages, 11 figures, 8 tables. v2: measures the placebo baseline's own sampling variance, finds it exceeds the permutation null's in every …

Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at release. A recent alternative reads it off a linear probe on internal activations. We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against. The protocol reports a zero-sum contrast on the depth profile of probe accuracy, recentred on a level-matched placebo baseline, tested against a label-permutation null, with the reference set twice the size of the suspect set. Each choice replaces a simpler alternative we rejected on measurement. Reporting the level of excess separability rather than its shape makes the false positive rate track the size of the analyst's own control set, 0.03 to 0.99 under a true null. Contrasting against a flat depth profile rejects a true null 0.72 of the time when surface decodability rises with depth, and loses all power when it falls. On real transformers the protocol fails a test the simulations did not pose. The recentring subtracts an estimate, and the permutation null holds it fixed. Re-estimated across split seeds on four audits of contaminated checkpoints, its standard deviation is 1.30 to 1.56 times the null's own in every arm: what is subtracted to remove a bias is more variable than what it corrects. The one nominally significant result, p = 0.0075, becomes 0.0745 once that variance is propagated, and no verdict is issued. The simulations missed this because their surface key is the covariate driving item variation; on real text it is a proxy, and degrading key quality in simulation reproduces it. We add a companion measurement and a widened null. No arm shows contamination.

0.60

An Investigation of Model Coherence: Narrow Finetunes Contradict Themselves Under Resampling

cs.AI ·2026-10-08 ·Robert Graham, Yariv Barsheshat, Phil Blandfort, Sabri Alouache
code/data similarity 0.54
Abstract · 18 pages, 5 figures, 4 tables. Code and dataset: https://github.com/themachinefan/twominds

A large body of research measures model coherence based on output variance without adequately considering competing causes. We identify two such causes, ambiguity and indifference, and we introduce a set of 175 questions where contradicting answers cannot easily be explained by either. We then measure incoherence in terms of contradictions when resampling answers to the same question. In contrast to other methods our metric has high specificity, and only ranks models as incoherent when the issues are glaring. Even so, we find narrow finetunes score poorly. Inspecting inconsistencies flagged by our method, we find that model organisms from the literature display severe issues such as identity conflation, introspection failures and rationalizations. These findings suggest that the pathologies induced by narrow finetuning may limit what these models can tell us about coherent misaligned behaviour.

0.41