CS Paper Radar

SEAT: Sign-Type Evidence-Audited Transport for Multi-Label Tongue Diagnosis

Liuyi Xu, Qing Wang, Zikai Liang, An-Yang Lu, Yining Qian

cs.CV ·submitted 2026-11-15 03:21 UTC · v1
benchmarkcode/data

Abstract

Automated tongue-sign recognition can improve the consistency of non-invasive examinations that otherwise rely on subjective visual judgment. A single tongue image often exhibits multiple coexisting signs characterized by heterogeneous evidence, including localized morphology, global chromaticity, and inter-label dependence. However, existing methods typically rely on shared representations and uniform multi-label heads, making it difficult to verify whether predictions stem from clinically relevant evidence rather than spurious visual correlations. We propose SEAT, an evidence-audited framework that matches distinct evidence types with dedicated prediction and evaluation mechanisms. Built on frozen DINOv2 patch features, SEAT couples mask-guided sign-specific attention for localized morphology, a semi-relaxed optimal transport pathway over LAB/HSV color prototypes to enhance pale-tongue recognition, and a co-occurrence-aware transport regularizer to shape the batch-level predicted label marginal. These components are evaluated through audits matched to their evidence types. On the TongueDx benchmark, SEAT achieves a mean macro-F1 of 72.59 with a standard deviation of 1.04 across six random seeds, with a six-model ensemble reaching 74.15. The ensemble outperforms a backbone-matched baseline and a larger, domain-specialized vision-language model by 8.06 and 2.99 percentage points, respectively. Our results demonstrate that aligning dedicated prediction pathways and audits with sign-specific characteristics improves multi-label accuracy while offering more targeted evidential grounding than uniform post-hoc spatial explanations. Code is available at https://github.com/NEU-TCM-Lab/Sign-type-Evidence-Audited-Transport.

Comment
8 pages, 3 figures, 4 tables

Closest papers

Nearest neighbours by meaning across the whole index.

OmniTongue: A Comprehensive MLLM Benchmark for Automated Tongue Diagnosis in Traditional Chinese Medicine

cs.CV ·2026-11-17 ·Liuyi Xu, Yun Guo, Shusheng Li, Xuewen Zhang, Yining Qian +1 more
benchmarkcode/data similarity 0.63
Abstract · 11 pages, 2 figures, 3 tables

Multimodal Large Language Models (MLLMs) have excelled in Western medical imaging, yet their potential in Traditional Chinese Medicine (TCM) tongue diagnosis remains systematically unevaluated. Tongue diagnosis requires sophisticated global and local perception, serving as a rigorous touchstone for assessing the visual understanding depth of MLLMs. However, existing datasets are limited by single dimensions, coarse granularity, and ambiguous labels, hindering the evaluation of authentic clinical capabilities. To bridge this gap, we introduce OmniTongue, the first comprehensive, fine-grained Visual Question Answering benchmark for TCM tongue diagnosis. This benchmark integrates 1,351 expert-validated images and 21,616 instruction pairs. Its core design features 16 clinical dimensions for multidimensional evaluation, region-specific annotations for pathological localization, and 66 fine-grained labels for precise diagnostic representation. Extensive evaluations of existing MLLMs reveal significant performance gaps, demonstrating that medical-specific models lack domain advantages, while universally poor Macro-F1 scores expose a reliance on statistical shortcuts rather than genuine understanding of tongue pathology. These findings not only highlight the current limitations of MLLMs in interpreting regional medical features but also underscore OmniTongue's significance for developing robust automated tongue diagnosis models. The code and dataset will be made publicly available in a future release.

0.30

Bilateral Evidence Selection and Reasoning for Mammography Vision--Language Models

cs.CV cs.AI ·2026-11-19 ·Zhengbo Zhou, Dooman Arefan, Shandong Wu
similarity 0.61
Abstract

Medical vision--language models (VLMs) often need to identify small, subtle lesions within high-resolution images containing extensive normal anatomy and redundant visual information. Visual-token compression improves efficiency, but preserving salient regions alone can discard the context needed for diagnostic interpretation. This context can extend across images: in bilateral mammography, the opposite breast provides a patient-specific anatomical reference. Yet retaining such evidence does not ensure that the model uses it effectively. We propose Bilateral Evidence Selection and Reasoning (BESR), which connects evidence preservation with comparative assessment through a bilateral-guided token selector and a bilateral reward. Under a shared token budget, the selector jointly prioritizes lesion appearance and its local and bilateral context, while the reward encourages assessments that benefit from the preserved comparison. Experiments on a bilateral dataset constructed from CBIS-DDSM and on VinDr-Mammo demonstrate improved evidence retention and diagnostic performance under constrained visual-token budgets. Inference benchmarking further shows reduced prefill latency.

0.30

DINO Eats Geometry: Self-Supervised Sign Representations that Absorb Articulated Hand Geometry

cs.CV cs.CL ·2026-09-05 ·Junyi Hu, Zhewen He, Haomian Huang, Zhenhua Li, Zhifei Li +2 more
hot direction similarity 0.59
Abstract · 22 pages, 7 figures

Handshape and orientation are geometric components of sign language, yet many sign encoders learn from appearance alone. We introduce SignDEG (DINO Eats Geometry), a temporal sign encoder that uses articulated hand geometry as a pre-training target and requires only video at inference. Frozen DINOv3 features of tracked hands and faces feed per-stream temporal Transformers. Geometric latent prediction (GLP) supervises an intermediate block with WiLoR latents, MANO articulation, and 3D joints, including at masked frames. Geometric relation transfer (GRT) matches cross-video frame-similarity distributions at the final block. Both objectives are learned jointly with temporal self-distillation; WiLoR and the geometry heads are then discarded. Under the SHuBERT public-data evaluation protocol, SignDEG reaches 17.9/24.5 BLEU-4 on How2Sign/OpenASL, compared with 16.2/23.2 for SHuBERT. Rank-1 adaptation improves ASL Citizen and Sem-Lex recognition by 0.02 Recall@1 each, and phonological classification improves across 16 attributes. Removing GRT while retaining GLP reduces How2Sign BLEU-4 by 1.4 and ASL Citizen Recall@1 by 0.034. The results support transferring articulated structure into an appearance encoder without running a hand reconstruction model at deployment.

0.37

EviDent-CBCT: Evidence-Bottlenecked Report Generation from Dental CBCT under Non-Exhaustive Report Supervision

cs.CV cs.AI ·2026-10-01 ·Ruiyang Hao, Zhi Qin Tan, Yulan He, Owen Addison, Yunpeng Li
similarity 0.59
Abstract

Dento-maxillofacial cone-beam CT (CBCT) reports may contain dozens of tooth-specific, anatomical, and spatial findings from a single 3D scan. Learning to generate such reports from limited clinical data is challenging because routine reports may not exhaustively document image findings, and a non-mention may reflect either absence or non-reporting. We present EviDent-CBCT, an evidence-bottlenecked framework designed for this incomplete supervision. An anatomy-aware network maps each CBCT scan to a discrete record of tooth-level, global, and tooth-IAC evidence. A dental-logic consistency projection reconciles incompatible evidence before a deterministic renderer and an image-blind local language model generate the report using only this record. For tooth-level evidence, reliability-aware training uses eligible non-mentions as reduced-weight negatives, while unreported global and tooth-IAC labels remain unknown. A metal-sensitive input channel preserves intensity cues from dental materials. Across three validation runs, EviDent-CBCT achieves $0.666\pm0.006$ merged evidence set-F1 and $0.402\pm0.003$ RadFact-Lite-Dental logical-F1, versus $0.371\pm0.018$ for the strongest controlled direct baseline. In the ODIN 2026 challenge, it ranked second in automated evaluation and third in blinded clinical Arena comparison on the hidden test set. These results support the discrete evidence record as an effective and auditable interface for CBCT report generation.

0.07

Auditable Hierarchical Inference with Multimodal Evidence and Bayesian Active Inquiry

cs.LG ·2026-10-28 ·Yucheng Liu, Everett X. Wang
similarity 0.58
Abstract

Most multimodal classifiers assume that all useful evidence is available upfront, whereas clinical inquiry is sequential. We present a synthetic evaluation framework separating objective multimodal inference from active inquiry. An objective model maps synthetic tongue, bilateral multipressure pulse, and acoustic observations to eight syndrome families. A provenance-bearing Knowledge Core and training-only answer model support Bayesian narrowing over 33 fine terms using 25 typed questions, while seven safety questions remain outside the diagnostic posterior. Weak shared symptom factors limit fine-term influence on objective phenotypes, and an objective-only probe tests leakage. In a preregistered 600-case synthetic evaluation, objective family accuracy was 87.0%. Cost-sensitive expected information gain (cost-EIG) raised family accuracy to 95.5% and exact fine-term accuracy to 42.5%, versus 92.7% and 33.0% for an unoptimized fixed-order baseline. Selective family accuracy reached 96.9% at 95.3% coverage. However, cost-EIG asked more questions on average than fixed order (9.795 versus 9.352). Missing-audio routing retained 94.0% family accuracy, and a workflow invariant enforced abstention when all objective modalities were absent. Permuting fine-term answer likelihoods reduced family accuracy to 78.7%. Five of six frozen checks passed, but the question-efficiency hypothesis failed. Comparisons are descriptive because only aggregate, not paired case-level, outputs were retained. These results establish a reproducible synthetic workflow, not clinical diagnostic or treatment validity.

0.32

CAST: Calibrating Correspondences for Training-Free Biomedical In-Context Segmentation

cs.CV ·2026-11-17 ·Mingze Jiang, Jiashu Han, Haonan Chen, Zhiyi Li, Yilun Du +1 more
similarity 0.58
Abstract · 10 pages, 6 figures

Biomedical image segmentation must keep up with new anatomical structures, subjects, imaging modalities, and acquisition protocols, yet dense expert annotations are scarce and models trained on one cohort often struggle to transfer. In-context segmentation (ICS) offers a way out: a new structure is specified by a few annotated exemplars at inference time. However, reliable correspondence remains challenging in biomedical images, where adjacent structures often share similar appearances, and variations across subjects and imaging conditions further obscure their distinctions. We introduce CAST, a training-free framework built on a frozen DINOv3 backbone to calibrate fine-grained correspondences. Its stability test improves feature reliability at both global and patch-specific levels, while its coherence test assesses whether candidate correspondences are supported by consistent local structural context. Together, these tests enable more reliable correspondence across diverse anatomical structures and imaging conditions. Across eight biomedical benchmarks spanning 2D and 3D segmentation, cross-subject, and cross-modality transfer, CAST outperforms the strongest overall baseline by 12.34 percentage points in Dice and ranks first across all six volumetric settings. The versatility of CAST further opens up opportunities for challenging biomedical applications, including 2D-to-3D segmentation and atlas label transfer.

0.23

Grounding Image Composition with Label-Conditioned Visual Evidence

cs.CV ·2026-10-29 ·Armin Dadras, Tingyu Lin, Robert Sablatnig, Franziska Proksa, Markus Seidl
publishedbenchmark similarity 0.57
Abstract · Accepted at WACV 2027

Image composition labels such as center, diagonal, or rule of thirds describe visual structure rather than object categories. KU-PCP provides image-level labels and class-specific geometric elements, but does not show which visible regions people associate with each of its nine classes. We introduce KU-PCP Elements (KU-PCPE), a label-conditioned region-evidence benchmark for photographic composition. Given an image and a label already judged applicable, the task is to segment the visible regions that explain why the label applies. KU-PCPE contains 4,093 single-annotator training masks and a 1,327-task expert test split with three masks per image-label task. The annotations also include exploratory segment-versus-line ratings that describe whether region or semantic-guideline evidence better explains the label. A held-out expert reaches .867 Dice against the other experts, while the strongest deployable model reaches .664, a proposal oracle .672, and the best vision-language model (VLM) proposal selector .395. KU-PCPE exposes the gap between recognizing a composition label and grounding the human-aligned visual evidence that makes it plausible in a particular image.

0.39

Learning from Unlabeled Photographs for General Oral Image Analysis

cs.CV ·2026-11-03 ·Zhong-Shi Zhang, Zi-Li Yu, Xiao Xiao, Huan Liu, Gui-Song Xia +2 more
similarity 0.57
Abstract

Oral photographs contain rich clinical information and hold significant potential for artificial intelligence applications; however, most models are trained for individual tasks and rely on costly expert annotations. We hypothesized that self-supervised training on unlabeled oral photographs could produce transferable visual representations for heterogeneous vision tasks. To test this hypothesis, we developed DOVE (Dental Optical Visual Encoder) by domain-adaptive self-supervised pretraining on approximately 530,000 unlabeled oral photographs collected from publicly accessible sources. DOVE was benchmarked against 3 established baselines across 5 classification datasets and 4 segmentation datasets, covering both frozen backbone evaluation and end-to-end finetuning. DOVE achieved the strongest overall performance, ranking best in few-shot transfer and label efficiency on most benchmarks. Under the 20-shot frozen-backbone protocol, DOVE outperformed the second best DINOv3 by 4.37 percentage points in mean macro-F1 across classification datasets and by 4.49 percentage points in mean macro-Dice across segmentation tasks. Under end-to-end finetuning, DOVE achieved at least 80% of the strongest full-data baseline's performance across most evaluation datasets using only 10% of the labeled training data. Qualitative retrieval and feature analyses further indicated better organization of image-level and spatial representations. These findings support DOVE as a generalizable and label-efficient backbone for oral vision research.

0.17