CS Paper Radar

Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

Jinen Setpal, Chaoyue Liu

cs.CV cs.LG ·submitted 2026-11-05 15:16 UTC · v1
top venue

Abstract

Image classification models often encounter objects in diverse and unpredictable contexts at test time, which may differ substantially from those seen during training. This phenomenon, termed context shift, exposes a key fragility in neural networks: when models rely on non-causal contextual cues for prediction, shifts in context can lead to significant performance degradation. In this paper, we propose a novel training framework to improve the robustness of convolutional neural networks under context shift. Our approach leverages spatial feature attribution to guide models toward making predictions that rely less on contextual regions and more on object-relevant features. As a first step, recognizing a theoretical limitation of existing feature attribution methods we introduce a new variant, ContrastiveCAMs, which produces more faithful attribution maps of model predictions. Building on ContrastiveCAMs, we further propose Context-Regularized Cross-Entropy (CR-CE), a modification of the standard cross-entropy loss that regularizes the model's attention by suppressing the influence of contextual regions, thereby improving robustness to context shift. We evaluate the effectiveness of our approach on several medium-to-large scale datasets (Waterbirds, Spawrious, ImageNet/ImageNet-BG, Hard-ImageNet), and report consistent improvements in context-shift robustness.

Comment
accepted to NeurIPS 2026
Journal ref
NeurIPS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Seeing Through Distractions: Context-Stable Attribution via the Least Core

cs.CV cs.GT ·2026-10-26 ·Sai Ganesh Nagarajan, Toshinori Yamauchi, Hiroshi Kera
similarity 0.70
Abstract · 22 pages, 6 figures, 3 tables

Shapley-value-based explanations are widely used for feature attribution in machine learning. However, in computer vision, a feature may reduce the target-class score when added to the complete genuine image, even when the predicted label remains unchanged. We formalize such features as contextual distractors and investigate their attributions under semivalues and the least core. Under a weak average marginal monotonicity assumption, we derive quantitative semivalue--least-core gap bounds and sufficient conditions based on the probability of sampling smaller contexts. We further show that shifting semivalue weights toward larger contexts decreases the signed gap. Finally, controlled image-classification experiments and evaluations on Salient ImageNet provide strong empirical support for our theoretical findings.

0.21

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

cs.CV cs.AI ·2026-09-07 ·Seungmin Oh, Seunghun Kang, Jongbin Ryu
publishedcode/data similarity 0.69
Abstract · Accepted by British Machine Vision Conference (BMVC) 2026

Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.

0.41

RepICL: Reusable In-Context Prediction Across Heterogeneous Representation Spaces

cs.LG ·2026-10-05 ·Yu-Hsiang Liu, Kuan-Yu Chen, Chih-Sheng Chen, Meng-Hsuan Chang, Yu-Chen Den +1 more
benchmark similarity 0.67
Abstract

Frozen representations are widely reused for downstream classification, yet each new task typically requires fitting a new predictor. We ask whether the few-shot prediction procedure itself can instead be learned once and reused across datasets and representation spaces. To study this question, we introduce RepShiftBench, comprising 1,218 encoder--dataset tasks across text, image, and audio, with separate evaluation of generalization to unseen datasets, unseen encoders, jointly unseen datasets and encoders, and unseen modalities. The benchmark exposes a substantial gap: Logistic Regression fitted independently on each episode outperforms every evaluated in-context learner across all settings. We introduce RepICL, a meta-trained in-context learner that canonicalizes each episode through episodic whitening before prediction. Its inductive variant, RepICL-I, surpasses Logistic Regression in all 12 benchmark settings, while RepICL-T substantially outperforms existing transductive methods. Ablations identify episodic whitening as the primary source of these gains, while showing that it is not a universally beneficial preprocessing step. Across both variants, the gains concentrate on queries for which simple support prototypes favor the wrong class or provide little separation between the true class and competing classes. Transduction provides its largest additional gains when limited support coverage gives a misleading view of class separation. Together, these results demonstrate that a shared few-shot prediction procedure can generalize beyond the representation spaces observed during training.

0.23

Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

cs.LG ·2026-08-28 ·Kazi F. Akhter, Ibna Kowsar, Manar D. Samad
publishedcode/data similarity 0.66
Abstract · Accepted for publication at Neural Networks (Elsevier)

Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.

0.45

SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging

cs.CV ·2026-08-18 ·Shenhav Nadir, Meir Yossef Levi, Eyal Gofer, Guy Gilboa
similarity 0.66
Abstract

Despite the rapid progress of deep neural networks in visual recognition, their adoption in high-risk medical applications remains limited due to reliability and robustness concerns. Models may exploit spurious correlations, particularly in medical imaging, where devices or treatment artifacts often co-occur with pathology. In small or imbalanced datasets, such cues further reduce worst-group performance and undermine clinical trust. To solve these issues, two major challenges should be addressed: identifying dataset-specific spurious cues, which typically require domain knowledge, and mitigating reliance on them. To tackle both, we propose SpurCon, a lightweight framework based on a novel supervised contrastive loss formulation that leverages available metadata and predicted spurious labels to enhance robustness. We introduce a fast few-shot procedure, without network training, to estimate spurious labels using a small number of expert-annotated samples. We then propose a weighted supervised contrastive objective, WtSupCon, that reshapes the representation geometry by assigning sample-specific weights that depend on the [pathology, spurious, metadata] combination. For example, the highest weight is assigned to samples that differ only in their spurious label. This yields highly similar representations for images with the same metadata and pathology, differing only in the predicted spurious label. Our method operates on pretrained image encoders (such as BiomedCLIP) and trains only a lightweight projection head. We evaluate SpurCon on a synthetic setting and on Waterbirds, CheXpert, a chest X-ray classification dataset, and ISIC 2020, a skin cancer classification dataset. Our approach delivers the best spurious-mitigation performance, balancing well worst-group and overall accuracy on multiple datasets.

0.16

Learning Continuous Source Responses For Generalizable AI-Generated Image Detection

cs.CV ·2026-09-13 ·Manni Cui, Ruiqi Liu, Zijian Yu, Hao Tan, Zibo Wei +6 more
code/data similarity 0.66
Abstract

Advances in image generation have made synthetic images increasingly difficult to distinguish from real photographs, raising concerns about the trustworthiness of visual media. Existing AI-generated image detectors often perform well on in-domain data, but their robustness and cross-generator generalization remain limited. These limitations are commonly attributed to overfitting to shortcut cues. Although many methods seek to suppress shortcut learning, most retain binary classification as the training task without reconsidering how the task itself shapes the learned representations. We introduce CuRe, a framework for learning Continuous Source Responses that revisits authenticity detection from the perspective of the training task. CuRe reformulates backbone adaptation as regression of real-generated mixing ratios, providing finer supervision that encourages the model to capture authenticity-related variation beyond binary endpoint separation. We further select a compact source-response subspace to suppress nuisance variation and limit the final classifier's access to potential shortcut cues. Across ten public benchmarks, CuRe achieves an average balanced accuracy of 89.7%, exceeding the second-best method by 5.2 percentage points. Further experiments demonstrate consistent generalization gains across visual backbones and strong robustness to common image degradations. Code is available at https://github.com/manic-cui/CuRe

0.24

Spatial Attention Noise Masking for Causally Sufficient Interpretability

cs.CV ·2026-08-12 ·Benjamin Formby, Kuang-Ching Wang, D Hudson Smith
similarity 0.66
Abstract

We present a novel causal approach to interpretability for computer vision models that dynamically masks the input image prior to classification. The interpretability of deep learning predictions is critical in high-stakes fields such as medical imaging, security, and autonomous driving. Most interpretability methods are applied passively to already trained models, which typically result in correlational rather than causal explanations. Existing causal interpretability methods are limited to post hoc analysis, weakening the causal claims. Additionally, existing active methods generally lack explanations that explicitly assign responsibility to input features. This work proposes a spatial attention noise masking framework that provides causal explanations about the features sufficient for the prediction. The proposed framework consists of: 1) a UNet-style mask generator, and 2) a Resnet18 encoder and linear classifier that classifies both masked and unmasked versions of an input image. The generated masks are regularized to be sparse and spatially smooth, while masked image embeddings are constrained to remain consistent with embeddings from the corresponding unmasked images. The resulting masks can be interpreted as feature attribution maps that are competitive with related interpretability methods while additionally providing strong causal explanations of model predictions. Quantitative evaluations demonstrate mask faithfulness, near-baseline classification performance across five classification tasks despite substantial masking of image information, and robustness to distribution shifts such as background swapping and natural adversarial examples. Qualitative comparisons further demonstrate mask behavior and competitive interpretability relative to state-of-the-art feature attribution methods.

0.04

Evaluating Contextual Bias in CNN Image Classification: Evidence from Agricultural Benchmark Datasets

cs.CV ·2026-09-13 ·Abhilekha Dalal, Michael Okonoda, Eder Martinez, Lior Shamir
benchmark similarity 0.66
Abstract

Convolutional neural networks (CNNs) are typically evaluated using held-out classification accuracy, an approach that presupposes predictions are based primarily on the intended object of interest rather than incidental surrounding context. We test this assumption in CNN-based agricultural image classification by comparing model performance on original images with performance on background-dominated patches extracted from the same images across eight publicly available agricultural benchmark datasets and four widely used CNN architectures. Background-dominated patches were classified above dataset-specific random chance for six of the eight datasets, and substantially above chance for four of them, indicating that contextual information contributes to model predictions for the majority of datasets evaluated. For these four datasets, we further evaluated whether this behavior reflected genuine class-discriminative information or was primarily attributable to class imbalance using macro-averaged precision, recall, and F1 together with class-balanced test subsets. The results show that contextual reliance does not admit a single explanation: class imbalance accounts for a substantial portion of the observed signal for some datasets and architectures, whereas above-chance contextual classification persists after balancing for others. Together with previous evidence from curated object recognition and cancer pathology imaging, these findings support the growing view that contextual bias is a recurring characteristic of CNN-based image classification rather than a phenomenon confined to a single application domain. More broadly, this work provides a systematic framework for quantifying contextual bias across heterogeneous image datasets by combining dataset-specific random-chance baselines, contextual bias categorization, macro-averaged evaluation, and class-balanced robustness analysis.

0.18