CS Paper Radar

USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation

Md Khalid Syfullah, Alvi Ataur Khalil

cs.CV cs.AI ·submitted 2026-09-10 19:56 UTC · v1

Abstract

Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.

Closest papers

Nearest neighbours by meaning across the whole index.

Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models

cs.CV cs.AIcs.CR ·2026-09-14 ·Md Khalid Syfullah, Alvi Ataur Khalil
similarity 0.74
Abstract

Visual Question Answering (VQA) with Vision-Language Models (VLMs) is increasingly used in privacy-sensitive and bandwidth-constrained settings. Federated Learning (FL), Split Learning (SL), and U-Shaped Split Learning (USL) keep raw data local, but transmitting all visual tokens across a model partition remains costly and can expose private information. We propose QPriv-VL, a question-guided, privacy-aware token-pruning framework for FL, SL, and USL that prunes visual tokens before transmission based on task utility and privacy sensitivity. Its core component is a lightweight Dynamic Threshold Predictor (DTP) that jointly estimates a sample-specific pruning ratio and a token-level retention mask in one forward pass. DTP combines question relevance, computed from cross-modal similarity between visual patches and the pooled question embedding, with a sensitivity signal derived from frozen DINOv2 features. This allows the model to suppress potentially sensitive regions while preserving patches useful for answering the question, without requiring sensitivity labels. We evaluate QPriv-VL on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against four privacy attack families: FSHA, FORA, iDLG, and attribute-inference membership inference attacks. DTP matches or outperforms fixed-ratio pruning while using substantially fewer transmitted tokens. On VQA-RAD, it reduces membership-inference attack success from 0.99 to 0.76-0.79, lowers FSHA and FORA reconstruction PSNR relative to fixed-ratio pruning, and preserves competitive VQA accuracy using about 40% of the original visual-token budget. A sensitivity exclusion ratio of 1.20 +/- 0.18 indicates preferential removal of privacy-sensitive patches, while explainability analysis shows that retention adapts to question semantics rather than generic visual saliency.

0.11

FedPix: Learning What to Share with Less Leakage in Federated Intelligence

cs.CR cs.AI ·2026-10-15 ·Jingwei Zhao, Tiantong Sun, Yuanyuan Song, Jinhuan Guo, Zishuo Wei +4 more
similarity 0.72
Abstract · 31 pages, including appendices. Jingwei Zhao and Tiantong Sun contributed equally. Corresponding author: Xiaojin Zhang

Existing federated intelligence paradigms primarily focus on how models are trained or aggregated, while overlooking a more fundamental question: \textbf{what information should leave the client?} Existing methods often share model updates that users cannot easily inspect, or data representations that may reveal sensitive information. Moreover, generating these representations can be computationally expensive for resource-constrained clients. We observe that image classification does not require all the visual details needed to reconstruct the original image. This suggests that clients could share less information while preserving task performance. However, learning what to preserve is challenging when the server cannot access raw client data. To address this challenge, we propose \textbf{FedPix}, which learns to share compact, visually inspectable images instead of model updates or intermediate features. The key idea is to use the shared representations themselves to learn what information should be preserved and what can be discarded. By balancing task utility and reconstruction resistance, FedPix enables the server to improve the representations without accessing raw client data, while keeping client-side computation lightweight. Extensive experiments on BloodMNIST, CIFAR-10, FashionMNIST, and MNIST demonstrate that FedPix achieves strong reconstruction resistance, with reconstruction PSNR as low as 6.87~dB and synthetic PSNR as low as 6.74~dB. Meanwhile, FedPix reduces client runtime by up to 96.0\% (from 1223~s to 49~s on CIFAR-10), while maintaining competitive classification accuracy of 99.18\% on MNIST, 89.97\% on FashionMNIST, 82.24\% on BloodMNIST, and 65.63\% on CIFAR-10.

0.22

FedVP: Federated Learning across Heterogeneous Models by Pooling Locally Trained Verifiers

cs.LG ·2026-11-10 ·Yifan Li, Xi Chen
similarity 0.70
Abstract

Conventional federated learning relies on the shared model backbone to aggregate client updates, limiting collaboration among institutions that deploy large language models (LLMs) from different families and at different scales. We propose Federated Verifier Pool (FEDVP), a new federated learning paradigm that enables heterogeneous clients to collaborate through shared verification rather than parameter aggregation. Each client trains a verifier using local data and responses generated by its chosen LLM backbone. The server pools these independently trained verifiers and combines their judgments to select answers without merging model parameters. This allows both LLM backbones and verifiers to differ in architecture and size, while keeping the LLM backbones frozen and deriving collaborative gains from improved answer selection. We evaluate FEDVP across five public medical question-answering datasets treated as federated clients, using Qwen and LLaMA backbones ranging from 3B to 14B parameters. FEDVP consistently improves performance across all five clients and outperforms the strongest conventional federated baselines by 1.46-5.79 percentage points in mean accuracy.

0.13

Coordination on a Budget: Federated Active Learning with Few Labels

cs.LG ·2026-08-19 ·Liam Mohr, Daphna Weinshall
similarity 0.66
Abstract · 16 pages

Federated Active Learning (FAL) addresses the dual challenges of data privacy and label scarcity, where the absence of a global data view introduces additional hurdles for coordinated query selection. We study cross-silo FAL in the low-budget regime, where annotation decisions are most critical. We characterize, both theoretically and empirically, a heterogeneity reversal: in low-budget settings, homogeneous (IID) data requires stronger coordination to avoid redundant queries, whereas heterogeneous data naturally promotes diversity; this trend reverses at higher budgets. Thus, in contrast to the standard federated learning (FL) narrative where heterogeneity is a primary challenge, we show that IID settings are more challenging for query selection in FAL. Motivated by these findings, we propose a new FAL framework that utilizes federated representation learning to align client data in a shared embedding space. This enables the server to perform globally coordinated active selection over optionally obfuscated client embeddings, while annotation remains local to each client. Although our framework operates in the more challenging low-budget regime, it achieves performance that surpasses existing FAL methods even when they are given substantially larger annotation budgets, demonstrating the value of centralized coordination under privacy constraints.

0.07

Explainable Federated Learning via Global-Local Attribution Alignment

cs.LG cs.AI ·2026-11-16 ·Dawood Wasif, Terrence J. Moore, Chang-Tien Lu, Jin-Hee Cho
top venue similarity 0.66
Abstract · 26 pages, 2 figures, 9 tables. Published in Proceedings of the 43rd International Conference on Machine Learning (ICML 2026)

Federated learning enables on-device training without centralizing data, yet existing systems still struggle to provide explanations that are both locally faithful and globally consistent under strict privacy and bandwidth constraints. Prior approaches either keep explanations siloed across clients, transmit heavy or sensitive artifacts, or replace expressive task models with interpretable surrogates that sacrifice accuracy. We propose xFedAlign, a model-agnostic framework that decouples task optimization in parameter space from explanation coordination in a compact group space. Each client distills a lightweight surrogate to produce private, per-class top-k attribution artifacts, which are robustly aggregated by the server into a Global Explanation Prior that softly aligns client explanations without constraining task learning. Across image, text, and tabular benchmarks with IID and non-IID partitions, xFedAlign matches FedAvg accuracy while consistently reducing explanation drift and improving deletion and insertion AUC relative to Local-XAI, FedAttr-Agg, and Fed-XAI, with only a few kilobytes of additional communication per round. Privacy and robustness evaluations further demonstrate reduced membership inference advantage and increased resistance to attribution poisoning, enabling consistent and trustworthy explanations in federated learning.

0.34

Enhancing Privacy in Federated Learning via Dual Obfuscation of Gradients and Training Images

cs.CR ·2026-08-20 ·Yuki Itabashi, Hiroto Sawada, Mare Hirose, Shoko Imaizumi, Hitoshi Kiya
published similarity 0.66
Abstract · 6 pages, 9 figures, accepted to APSIPA ASC 2026

Federated learning enables collaborative model training while keeping data locally at each client; however, recent studies have shown that training data can be reconstructed from shared model updates. To address this issue, this paper proposes a dual obfuscation method that enhances robustness against image restoration attacks by jointly obfuscating updated information and training images. The proposed method combines a robustness enhancement technique based on random binary weights, which randomly sets a portion of gradient elements to zero, with an image encryption technique. These techniques provide complementary protection by reducing the amount of original gradient information available to an attacker and the visual interpretability of reconstructed images, respectively. Furthermore, the image encryption technique allows independent keys to be used for each client and each image, avoiding explicit key sharing. Experimental results on an image classification task using a Vision Transformer (ViT) show that the proposed method reduces the visual information recovered by Attention Privacy Leakage (APRIL) under the evaluated settings without causing additional degradation in classification performance beyond that caused by image encryption. Although the proposed combination does not provide an absolute security guarantee, the results demonstrate the potential benefit of combining gradient modification and image encryption for privacy-enhanced federated learning.

0.32

Fed-IGTD: A federated learning-based framework for tabular-to-image encoding

cs.LG cs.CV ·2026-10-28 ·David Chushig-Muzo, Eva Milara, Francisco J. Lara Abelenda, Luis Carlos Estrada-Petrocelli, Diego H. Peluffo-Ordóñez
similarity 0.66
Abstract

Tabular-to-image encoding methods, which transform tabular data into image-like representations, have shown promising results in predictive tasks. Despite their benefits, these methods follow a centralized paradigm, in which the data are collected and models are trained on a central device. This raises privacy and security concerns, particularly the risk of data leakage, unauthorized data collection and access to users' private information. Federated Learning (FL) provides a decentralized framework in which multiple clients jointly optimize a shared model without directly sharing training data. We introduce Fed-IGTD, a horizontal FL-based approach in which clients share a common feature space and upload only local statistics, from which the server reconstructs the global correlation matrix and obtains a feature-to-pixel mapping to train a convolutional neural network via Federated Averaging (FedAvg). We studied two differential privacy (DP) mechanisms: (i) a one-shot Gaussian mechanism protecting the correlation-aggregation step; and (ii) a client-level DP-FedAvg variant protecting the training with privacy loss composed via a Renyi-DP accountant. We evaluated Fed-IGTD on five datasets under a systematic sweep of non-IID severity and privacy budgets. Privacy protection applied to the mapping-construction step has little impact on predictive performance. By contrast, DP-FedAvg substantially degrades utility at moderate privacy budgets with a small client cohort (K=5). Ablation study indicates that this degradation is driven mainly by the injected noise rather than by uniform averaging or update clipping. Macro-F1 exhibits sensitivity to performance degradation than accuracy under non-IID heterogeneity and class imbalance.

0.17

FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification

cs.CV ·2026-10-07 ·Ankita Das, Ambarish Parthasarathy, Sumohana S. Channappayya, C. Krishna Mohan
similarity 0.65
Abstract

Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails to capture token-level structure and cross-modal interactions prior to the classification stage. This is especially critical in biomedical applications under federated constraints, where data sharing is restricted, labeled data is scarce at each site, and it differs widely across institutions, leading to substantial statistical heterogeneity. To overcome this issue, we propose FedSSMCoOp, a federated few-shot image classification framework that enables multimodal learning while preserving data privacy. With the help of the SSM-based Vision Mamba and Cross Mamba blocks, and by optimizing only the soft-prompt and communication-prompt updates in the federated setting, the framework prioritizes both computation and performance. Importantly, this eliminates the need to use an external Large Language Model (LLM) for feature alignment. The framework is further trained and evaluated on various biomedical image datasets, and its performance is assessed. The proposed framework delivers stable performance relative to the baselines and is, on average, 1.96 times lighter. The corresponding script will be made available soon.

0.16