CS Paper Radar

AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration

Xiangrui Xu, Sicong Liu, Zhiwen Yu, Zimu Zhou, Bin Guo

cs.LG ·submitted 2026-11-16 03:32 UTC · v1
top venueposition

Abstract

Multimodal foundation models (MFMs) support generalizable inference in distributed mobile sensing applications by integrating heterogeneous data streams. However, existing MFM inference pipelines typically require complete and synchronized multimodal inputs, whereas mobile sensing streams arrive asynchronously due to network variability and heterogeneous data volumes across modalities. Waiting for all modalities increases end-to-end latency, while inference with incomplete inputs can reduce accuracy. We argue that multimodal inference should depend on the utility of available information rather than modality completeness. Based on this principle, we propose AdaModalX, a progressive multimodal inference framework that decouples inference execution from modality synchronization through runtime utility-aware orchestration. At each device, importance-aware data uploading uses a lightweight probe distilled from the edge-side MFM to allow reliable samples to exit locally while uncertain samples continue uploading. At the edge, on-demand MFM activation constructs candidates from fully received samples and, when needed, imputed features derived from partially received delayed inputs. A lightweight contribution assessor built on frozen MFM encoders estimates candidate utility and activates the MFM once the candidate meets its corresponding utility threshold. Experiments on four mobile sensing applications show that AdaModalX improves the accuracy-latency trade-off, achieving up to 26.97% accuracy improvement and up to 64.75% end-to-end latency reduction.

Journal ref
IEEE Transactions on Mobile Computing, 2026
DOI
10.1109/TMC.2026.3742480

Closest papers

Nearest neighbours by meaning across the whole index.

AdaFlowLite: Scalable and Non-blocking Inference on Asynchronous Mobile Data

cs.LG cs.AI ·2026-11-16 ·Sicong Liu, Fengmin Wu, Yuan Gao, Bin Guo, Zimu Zhou +2 more
top venue similarity 0.74
Abstract

The rise of mobile devices equipped with numerous sensors, such as LiDAR and cameras, has driven the adoption of multi-modal deep intelligence for distributed sensing tasks, such as smart cabins and driving assistance. However, the arrival time of mobile sensory data vary due to modality size and network dynamics, which can lead to delays (if waiting for slow data) or accuracy decline (if inference proceeds without waiting). Moreover, the diversity and dynamic nature of mobile systems exacerbate this challenge. In response, we present a shift to \textit{opportunistic} inference for asynchronous distributed multi-modal data, enabling inference as soon as partial data arrives. While existing methods focus on optimizing modality consistency and complementarity, known as modal affinity, they lack a \textit{computational} approach to control this affinity in open-world mobile environments. % \sysname pioneers the formulation of structured cross-modality affinity in mobile contexts using a hierarchical analysis-based normalized matrix. This approach accommodates the diversity and dynamics of modalities, generalizing across different types and numbers of inputs. Employing an multi-modal lightweight Swin Transformer (MMLST), \sysname facilitates real-time and flexible data imputation, adapting to various modalities and downstream tasks without retraining. Experiments show that \sysname significantly reduces inference latency by up to 80.4\% and enhances accuracy by up to 62.1\%, while achieving nearly a 50\% reduction in energy consumption, outperforming status quo approaches. Also, this method can enhance LLM performance to preprocess asynchronous data.

0.35

MobiInfer: Accelerating On-Device GUI Agents with Workload-Aware Inference

cs.OS ·2026-08-11 ·Siming Chen, Erhu Feng, Dahu Feng, Yubin Xia
benchmark similarity 0.66
Abstract

Mobile GUI agents assist users in executing phone tasks through iterative observe-reason-act loops. However, deploy ing multimodal GUI agents directly on smartphones is challenging due to strict end-to-end latency requirements and significant memory overhead. We observe that GUI-agent work loads exhibit recurring workflows and structured I/O: their prompts contain reusable context, while their outputs follow fixed schemas with phase-dependent predictability. Based on this observation, we present MobiInfer, a workload-aware inference engine for on-device multimodal GUI agents. MobiInfer combines trajectory-aware prefix reuse, phase-indexed vocabulary specialization, and schema-guided speculative decoding to eliminate redundant prefill computation and reduce structured-generation overhead. We implement MobiInfer on top of MNN and evaluate it on AndroidWorld and a real world dataset using three real Android devices. Across the four model-benchmark settings, MobiInfer achieves average speedups of 2.6x in prefill and 2.5x in decoding, reduces end-to-end task latency by 2.4x on average, and cuts total deployment storage by 30%, while limiting task-success degradation to less than one percentage point.

0.04

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

cs.AI cs.LG ·2026-09-28 ·Zongshang Shen, Wangsong Yin, Daliang Xu, Mengwei Xu, Xuanzhe Liu
similarity 0.65
Abstract

On-device streaming omni-modal inference safeguards user privacy and eliminates prohibitive per-token API costs, but faces a critical bottleneck: the continuous influx of multimodal data rapidly exhausts constrained memory and compute budgets via monotonic KV cache growth. Existing sparse attention methods fall short, either incurring prohibitive online estimation latency or destroying interleaved cross-modal context, while failing to resolve physical memory fragmentation. We present OmniTide, the first algorithm-system co-design tailored for efficient on-device streaming omni-modal inference. Driven by the observation of modality-aware structural sparsity, OmniTide adopts a unit-based abstraction with two components: (1) At the algorithm level, OmniPick logically retains critical multimodal context based on unit boundaries and modality importance to preserve task accuracy; (2) At the system level, OmniPage physically partitions the cache by retention likelihood and dynamically compacts surviving sparse tokens, minimizing both memory fragmentation and data-movement overhead. Extensive evaluations across three streaming benchmarks and two consumer-device architectures show that OmniTide achieves up to $12.72\times$ kernel speedups and $2.40\times$ lower stream-loop latency. On StreamingBench, it improves accuracy by up to 18.0 percentage points over sliding-window baselines at comparable session cost. OmniPage further reduces the physical KV span by up to 26.7% relative to native logical eviction, unlocking real-time, infinite-context streaming on edge devices.

0.18

EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

cs.LG cs.DC ·2026-09-10 ·Motahare Mounesan, Irfan Khan
benchmark similarity 0.64
Abstract

Recent advances in multimodal large language mod- els (MLLMs) have opened new opportunities for edge intelligence by enabling reasoning across heterogeneous sensor modalities, such as vision, text, and telemetry data. However, deploying these capabilities on resource-constrained edge platforms remains challenging due to the substantial computational, memory, and communication demands of modern MLLMs. Rather than transmitting raw sensor observations or partitioning neural networks at intermediate layers, Edge Multi-Modal Intelligence (EMMI) communicates a compact representation between edge devices and server resources, enabling communication-efficient edge MLLM inference. To achieve this, EMMI performs modality-specific encoding, cross-modal representation fusion, and learned compression at the edge, transmitting only a compact latent representation to server-side resources for high-capacity MLLM reasoning. This representation-centric design reduces communication overhead, preserves local data privacy, and provides a fixed-size interface between heterogeneous edge devices and server-side MLLMs. Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.

0.11

CST: Collaborative Selective Transmission for Communication-Efficient Multimodal Edge Inference

cs.LG ·2026-08-22 ·Hai Chi, Junrui Zhang, Rui Ning, Chonggang Wang, Robert Gazda +2 more
similarity 0.63
Abstract · 11 pages, 6 figures, 6 tables

Collaborative multimodal inference improves edge perception by combining observations from distributed sensing devices, but transmitting high-dimensional helper representations incurs substantial communication overhead and can lead to high end-to-end latency. Existing communication-efficient methods reduce payloads through compression, semantic coding, or feature selection, yet typically optimize compactness or task relevance without explicitly accounting for information already represented at the main device. Consequently, task-relevant but redundant helper features may still consume bandwidth. We present Collaborative Selective Transmission (CST), a main-directed query--response framework that retrieves only helper information complementary to the current main representation. Inspired by Partial Information Decomposition and the Multiview Redundancy Assumption, CST learns sample-adaptive, helper-specific sparse retrieval supports while discouraging retrieval of semantics already covered by the main device or duplicated across helpers. During inference, the main device transmits only support indices, and each helper returns the corresponding latent values, avoiding dense helper-feature exchange. Across three real-world multimodal sensing benchmarks, CST transmits no more than 14.18% of helper feature values while achieving best or near-best task performance among the evaluated methods. Experiments on a five-node NVIDIA Jetson Orin Nano testbed across 5--100 Mbps demonstrate up to a $4.27\times$ speedup over Transmit-All in end-to-end inference, confirming practical end-to-end latency reductions.

0.20

MODAIC: Coordinated Asynchrony for Decentralized Multimodal Learning

cs.DC cs.NI ·2026-10-12 ·Yanhang Shi, Xiaoyu Wang, Houwei Cao, Jian Li, Yong Liu
similarity 0.63
Abstract

Multimodal decentralized learning enables collaboration across heterogeneous edge devices without sharing raw data, but differing modality sets create coupled computation and communication bottlenecks. Multimodal agents support cross-modal alignment while training more model components and sharing bandwidth across multiple modality overlays. Under bandwidth limits, these agents can become hot spots: transfers miss aggregation windows, and their updates become stale or underrepresented, degrading worst-agent accuracy. Asynchrony removes global blocking but does not by itself coordinate local cross-modal interaction with timely peer-update exchange. We present MODAIC (Modality-Decoupled Asynchrony with Interleaved Coordination), a wait-free system that coordinates local modality progress, peer-to-peer update delivery, and aggregation under heterogeneous compute and bandwidth. Hierarchical asynchrony lets modalities progress independently while enforcing reciprocal feature exchange; load-aware update fetching overlaps training with communication, limits concurrent serving load, and reuses late arrivals; and consensus-aware peer selection uses lightweight metadata to account for update availability, peer participation, and loss similarity without active probing. We implement MODAIC as a containerized multi-agent framework and evaluate it on local and cross-region cloud testbeds. Across AVE and KU-HAR, MODAIC improves average accuracy and multimodal worst-agent performance. On AVE at 200 Mbps per agent, it improves multimodal worst-agent accuracy by 11.0 percentage points at 1000 seconds over the strongest baseline. On the cloud testbed, it reduces time-to-accuracy by up to 30%. Ablations show that simpler load-balancing and delayed-aggregation variants leave a worst-agent accuracy gap, and stress tests show tolerance to WAN variability and partial tracker unavailability.

0.26

Partition-Aware Scheduling for Mobile Heterogeneous Inference Co-Execution

cs.DC cs.LGcs.PF ·2026-09-13 ·Zhuojin Li, Marco Paolieri, Leana Golubchik
published similarity 0.62
Abstract · Accepted for publication in the Performance Evaluation journal (presented at IFIP Performance 2026, in Ghent, Belgium)

Modern mobile inference runs on heterogeneous platforms combining mobile GPUs with multiple CPU core clusters. Existing optimizations typically exploit either inter-operator parallelism, by assigning entire operators to CPU cores or to the GPU, or intra-operator parallelism, by partitioning each operator for CPU-GPU co-execution. We consider these two forms of parallelism together, to improve inference latency of tasks that can be represented by a static DAG of operators with predefined input/output tensor shapes (e.g., CNNs or vision transformers). We define the problem of partition-aware DAG scheduling for mobile heterogeneous inference, illustrating that the best strategy depends on the structure of the inference DAG, thus motivating a joint formulation capturing operator partition choices, device assignment, and execution order. We propose an online iterative search framework, which decomposes large DAGs into stages, focuses search on critical operators, and uses latency predictors to estimate partitioned execution without exhaustive profiling. Across representative mobile inference workloads, our approach achieves latency close to an offline solution while keeping scheduling overhead to a fraction of the model initialization cost, allowing platform-specific scheduling at deployment time.

0.29

Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference

cs.AI ·2026-08-28 ·Xinyuan Gui, Shaowen Wang, Sheng Sun, Zijian Wang, Zishu Yu +1 more
benchmarkcode/data similarity 0.62
Abstract · 9 pages, 7 figures, 2 tables. Code: https://github.com/xinyuangui2/pro-router

The remarkable performance of multimodal large language models (MLLMs) comes at the cost of substantial computational overhead, posing significant challenges to real-time deployment and cost effectiveness. Existing model routing approaches either decide from coarse request-level features alone or spend one or several extra language model passes to inspect the generated response, leaving the token-level uncertainty signals that emerge during generation unused. To address these limitations, we propose Pro-Router, a token-aware progressive model routing method with adaptive edge-cloud collaboration for efficient multimodal LLM inference. Pro-Router employs a two-stage progressive decision mechanism. First, a lightweight prompt pre-scorer module performs rapid pre-screening before token generation begins, guiding apparently simple requests to small models. Second, a token-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model's confidence in its own output to determine, per request, whether the answer ships or escalates to the cloud-based high-precision model. Furthermore, we design an adaptive edge-cloud serving pipeline that sizes every dispatch to each device's measured service rate, so both the edge and the cloud tiers stay fully utilized without manual parameter tuning and are not impacted by the network latency. Extensive experiments on multiple multimodal benchmark datasets and models demonstrate the effectiveness of Pro-Router. Compared to other methods, it achieves the highest routing accuracy and improves routing speed by more than 10x. Its serving pipeline also reaches more than 75% higher end-to-end throughput than the existing model routing pipeline. Our code is available at https://github.com/xinyuangui2/pro-router.

0.43