CS Paper Radar

Multi-Agent Reinforcement Learning for Movable Antenna-aided Cell-Free Massive MIMO Systems

Bokai Xu, Jiayi Zhang, Shuaifei Chen, Ziheng Liu, Huahua Xiao, Derrick Wing Kwan Ng, Bo Ai

cs.IT ·submitted 2026-10-06 14:07 UTC · v1
published

Abstract

The inherent non-convex minimum-separation constraints introduced by movable antennas present a formidable challenge to the joint optimization of antenna positions and transmission strategies, rendering conventional methods computationally infeasible, particularly in large-scale cell-free massive multiple-input multiple-output (MIMO). In this work, we propose the graph-based learning individual intrinsic reward heterogeneous-agent proximal policy optimization (GLIIR-HAPPO) algorithm, a novel heterogeneous multi-agent reinforcement learning (MARL) framework that fundamentally overcomes this impasse by systematically decomposing the original coupled optimization into coordinated subproblems. To ensure tractability, we embed the non-convex geometric constraints into a penalty-augmented reward structure and develop a specialized geometric solver that enables the positioning agents to efficiently navigate the high-dimensional action space. Specifically, we propose an architecture featuring a dynamic-interaction graph critic for adaptive cross-role coordination, together with role-conditioned federated distillation that synchronizes same-role policies through compact actor-output statistics. Beyond architectural design, we establish a rigorous theoretical analysis that derives monotonic performance improvement bounds and establishes convergence guarantees for the proposed bi-level optimization. Numerical simulations demonstrate that our framework yields significant sum-rate improvements over state-of-the-art MARL schemes. Moreover, the performance of our advanced architecture closely approaches its fully centralized counterpart, while drastically reducing communication overhead.

Closest papers

Nearest neighbours by meaning across the whole index.

Movable-Antenna Relay Enhanced Multiuser Communications: Reinforcement Learning-Based Antenna Positioning and Power Allocation

eess.SP ·2026-10-14 ·Yuanjun Zhang, Lipeng Zhu, Haobin Mao, Zhe Li, Zhenyu Xiao
similarity 0.67
Abstract · Submitted to an IEEE Journal for possible publication

Movable antenna (MA) has emerged as a promising technology for enhancing wireless communication performance by flexibly reconfiguring wireless channels via antenna movement. In this paper, we propose an MA relay-enhanced multiuser communication system that adopts a half-duplex decode-and-forward (DF) relay scheme. The relay node is equipped with an MA array, and a buffer is employed to enhance the end-to-end transmission performance. Specifically, we formulate an optimization problem to maximize the minimum average achievable rate among multiple users over a given transmission period by dynamically optimizing the positions of MAs, transmit power, and beamforming matrices, subject to the constraints on antenna movement, inter-antenna spacing, and total energy budget for antenna movement and transmission. To address this non-convex problem with highly coupled variables, we propose a reinforcement learning (RL)-based algorithm, where a twin delayed deep deterministic policy gradient (TD3) algorithm is utilized to optimize the antenna positions and power allocation, while the transmit and receive beamforming solutions are obtained through closed-form expressions in each time slot. Numerical results validate the efficacy of the proposed RL-based algorithm and demonstrate the superiority of the proposed MA relay-enhanced multiuser communication system over the conventional fixed-position antenna (FPA)-based relay system and other MA-based benchmarks.

0.23

World Model for Energy Efficiency Maximization in Dynamic Metasurface Antennas

cs.IT eess.SY ·2026-10-26 ·Abdolrasoul Sakhaei Gharagezlou, Mehdi Monemi, Matti Latva-aho, Mehdi Rasti
similarity 0.66
Abstract

Dynamic metasurface antennas (DMAs) have emerged as a promising technology for realizing large-scale antenna arrays in next-generation wireless networks owing to their ability to enable dense and energy-efficient antenna architectures. However, jointly optimizing digital beamforming and DMA element weights under time-varying channel conditions remains challenging due to their coupled effects on the received signal and interference powers. We formulate this problem to maximize time-averaged energy efficiency (EE) under time-varying channel conditions, subject to transmit power constraints, Lorentzian phase limitations, and minimum rate requirements. To enable long-term planning and improve sample efficiency compared with model-free reinforcement learning, we develop a world model framework that learns latent system dynamics from interactions with the environment and jointly optimizes digital beamforming and DMA element weights through imagined trajectories. Furthermore, we incorporate feasibility-preserving action mappings and safety cost signals to ensure constraint satisfaction during policy learning. Simulation results demonstrate that the proposed framework improves the time-averaged EE by approximately 20% compared with the deep Q-network baseline, while achieving higher sample efficiency and more reliable constraint satisfaction.

0.32

GTAC: A Graph-Transformer Actor-Critic for Carrier-Aware MU-MIMO User Grouping

eess.SP ·2026-11-02 ·Yan Huang, Charles Santhosam Lourdu Raja, Rajesh Gadiyar
similarity 0.66
Abstract · Submitted to IEEE ICC 2027

Achieving higher spectral efficiency (SE) in 5G and future 6G networks requires efficient massive multi-user multiple-input multiple-output (MU-MIMO) grouping under coupled physical-layer constraints and strict scheduling deadlines. Motivated by industry efforts in AI-native RAN, we propose Graph-Transformer Actor-Critic (GTAC), a deep reinforcement learning (DRL) architecture for carrier-aware MU-MIMO user grouping. The objective is to maximize the long-term proportional-fair (PF) utility of the cell subject to constraints from the practical stack protocol. GTAC comprises a shared actor--critic encoder with spatial-edge-biased attention mechanism, a pointer decoder actor that generates grouping decisions, and a multi-stage transformer critic that estimates the carrier-wide state-value during training. Training begins by cloning a teacher policy that cannot meet real-time deployment deadlines, followed by critic warmup and two-stage reinforcement learning using proximal policy optimization (PPO), first on a narrow, power-matched carrier and then on the full carrier with a Kullback--Leibler (KL) policy anchor. After two-stage PPO, the best GTAC policy outperforms the non-real-time teacher by 11.6% in carrier SE, while exceeding its PF utility and preserving the cell-edge throughput. A CUDA implementation of GTAC achieves a 69.6 us inference latency on an RTX PRO 4500 GPU, below the targeted 100 us inference budget in practical systems.

0.17

Cooperative Multi-Agent Reinforcement Learning via Epigraph-Form Guided Exploration

cs.AI cs.MA ·2026-11-04 ·Sungil Son, Hoseong Jung, Dahyun Oh, H. Jin Kim
publishedbenchmark similarity 0.62
Abstract · NeurIPS 2026

Discovering cooperative behavior in multi-agent reinforcement learning (MARL) is challenging due to the combinatorial complexity of joint state-action spaces, which hinders the emergence of coordinated behaviors from trial-and-error alone. Intrinsic rewards are often used to aid discovery, but naively combining them with team objectives can distort the learning signal, compromising task performance. In this paper, we propose EFXPLORER, a constrained exploration framework that maximizes exploration objectives subject to a constraint that preserves established task performance. We solve the resulting problem using an epigraph reformulation that introduces adaptive exploration budgets. This approach separates intrinsic rewards from task objectives and regulates exploration through task feasibility. To further encourage diverse and temporally extended exploration, we incorporate a successor distance-based intrinsic reward that captures long-horizon dependencies. Empirically, our method outperforms strong baselines and induces novel cooperative strategies across SMAX, VMAS, and MPE benchmark suites.

0.20

Finite-Sample Convergence in Networked Average Reward MARL: Decentralization Pitfalls and Entropy Remedies

cs.LG cs.MA ·2026-11-02 ·Yizhou Zhang, Yashaswini Murthy, Laixi Shi, Adam Wierman
similarity 0.62
Abstract

We study the problem of average reward Multi-Agent Reinforcement Learning (MARL) where agents interact within a network. Each agent in the network conducts local updates with information within its $k$-hop neighborhood and collaboratively maximizes the overall reward of the entire network. We first provide impossibility results in achieving convergence to the global optimal joint policies in our setting. These impossibility results highlight the challenges of decentralized learning with local information in multi-agent systems, namely the issues of multiagency, where each agent independently optimizes its policy, and partial observability, where agents have limited visibility into the global state. Given these challenges which indicate that decentralized policy optimization can generally be certified only up to stationarity, we provide finite-sample convergence guarantees for a decentralized actor-critic algorithm with linear function approximation that converges to an approximate stationary point with small gradient norm. To overcome such impossibility results, we further show that adding fixed-state entropy regularization reshapes the objective landscape which leads to even stronger global convergence guarantees for our proposed actor-critic algorithm.

0.18

Interference-Graph-Aided Multi-Agent Reinforcement Learning for Beam Hopping LEO Satellite Networks

eess.SP ·2026-10-20 ·Hao Yuan, Ziyi Yang, Xing Zhang
similarity 0.62
Abstract

Beam hopping technology enables LEO satellites to dynamically switch beam coverage positions, providing flexible resource allocation for non-uniformly distributed traffic demands. However, the severe spatial interference coupling among beams makes coordinated scheduling challenging. To address this problem, in this letter, we propose an interference-graph-aided multi-agent reinforcement learning approach that constructs an interference graph to capture beam coupling relationships and injects this structural prior into agent-side representation learning. Each beam agent aggregates information from interference neighbors via graph convolution before estimating local Q-values, which are then decomposed by the standard mixing network. Simulation results demonstrate that the proposed method achieves 34.4\% higher throughput than QMIX with more stable interference management, under non-uniform traffic and co-channel interference settings.

0.14

FARM: Fundamental Agentic Reward Model For Multi-task Wireless Network Optimization

eess.SY ·2026-10-02 ·Feiran You, Changxu Ni, Haozhe Ma, Jun Li, Hongyang Du
similarity 0.62
Abstract

Future wireless networks require learning agents to adapt across heterogeneous channel conditions, traffic patterns, quality-of-service (QoS) requirements, objectives, and operational constraints. Reusing decision knowledge across such tasks is challenging because conventional multi-task and transfer reinforcement learning methods primarily share or transfer policies, coupling transferable knowledge with task-dependent action mappings. This paper proposes FARM (Fundamental Agentic Reward Model for Multi-task Wireless Network Optimization), a reward-space transfer framework that shifts cross-task knowledge reuse from policy space to trajectory-level decision evaluation. FARM introduces an Agentic Reward Model (ARM) that learns a task-conditioned reward prior from heterogeneous source-task trajectories and provides auxiliary guidance for task-specific policy optimization. In Stage I, ARM jointly models task conditions, temporal trajectory dependencies, and objective-dependent reward structures while each source task retains its own controller. In Stage II, the learned reward prior is frozen and reused to guide the adaptation of a target-specific controller for previously unseen tasks, without transferring source-task policies. Experiments on heterogeneous multi-access edge computing (MEC) tasks show that FARM achieves a mean late-stage gain of 29.8% over Single-task SAC on unseen Rate-Latency targets, compared with 16.1% for CRA Transfer, and reaches a 46.2% gain on the moderate-OOD FAR-M case. Further analysis shows that both Mamba and Transformer trajectory encoders support Reward-Space Transfer, while Mamba provides improved robustness as longer history dependencies are introduced.

0.18

Movable-Element Multi-Functional RIS-Assisted Hybrid RSMA-NOMA: A Constrained Hybrid Reinforcement Learning Approach

cs.AI eess.SP ·2026-10-26 ·Li-Hsiang Shen, Chi-Te Kuo
top venue similarity 0.62
Abstract · Accepted by IEEE TCOM

In this work, we propose a movable-element (ME) multi-functional reconfigurable intelligent surface (MF-RIS) architecture, termed MM-RIS, which integrates element mobility with adaptive signal reflection, transmission, amplification, and energy harvesting (EH) to enhance communication flexibility and energy self-sustainability. A hybrid rate-splitting multiple access (RSMA) and non-orthogonal multiple access (NOMA) transmission scheme is further designed to improve spectral efficiency. We formulate an energy efficiency (EE) maximization problem by jointly optimizing transmit beamforming, power and rate allocation, MM-RIS configurations, panel deployment, and ME positions. To address the resulting non-convex mixed continuous-discrete optimization problem, we propose constrained hybrid deep reinforcement learning with federated transformer and parametrized sharing (CRAFTS). CRAFTS integrates multi-agent proximal policy optimization (MAPPO) and multi-agent deep Q-networks (MADQN) through bi-directional parametrized sharing to handle continuous and discrete actions, respectively. Moreover, a transformer-based architecture and federated learning (FL) are incorporated to enhance state representation and distributed knowledge sharing, whilst a Lagrangian-based adaptive penalty mechanism facilitates the constraint satisfaction. Simulation results have demonstrated that the proposed MM-RIS architecture and CRAFTS framework substantially improve EE and constraint satisfaction compared to the benchmarks in the open literature. Moreover, the results validate the benefits of spatially reconfigurable MM-RIS, hybrid RSMA-NOMA transmissions, and intelligent collaborative learning, highlighting the potential of the proposed framework for energy-efficient and adaptive next-generation wireless networks.

0.28