CS Paper Radar

Dependable Voice Command for Multi-UAV Swarms Using Small Language Models

Afshin Ameri, Luiz Giacomossi, Håkan Forsberg, Ivan Tomasic, Baran Cürüklü

cs.SD cs.HCcs.ROeess.SY ·submitted 2026-11-13 20:26 UTC · v1

Abstract

Supervisory control of multiple autonomous unmanned aerial vehicles (UAVs) imposes high visual and cognitive demands on human operators, particularly when communicating mission objectives and system constraints. Voice command provides a natural interaction modality that can reduce operator workload; however, its reliability is affected by environmental noise, speech variability, and recognition errors. This paper presents a dependable voice command framework for multi-UAV swarm supervision using Small Language Models (SLMs). The proposed architecture integrates an Automatic Speech Recognition (ASR) module with an SLM-based semantic processing layer to transform spoken operator commands into structured task definitions for downstream multi-UAV planning and execution. To evaluate robustness, an end-to-end assessment pipeline is developed using a Probabilistic Context-Free Grammar (PCFG) to generate synthetic supervisory commands, which are converted into audio samples with varying operator accents and environmental signal-to-noise ratios (SNR). The resulting speech signals are processed by the ASR module, and the SLM performs contextual interpretation and phonetic correction of transcriptions when required. The evaluation isolates the resilience of the voice command pipeline under acoustic degradation and speech variation, demonstrating reliable task generation for multi-UAV supervision in challenging operational conditions.

Comment
10 pages, 4 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

AeroWeaver: An Embodied-Agent Harness for Weaving Aerial Skills into Distributed, Adaptive Swarm Execution

cs.AI ·2026-09-16 ·Jiabin Lou, Yirong Yang, Haopeng Wang, Xuxin Lv, Xinyu Liu +4 more
code/data similarity 0.67
Abstract · 8 pages, 7 figures

Collective intelligence is a collaborative autonomy paradigm in which multiple agents pursue shared objectives through local perception, information exchange, and coordinated action. UAV swarms embody this paradigm by coordinating multiple vehicles in tasks such as search, inspection, and tracking. Recent advances in large language model (LLM) agents have strengthened natural-language task understanding and high-level planning, providing a flexible semantic interface between mission descriptions and collective behavior. While these advances expand semantic reasoning, applying LLM agents to UAV swarms raises challenges in grounding model decisions in executable capabilities, reconciling global task reasoning with distributed execution, and using mission-specific experience for continual adaptation. To address these challenges, we introduce AeroWeaver, an embodied-agent harness that weaves individual UAV skills into coordinated mission-level behavior. AeroWeaver connects semantic decisions to governed skills, organizes role-conditioned local agents for distributed coordination, and uses role-indexed state-action-reward experience to refine skill selection online. Experiments and runtime validation show that AeroWeaver maintains valid skill execution under tested conditions and supports body-local multi-UAV operation without a central agent generating joint actions from global context, while reward-guided online updates provide a training-free path for adaptive learning swarm agents from accumulated execution experience. Code: https://github.com/Admire-ljb/AeroWeaver.

0.59

AERIAL: Mission-Aware LLM Selection for Language-Driven UAV Control in Flying Networks

cs.NI ·2026-10-22 ·Diana Nunes, Ricardo Amorim, Pedro Ribeiro, André Coelho, Rui Campos
benchmark similarity 0.65
Abstract · 13 pages, 3 figures

Unmanned Aerial Vehicles (UAVs) in Flying Networks (FNs) increasingly require mechanisms to translate high-level natural language objectives into executable missions. Although Large Language Models (LLMs) have demonstrated the feasibility of language-driven UAV control, existing approaches typically rely on static model configurations and overlook mission-aware performance variability. This paper presents AERIAL, a benchmark-driven framework for adaptive LLM selection in hybrid edge-cloud UAV control. AERIAL separates systematic model benchmarking from mission execution, deriving empirical scenario-model mappings and applying them at runtime for mission-aware selection. Evaluation across 720 automated simulation runs spanning navigation, spatial reasoning, and algorithm invocation revealed significant performance variability depending on scenario-model pairing. A case study integrating AERIAL with a FNs placement algorithm further validated the framework's ability to interface with domain-specific optimization modules through natural language. These results validate the effectiveness of AERIAL's adaptive model selection in improving reliability and operational robustness for autonomous UAV control in FNs.

0.25

AgenticSwarm: Semantic Perception and Adaptive Task Allocation for Heterogeneous Multi-UAV Missions

cs.RO ·2026-09-18 ·Muhammad Ahsan Mustafa, Yasheerah Yaqoot, Faryal Batool, Roohan Ahmed Khan, Valerii Serpiva +1 more
similarity 0.63
Abstract

Multi UAV missions in complex environments require the system to understand both the surrounding scene and the intent of a human operator while maintaining feasible task allocation as mission conditions change. This paper presents AgenticSwarm, an agentic framework for semantic perception and adaptive task allocation in heterogeneous multi UAV missions. An agent interprets aerial imagery and natural language instructions to construct a grounded mission representation that links perceived objects and regions with task requirements, capability constraints, and mission dependencies. This information augments a constrained task allocation process in which obstacle aware path feasibility, energy consumption, and protected return home requirements are incorporated before assignment. During execution, changes such as UAV failure, battery degradation, or task modification trigger residual mission reconstruction from the current system state, while completed work and reconnaissance progress are retained. AgenticSwarm is evaluated across five diverse Gazebo environments and an indoor real test environment, demonstrating its ability to connect semantic reasoning with constrained allocation and adaptive multi UAV mission execution. Compared with a Grounding DINO+SAM~2.1 perception baseline, the SAM3-based pipeline improves class-aware recall by 25.2 percentage points (pp) and semantic label accuracy by 29.5 pp. Ablating residual mission replanning increases mean repeated work from 0% to 61.7% and post-event recovery time by 58.6%, highlighting the contribution of adaptive replanning to mission execution.

0.18

UAV-SwarmAgent: Scalable Multi-UAV Reconnaissance and Task Allocation via Code Generation and Vision-Language Reasoning

cs.RO ·2026-10-23 ·Oleg Sautenkov, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Grigorii Kalinin, Chih-Yung Wen +1 more
similarity 0.62
Abstract

Supervising a multi-UAV reconnaissance mission usually means plotting waypoints and watching every drone. We present UAV SwarmAgent, a hierarchical multi-agent system in which an operator states the mission in one natural-language sentence and a ground-station agent writes and executes code that turns it into per-UAV routes. The agent grounds inspection targets in an aerial map with a vision-language model, splits them among N UAVs by balancing route length, and dispatches worker agents that verify incidents with a compact onboard decision model. In simulation over 4 incident types and 3 foundation models, the swarm located the incident in 244 of 255 missions (95.7%). A paired scaling study with one to 5 UAVs shows that flight time follows an Amdahl-type law. The second UAV delivers half of the attainable saving and the speedup is bounded at 1.52x. We discuss what this predictability, and the false alarms we observed, imply for an operator who supervises the swarm, and outline the user study needed to test it.

0.08

IVG-UAV: An Intelligent Voice-Guided UAV System for Autonomous Ripe Fruit Harvesting with Vision-Based Classification and Adaptive Path Planning

cs.RO ·2026-10-07 ·Dinh Trung Duong
similarity 0.61
Abstract

In tropical regions, their agricultural sectors remain highly dependent on manual labor for fruit harvesting. On large-scale farms, this dependency often results in significant labor cost and logistic complexities. This project presents the development and simulation of a voice-controlled Unmanned Aerial Vehicle (UAV) system designed to automate harvesting tasks in extensive plantations. The proposed system integrates speech recognition using Whisper [1] and LLM, computer vision-based ripeness classification, and adaptive path planning within a unified framework. The entire system is modeled and validated in a Gazebo simulation environment, allowing performance evaluation under controlled agricultural scenarios

0.30

Managing Context and Communication in Distributed Agentic UAV Swarms

cs.MA cs.AIcs.LGcs.NI+1 ·2026-10-01 ·Andrea Iannoli, Ivan Zyrianoff, Angelo Trotta, Lorenzo Gigli, Marco Di Felice
published similarity 0.61
Abstract · 12 pages, 4 figures. This paper has been accepted for presentation at the 24th IEEE Consumer Communications & Networking Conference 2027 (CC…

Unmanned aerial vehicle (UAV) swarms increasingly rely on language-model agents to provide adaptive mission-level reasoning in uncertain environments. Fully distributed control, in which each UAV hosts an independent Small Language Model (SLM), removes reliance on a centralized coordinator but introduces an information-management problem: long-running interaction histories can degrade the reasoning context, while indiscriminate information dissemination increases communication and inference overhead. We address these challenges with a distributed UAV-agent architecture that enables continuous local SLM control through an event-driven reason-act-observe lifecycle. Runtime knowledge is represented as structured atomic notes and organized into core, local, and peer-specific memory. A deterministic interest-aware gossip engine selectively disseminates these notes according to recipient-specific semantic novelty and recency. We evaluate the architecture using ten UAVs in a simulated search-and-rescue mission. Our approach completes all experimental runs, whereas unrestricted flooding messages completes only 70-85\%, and delegating forwarding decisions to the SLM prevents mission completion in every run. Compared with unrestricted flooding, our approach approximately halves inference-token consumption, reduces transmitted data, and achieves lower survivor-count error.

0.25

Vision-Language Models as copilots for Autonomous UAV Navigation: Analysis of Latency and Reliability in Degraded Environments

cs.RO ·2026-08-10 ·Hiago Sodre, Sebastian Barcelona, Vincent Sandin, Pablo Moraes, Ahilen Mazondo +4 more
similarity 0.60
Abstract

The integration of Vision-Language Models (VLMs) in autonomous Unmanned Aerial Vehicles (UAVs) offers unprecedented semantic reasoning capabilities. However, real-time closed-loop navigation requires not only low inference latency but also obedience to structured flight commands. This paper proposes a hybrid FSM-VLM control architecture for UAVs in GPS-free environments. The system combines a deterministic Finite State Machine (FSM) for low-level physical control with an asynchronous VLM copilot for high-level semantic pathfinding. We evaluate three models with different parameter scales in a Software-In-The-Loop (SITL) simulation. The framework isolates and measures syntax errors at the format level versus semantic hallucinations at the logic level in a normal and degraded scenario. This study demonstrates that parameter scaling, and not pure latency, remains the primary bottleneck for the safe and compatible integration of VLM into autonomous flights.

0.04

Commanding at the Speed of Sound: Audio-Centric Referring Expression Segmentation for Aerial Scenes

cs.SD eess.ASeess.IV ·2026-09-28 ·Ziyun Chen, Yijun Wang, Chuanyi Zhang, Liang Yao, Yuexuan An +3 more
benchmarktheorycode/data similarity 0.59
Abstract

Referring expression segmentation (RES) aims to localize targets at the pixel level guided by human instructions. However, current models predominantly follow a text-centric paradigm that treats discrete text as the primary interaction medium for these human instructions. This assumption is especially limiting in unmanned aerial vehicle (UAV) applications, where rapid response, hands-free operation, and natural communication are often critical. In this paper, we reconstruct UAV referring expression segmentation through an audio-centric perspective. Grounded in information theory, we demonstrate that raw audio provides a higher upper bound for extracting rich semantic features than transcribed text. To achieve this, we construct an audio-driven UAV referring expression segmentation benchmark named Aerial-AudioRES by augmenting VisDrone with synthesized speech, forming "audio-text-image-mask" quadruples. Specifically, we employ an ambiguity filtering mechanism to ensure referential uniqueness, and incorporate eight distinct timbres alongside 4 realistic noises to simulate authentic environments. To drive the paradigm shift from text-centric to audio-centric, we propose the end-to-end Audio-centric Referring Expression Segmentation Network (ARES-Net). Distinct from discrete text-visual matching, our approach learns to establish a direct soft alignment between continuous acoustic manifolds and multi-scale visual representations. Comprehensive experiments show that our method obtains a peak accuracy of 64.23% in oIoU and demonstrates robust generalization on a collected real-human test set. Furthermore, it significantly reduces total interaction latency to one-third of the text-based method. Our code is available at https://github.com/LemonGauss/ARES-Net.

0.26