CS Paper Radar

Model Discovery for Virtual Chaos Engineering with Incomplete Tracing: An eBPF Study

Anatoly A. Krasnovsky

cs.SE cs.DC ·submitted 2026-11-13 05:42 UTC · v1

Abstract

Model-based fault experiments can assess the consequences of component failures before intervening in a running application. Recent work on model discovery and graph simulation acquired operation models from application telemetry and used them for availability prediction. Applying this approach to operational legacy systems raises an acquisition problem: the required traces may be absent or incomplete, while introducing them can require changes across heterogeneous components. These systems continue to serve business operations during extended maintenance and modernization. External observation offers a possible route to model-based fault analysis before a tracing retrofit, but collecting telemetry does not by itself establish that an executable operation model can be obtained. We investigate this route using extended Berkeley Packet Filter (eBPF) observations and the existing graph-based modelling approach. A 108-case study varied application software development kit (SDK) tracing across absent, partial and full conditions in three public applications. Discovered dependencies were combined with declared operation requirements, deployment facts and scenario assumptions; model forecasts preceded independent execution checks. With SDK tracing disabled, the tested OpenTelemetry eBPF Instrumentation path supported all seven operations in two applications across three repetitions. Forecasts agreed exactly in 17 of 21 operation-repetition groups; the maximum state-weighted discrepancy was 4.22 percentage points. Combining observed context links supported nine additional groups under partial tracing. The results establish an acquisition path for virtual fault experiments under the tested observation constraints and identify limits imposed by protocols, evidence association and operation semantics.

Comment
Submitted to Software and Systems Modeling journal

Closest papers

Nearest neighbours by meaning across the whole index.

eBPF Security in the Wild: Structural Concentration, Failure Mechanisms, and Discovery Gaps

cs.CR ·2026-08-27 ·Baihong Chen, Hua Ming, Weifeng Pan, Tian Xie, Xiaojun Qi +1 more
dataset similarity 0.59
Abstract

Extended Berkeley Packet Filter (eBPF) is a security-critical in-kernel execution framework, yet its vulnerability landscape remains fragmented across components, semantic gaps, and testing techniques. We present an empirical study of observed eBPF vulnerabilities. We construct a multi-source dataset from Linux kernel fixing commits, syzbot reports, and public CVE/NVD records, and analyze it through a unified framework covering structural concentration, mechanism-level failure modes, architectural distribution, and discovery gaps in representative techniques. Our results show that the observed eBPF vulnerability landscape is structurally concentrated rather than broadly dispersed across many unrelated weakness types. The dominant portion is associated with a limited set of recurring system-level failures, especially in runtime execution, concurrency, object lifecycle management, and semantic inconsistencies across trusted stages. These failures are unevenly distributed across the eBPF pipeline: Runtime is the dominant exposure surface, whereas the Verifier and JIT are lowerfrequency but structurally distinct security boundaries. A rubric-based comparison of representative techniques and a version-aligned Syzkaller case study on Linux v5.10 show that, despite visible raw coverage of Runtime, Verifier, and JIT, effective exploration is semantically narrow, and observed discoveries concentrate in a small subset of Runtime failures. Overall, raw coverage alone provides an incomplete view of discovery effectiveness.

0.24

Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering

cs.SE ·2026-08-25 ·Zahra Seyedghorban, Egor Klimov, Arie van Deursen, Annibale Panichella, Burcu Kulahcioglu Ozkan
publishedhighly cited similarity 0.57
Abstract

Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-agent systems in software engineering workflows. The tool instruments agent executions with trace-aligned telemetry across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-agent system for software development.

0.53

Trace-Based Execution-Level Observability of VDM-SL Specifications

cs.SE ·2026-08-20 ·Tomohiro Oda, Han-Myung Chang
theory similarity 0.55
Abstract · the 24th Overture Workshop

VDM has been pursuing rigorous verification through mathematical theorem proving and software testing via simulated execution. Animation through an interpreter enables validation of the specification to ensure it meets the required functionality. Step-by-step execution in a debugger also allows the user to follow the internal behavior of operations. In this paper, we propose the recording and utilization of execution traces of assignments, operation calls, and return statements to make the internal behavior of operations persistent and analyzable as state-based models. The data model of events in execution traces, its implementation in ViennaTalk, and its application to visualization will be introduced.

0.17

Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents

cs.CR cs.AI ·2026-08-20 ·Wei Shao, Chongzhou Fang, Zuxiong Tan, Zequan Liang, Setareh Rafatirad +2 more
similarity 0.55
Abstract

Long-horizon security LLM agents must carry information and decisions across many dependent interactions, where later actions often depend on services, state, or access discovered much earlier. This makes final task success difficult to interpret: an agent may fail before it ever reaches the point where the capability of interest can be exercised. We present a diagnostic methodology that instruments security tasks with checkpoints, separates failures before and after capability exposure, and uses controlled interventions to test suspected upstream bottlenecks. We evaluate the methodology across four task families involving delayed reuse of discovered information, reuse of observed state, recovery from failed strategies, and decision making after uncertain outcomes. On observed state reuse, checkpoint analysis shows that many Gemini 2.5 Flash failures occur before the model observes the state it is later expected to reuse. In a pre-specified 92-seed study, targeted protocol-disambiguation guidance increases state observation from 65.5\% under a matched non-guidance control message to 95.4\%. Repeating the same design with Gemini 3.7 Flash produces the opposite effect, while state observation no longer reliably predicts task completion. These results show that the dominant source of failure can shift across model generations, motivating evaluation that diagnoses where and why long-horizon security agents fail rather than relying only on aggregate task success.

0.30

Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures

cs.MA cs.AI ·2026-10-07 ·Xinwen Liu, Zhuocheng Pan, Isabella Zhu, Jawei Zhang, Xudong Liu +1 more
similarity 0.55
Abstract

Multi-agent LLM systems coordinate task execution through exchanges of information among agents. When coordination breaks down, similar symptoms in execution traces can reflect different problems in how information is passed, used, or verified. Communication topology captures how agents exchange information and provides structural cues for distinguishing coordination failure modes. Using these cues for diagnosis requires establishing how topology relates to failure patterns and recovering the relevant structure from execution traces that lack explicit topology labels. We analyze the relationship between communication topology and failure patterns and introduce MAScope, a two-stage framework for topology-conditioned diagnosis. Its Trace Structural Extractor TSE recovers communication topology from heterogeneous execution traces by grounding an interaction graph in message evidence. The Topology-Conditioned Judge TC-Judge then classifies failures using the trace, predicted topology, an empirical failure prior estimated from separate labeled traces, and a short description of topology-specific failure patterns. Under a fixed orchestration structure, the recovered topology can be reused across executions. Experimental results show a statistically significant association between communication topology and failure type, with $χ^2 = 409.9$ and $p = 1.2 \times 10^{-70}$. On the \num{851} MAST-clean traces, ground-truth topology context raises gpt-mini's Macro-F1 from $0.173$ to $0.350$. With predicted topology, the pipeline achieves $0.346$, approaching the trace-only gpt-5.4 baseline of $0.372$. For \num{1000} traces under a fixed orchestration structure, the projected pipeline cost, including one topology extraction, is approximately $6\%$ of repeated gpt-5.4 diagnosis cost. These results show that topology-conditioned context improves failure diagnosis and supports lower-cost deployment.

0.29

Mutation Testing of Simulink Cyber-Physical System Models: Challenges and Solutions in Practice

cs.SE ·2026-08-25 ·Murat Kavak, Onur Kilincceker, Serge Demeyer, Kevin Vandenbroucke, Halim Abdurrahman Ceylan
top venue similarity 0.54
Abstract · Accepted for publication at the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026), Munich, Germany, Octobe…

Several experience reports illustrate that mutation testing is capable of supporting a shift-left testing strategy, especially in industries where late bug discovery incurs very high costs and risks. In the context of cyber-physical systems, a shift left implies that engineers need to test the design models used to simulate, prototype, and analyse the feasibility of the system under design. In this paper, we analyse the challenges we encountered and the lessons we learned when incorporating mutation testing in the context of DUCO, a company producing ventilation systems for buildings. The engineers within DUCO have years of experience with model-based engineering centered around Simulink and StateFlow, including unit tests for their models. During a pilot project with our tool prototype MUT4SLX we learned that equivalent mutants, requirement traceability, and mutation testing for Stateflow represent particular challenges not yet reported in the academic literature.

0.33

Bug2Scenario: Root-Cause-Preserving Test Scenario Amplification for Autonomous Driving Software

cs.SE ·2026-11-04 ·Yuchen Cao, Zhenyu Mao, Jacky Wai Keung, Yifei Wang, Yihan Liao +1 more
publishedcode/data similarity 0.54
Abstract · 11 pages, 3 figures, and 2 tables. Accepted at APSEC 2026. Yuchen Cao and Zhenyu Mao contributed equally. Code and artifacts: https://github…

Simulation-based fuzzing exposes autonomous driving system (ADS) failures by repeatedly mutating and executing scenarios, but converting one failure into reusable regression scenarios remains difficult. Such variants test whether the mechanism persists after software changes, for example by varying weather or actor placement while retaining a collision's actor-route conflict. We present Bug2Scenario, a root-cause-guided post-fuzzing framework. From scenario data, traces, and oracle evidence, it extracts a root-cause pattern. An LLM proposes semantic variations over actors, missions, routes, and weather. Deterministic rules construct DriveFuzz-compatible scenarios. Execution evidence and manual review assess preservation. Separating these responsibilities prevents fluency, syntax, or recurring labels from masquerading as causal evidence. Under matched seeds, six LLM backbones, candidate budget, and execution pipeline, Bug2Scenario improves strict preservation by 6.7 percentage points (p=0.104; not significant) and boundary-inclusive preservation by 10.6 points (p=0.011). Feedback helps some initially unsuccessful seeds yield a strictly preserved candidate. Seed-level stopping then avoids further API calls and simulator runs. Remaining failures reflect actor-placement constraints, simulator instability, and incomplete or conflicting causal evidence. Bug2Scenario thus produces reliable ADS regression assets. Our implementation is available at https://github.com/YgcCoder/Bug2Sce.

0.37

SoK: From Finding to Deployment: Systematizing the OS Kernel Bug Lifecycle

cs.CR cs.OScs.SE ·2026-09-19 ·Luyao Bai, Gengda She, Kenan Alghythee, Hang Zhang, Xiaoguang Wang
survey similarity 0.54
Abstract

Automated kernel bug discovery has advanced rapidly. Continuous fuzzing and static analysis systems, such as syzbot, now expose Linux kernel bugs at a scale that downstream processes struggle to absorb. Yet a crash report is only the beginning. Before a bug is eliminated, it must be triaged, understood, patched, validated, reviewed, integrated, and often backported. These later stages remain far less automated, creating a persistent gap between bug discovery and patch deployment. This SoK systematizes the Linux kernel bug lifecycle from discovery to deployment. We organize prior work and production systems into five stages: discovery, triage, patch generation, patch validation, and integration. We explain the resulting automation gradient through kernel-specific challenges such as concurrency, implicit invariants, cross-syscall state, hardware dependence, lack of fault isolation, and architecture/configuration multiplicity. We further ground the analysis in a measurement of real syzbot-fixed bugs. The data shows that the crash-to-patch gap is not merely a backlog of unfixed reports but a structural failure mode of the repair pipeline: even after being fixed, bugs often remain open for weeks, require review-driven patch revisions, or lack reproducers that current repair and validation systems assume. This exposes a mismatch between where kernel-security automation is mature and where bug closure actually breaks down. These findings expose a deeper mismatch: today's repair and validation techniques often assume reliable reproducers, localized root causes, and checkable correctness oracles, yet these are precisely the artifacts missing from many real kernel bug reports. Closing the crash-to-patch gap, therefore, requires treating such artifacts as outputs to be produced, not prerequisites to be assumed.

0.20