CS Paper Radar

Challenges and Contributions in Quality of AI-Based Software: A Systematic Mapping Study

Maryum Hamdani, Mateen Ahmed Abbasi, Marko Jäntti, Markku Tukiainen

cs.SE ·submitted 2026-08-26 09:08 UTC · v1
published

Abstract

Artificial Intelligence (AI) is increasingly embedded in modern software systems, raising important questions about how its quality should be defined, assessed, and assured. This paper presents a Systematic Mapping Study (SMS) on the quality of AI-based software. The study synthesizes primary studies published between January 2020 and January 2026 and selected from five electronic data sources. A total of 33 primary studies were included after automated search, screening, and snowballing. The results identify six recurring challenge categories, with the most prominent being limitations in existing quality assessment models, followed by issues in non-functional requirement management, quality-aware development, and quality assurance. The findings suggest a call for collaboration of researchers and industrial practitioners with standardization organizations, that could possibly devise comprehensive quality assessments and their measurement methods.

Comment
Accepted in SEAA 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Empirical Insights from AI-Powered Software Engineering in Practice

cs.SE ·2026-10-16 ·Tina Mersch, Jack Johns, Steffen Klepke, Hanya Elhashemy, Artur Signell +2 more
similarity 0.64
Abstract · Submitted to IEEE Software

Generative Artificial Intelligence (GenAI) is becoming indispensable in future software engineering. Its application spans a wide spectrum - from assisted development to autonomous software engineering, requiring organizations to address associated risks and challenges. This paper investigates these issues through three case studies, highlighting practical problems encountered during GenAI integration. Based on these findings, it provides targeted recommendations for managing risks and ensuring reliable outcomes. This paper provides guidance for effective and responsible GenAI adoption in software engineering based on empirical insights.

0.11

Systematic Mapping Protocol. Generative AI-Augmented Requirements Engineering: A Systematic Mapping Study of Practices, Specifications, and Reuse in Software Development

cs.SE cs.AI ·2026-11-17 ·Fernando Pinciroli
similarity 0.63
Abstract

Generative artificial intelligence (GenAI), particularly large language models (LLMs), is rapidly being adopted in requirements engineering (RE). Existing secondary studies have characterized which RE activities are supported and which prompting strategies are used, but little is known about the specifications that GenAI-augmented RE produces and how they are used in subsequent activities such as specification-driven development (SDD), design and code generation, test derivation, verification, and reuse. This document presents the protocol of a systematic mapping study (SMS) that will identify and classify research on GenAI-augmented RE, with particular attention to the work products it generates and their downstream use. The protocol defines the research questions and their classification schemes, the search strategy, the study selection procedure, the data extraction form, the analysis approach, the tool support, and the threats to validity. It is published in advance to establish a prior commitment and to support the transparency and repeatability of the study.

0.18

Agent Trajectories as Evidence for Software Quality Assurance

cs.SE cs.AI ·2026-11-17 ·Jonathan Ullrich, Julien Siebert, Michael Kläs, Adam Trendowicz, Andreas Jedlitschka
similarity 0.63
Abstract · 3 pages, submitted to ICSE2027 AMPLIFY Workshop https://conf.researchr.org/home/icse-2027/amplify-2027#co-create-software-engineering-resear…

Autonomous software agents increasingly plan, implement, and test software with limited human oversight. Their decisions may introduce implicit assumptions that final code and passing tests do not adequately expose. We propose analyzing observable agent trajectories alongside software and agent specifications and AI generated artifacts as evidence for targeted quality assurance. The aim is to prioritize consequential decisions for scrutiny, investigate their possible contribution to software problems, and inform improvements to specifications and development practices. Industrial experience motivates the idea; available execution records and preliminary analysis methods make it plausible. However, trajectories alone do not establish that decisions are justified, outcomes acceptable, or corrective changes effective. We ask how this evidence can support review, diagnosis, and continuous quality improvement, and when simpler QA approaches suffice. Perspectives from requirements engineering, verification and validation, agent systems, causal inference, empirical software engineering, and human factors could help develop and evaluate the idea.

0.16

Rethinking Data Quality for AI-Driven Systems: Evidence from Practitioner Interviews

cs.SE cs.AI ·2026-09-25 ·Hariharan Gopinath, Jan Bosch, Helena Holmström Olsson
similarity 0.60
Abstract · This is a preprint version and the final version will appear in the proceedings of PROFES 2026

Data quality research has usually treated data as an input that is stored, processed, and validated. In AI-driven software-intensive systems, data also shapes model behavior, evaluation, and lawful use. Empirical evidence remains limited on how practitioners define, assess, and manage quality under these conditions. We interviewed 16 practitioners from nine organizations and analyzed the transcripts using reflexive thematic analysis and developed six themes from participants' accounts. In AI systems, traceability shifted from modular debugging to attributing model behavior, while using models as quality assessors introduced circularity. Agent context and memory became data objects, and synthetic and pseudo-labeled data made authenticity a quality concern. In foundation-model development, lawfulness became a gate for training data, while representativeness was judged through coverage of situations in which the system must behave safely. Prior ML research examines many of these problems separately. Our study provides a practitioner-grounded account of how they are encountered together as an engineering and organizational concern. We also interpret five recurring conditions as helping explain how the themes relate to reduced trust in data and AI outcomes. We synthesize these findings through lifecycle assurance: a conceptual framing focused on producing evidence that data can support a specific AI claim when its influence may be embedded in model behavior, model-based judgments, or agent actions.

0.28

Does AI-Assisted Development Homogenize Software? An Empirical Study of Code Similarity, Quality, and Evolution

cs.SE cs.AI ·2026-10-16 ·Luciano Marchezan, Romain Lefeuvre, Kevin Delcourt, Benoit Combemale, Houari Sahraoui
code/data similarity 0.59
Abstract

AI-assisted software development is evolving from interactive LLM assistance toward agentic systems that autonomously perform software engineering tasks, yet the effects of this shift remain insufficiently understood. We present an empirical study of 50 long-lived Java and Python open-source projects across three eras: pre-LLM, potentially LLM-assisted, and detected agentic development. Using snapshots from Software Heritage, we examine code similarity, software quality, and software evolution. Similarity is measured semantically through embedding-based cosine similarity and syntactically through CodeBLEU*. Quality is assessed with SonarQube metrics covering maintainability, complexity, reliability, and security. Evolution is characterized by annualized growth rates of LOC, functions, classes, files, and statements, and by commit activity in the three months preceding each snapshot. Results show no consistent evidence of increased similarity as semantic and syntactic similarity decreased significantly in Java for both the LLM and agentic eras relative to the pre-LLM era, while Python showed no significant differences. Software quality remained largely stable, although the agentic era showed significantly lower code-smell and SQALE index densities. Structural growth was generally lower in the agentic era, particularly for files, whereas development activity increased moderately in Java, especially in commits and added lines. These findings challenge the assumption that LLM- and agent-assisted development necessarily homogenizes code, degrades quality, or accelerates growth.

0.27

ARISMA: Guidelines for AI- and LLM-Assisted Systematic Reviews, Scoping Reviews, and Mapping Studies

cs.SE cs.AI ·2026-08-25 ·Mahyar Tourchi Moghaddam, Mina Alipour
published similarity 0.59
Abstract · Accepted to be presented at AGENTICS 2026, to be published by Springer

Systematic reviews, scoping reviews, mapping studies, and related evidence syntheses are increasingly difficult to conduct with fully manual workflows as search volumes, update cycles, and synthesis requirements continue to expand. At the same time, artificial intelligence, machine learning, and large language models are rapidly entering review practice across query formulation, screening, extraction, categorization, appraisal support, and reporting. Yet the empirical evidence remains uneven, task-dependent, and insufficient to justify unconstrained automation. Existing standards such as PRISMA 2020, PRISMA-S, PRISMA-ScR, PRISMA-P, PRESS, and SWiM remain essential, but none provides an end-to-end operational standard for when AI use is methodologically appropriate, how it should be validated, which review decisions must remain human-led, and how AI involvement should be reported so that readers can audit it. This paper proposes ARISMA, an AI Reporting and Integration standard for Systematic Methods and Analysis. ARISMA treats AI as an inspected, benchmarked, logged, and reversible assistant rather than an autonomous reviewer. It is built around one governing principle: every consequential scientific decision must remain human-interpretable, human-auditable, and human-accountable. The paper contributes a lifecycle taxonomy, process guidance, stepwise recommendations across the review pipeline, a governance and provenance model, a tool-support framework, an AI-integrated reporting checklist, and a validation matrix. It also addresses legal, privacy, infrastructure, and sustainability considerations. The framework was iteratively refined through structured expert consultation. The result is a practical and auditable guideline for responsible AI-assisted evidence synthesis.

0.56

Advancements in Cloud-Native Systems: A Systematic Mapping Study of Reliability and Scalability across Cloud-Native, Edge-Native and AI-Enabled Systems

cs.SE cs.AIcs.MAcs.NI+1 ·2026-11-08 ·Shrinivass Arunachalam Balasubramanian
similarity 0.58
Abstract

Cloud-native systems now extend to the network edge and host artificial-intelligence (AI) workloads, including AI agents, but evidence on their reliability and scalability is scattered and uneven. We report a PRISMA 2020 systematic mapping study with an evidence-graded synthesis. Five logged OpenAlex searches on 11 October 2026 covered cloud-native systems, edge AI, agentic AI for operations, reliability of agentic applications and physical AI; 189 records from an earlier exploratory search were added separately. After 301 duplicate removals and an automated peer-review venue rule, 4,611 records were screened against written criteria and a five-level relevance scale. The review includes 1,794 primary studies from 2017-2026 and 95 secondary studies, with every decision released in Dataset S1. Each primary study was assigned to a 13-category taxonomy and graded as simulation, prototype or testbed, production data, production deployment, or not determinable from the abstract. Of 1,003 studies with a determinable level, 66.2% used prototypes or testbeds, 18.0% simulations and 15.8% production data or deployments. A 93-study synthesis core, appraised from indexed abstracts, shows production deployments of cascading-failure prevention, trace- and graph-based root-cause analysis and predictive autoscaling, and production-trace evaluations of cold-start mitigation. Edge reliability and physical AI rely mainly on simulations and testbeds. AI-agent reliability uses tool-grounded diagnosis, explicit safety properties, trace-driven repair and workload-aware scheduling, while empirical studies show that agent frameworks remain fragile. Conclusions are limited to the documented corpus. The principal gaps are production evidence for edge-native systems, shared resilience benchmarks and dependable agentic operations.

0.28

Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Development

cs.SE cs.AI ·2026-09-25 ·Viktor Kjellberg, Srijita Basu, Simin Sun, Farnaz Fotrousi, Miroslaw Staron
similarity 0.58
Abstract

The emergence of AI agents is expected to reshape software engineering by moving beyond AI as assistants towards systems capable of planning, executing, and evaluating development tasks with increasing autonomy. This transition is particularly significant for embedded software organizations, where strict requirements for quality, traceability, verification, and long-term maintainability often apply. This paper presents a case study of a large embedded systems company and its transition toward becoming an AI-first organization. Through a mixed method, we analyzed data collected from a semi-structured workshop with 40 participants, including scrum masters, architects, management, and product owners. The findings show that the participants expect agentic AI to affect team structure, required competencies, organizational strategies, and developers' roles within the organization. Based on these findings, the paper discusses implications for federated AI team formation, human-in-the-loop practices in such an organization, and the sustainable adoption of AI agents in embedded software engineering. We also present a concrete roadmap for the organization towards becoming an AI-first organization.

0.17