CS Paper Radar

Bias Smells in AI Software Development: Recognizing Potential Sources of Fairness Debt

Ronnie de Souza Santos, Cleyton Magalhaes, Rodrigo Spinola

cs.SE ·submitted 2026-08-10 21:31 UTC · v1

Abstract

Context: Fairness debt arises from AI development shortcomings that may lead to societal harms. While technical and social debt concern software design decisions and team dynamics, respectively, fairness debt captures the long-term consequences of development decisions that may reinforce bias and inequities in AI-based software systems. Despite growing attention to AI fairness, limited evidence exists on how practitioners recognize potential sources of fairness debt during development. Aim: This study investigates the indicators practitioners recognize as signaling potential sources of fairness debt in AI-based software projects. Method: We conducted an exploratory case study of four AI projects within one organization. Data were collected from 25 professionals through semi-structured interviews and open-ended questionnaires, complemented by observation of internal communication channels and project documentation, and analyzed using iterative qualitative coding, memoing, and constant comparison. Results: We identified six recurring indicators, termed bias smells: Context Oversimplification, Dataset Imbalance, Metrics Inadequacies, Ad hoc Testing, Individual Diversity Unawareness, and Homogeneous Team Composition. These smells span technical and human aspects of software development and signal conditions that may introduce or reinforce bias and contribute to fairness debt. Conclusion: Bias smells extend the software smell paradigm to fairness and provide a foundation for incorporating fairness into software quality assurance through observable indicators.

Closest papers

Nearest neighbours by meaning across the whole index.

Technical Debt Risks in AI-Assisted Software Development: Insights from Practitioners

cs.SE ·2026-11-11 ·Nazia Jahan, Nafisa Alam, Sourov Debnath, Arafat Rahman, S. M. Waesh Ahmed +2 more
similarity 0.73
Abstract

AI coding assistants increasingly support software development, raising concerns about technical debt associated with AI-generated code and development practices. We investigate these concerns through a sequential mixed-methods study involving 137 survey respondents and 20 follow-up interviews with software professionals in Bangladesh. Drawing on ten established technical debt dimensions, we examine perceived risks, contributing circumstances, maintenance consequences, validation practices, and developer understanding. Respondents expressed the strongest concerns about People Debt (73.0\%), Management Debt (70.1\%), and Architecture Debt (68.6\%), suggesting that risks extend beyond code quality. Participants associated ambiguous requirements, insufficient project context, and unrequested AI-generated changes with architectural inconsistencies, rework, and maintenance difficulties. Their experiences revealed contrasting outcomes: AI assistance supported refactoring, maintainability improvements, and testing, but also introduced rework and context misalignment. Although 89.8\% reported regularly verifying AI-generated code, participants expressed concerns about AI dependence, reduced code understanding, and review effort. They also emphasized the need for stronger architectural awareness, requirements clarification, and validation support in coding assistants. Our findings suggest that managing technical debt risks requires attention to project context, implementation decisions, and developer understanding alongside generated code quality.

0.11

Cognitive Debt in the GenAI Era - Listening to Practitioners' Voices on the Loss of Understanding

cs.SE ·2026-10-27 ·Theocharis Tavantzis, Bianca Trinkenreich, Michaela Greiler, Júlia Condé Araújo, Marina Condé Araújo +4 more
similarity 0.66
Abstract · 12 pages, 3 figures. Manuscript under review at ICSE-SEIP

Fast-paced AI-assisted software development risks reducing developers' understanding of the systems they build. This growing gap between what the system does and what developers understand about it, described as ``cognitive debt'', is not new, but AI has exacerbated it. This has led to a recent surge in blog posts and online discussions, in which practitioners recognize and write about cognitive debt, and share their experiences well before academic studies can capture them. To understand their concerns and inform future research and practice, we conduct a grey literature review of 76 sources, focusing on how cognitive debt is conceptualized and manifests, its contributing factors, its consequences, and the practices to prevent, detect, and mitigate it. We find that cognitive debt can accumulate unnoticed behind working code and passing tests, particularly when developers accept AI-generated solutions without building understanding. Practitioners report consequences such as harder maintenance and debugging, fragmented knowledge in the team, and reduced productivity. They propose practices such as documenting design rationale, review-sized changes, comprehension gates, and knowledge-sharing rituals, along with signals like reliability metrics to detect it. Our findings give software teams a structured view of emerging practical approaches in AI-assisted development and stress the need to preserve human understanding alongside faster code production.

0.55

Building AI-Intensive Software with AI: Early Results and a Cautionary Tale on Measuring Development Cost

cs.SE cs.AIcs.LG ·2026-08-13 ·Victor Barros de Miranda Neves, Kiev Santos da Gama, Vinicius Cardoso Garcia
published similarity 0.65
Abstract · 4 pages, 2 figures, Accepted for publication at the International Workshop on Intelligent Software Engineering (ISE 2026) @ CBSoft 2026

Empirical reports on the true cost of AI-intensive software development remain scarce, and the few that exist are easy to get wrong in ways that never surface in the final number. We report early results from an ongoing case study: a six-person student team built a full conversational onboarding assistant -- RAG-based code chat, guided tours, dependency graphs, technical-debt analysis -- over one academic term using pervasive AI assistance. We instrumented development with a three-layer cost model (real AI spend, self-reported human effort, human counterfactual) and initially reported a 19.4x cost ratio. A follow-up pass revealed two independent errors -- inferring per-token cost under a flat-rate subscription, and pricing the counterfactual with the wrong regional labor rates -- that together had inflated the ratio by roughly 2x; the corrected figure is ~9.9x. We present this correction as an early, generalizable finding in its own right: both errors are easy to make, invisible in the final number, and plausibly common in similar reports. We outline next steps toward a more robust, replicable costing methodology for AI-intensive development.

0.23

Fairness Hazard Analysis for Socio-Technical Processes: A Multiple-Case Study in Bias-sensitive Organisational Settings

cs.SE ·2026-08-24 ·Giovanna Broccia, Lucio Lelii, Roberto Cirillo, Dario Di Nucci, Samuel Fricker +3 more
similarity 0.63
Abstract

Fairness is increasingly recognised as a first-class requirement in socio-technical processes, where interactions among human actors, software systems, and AI technologies may lead to unfair outcomes in decision-making workflows. If left unaddressed, fairness hazards may accumulate and reinforce systemic bias, highlighting the need to engineer fairness proactively. Despite growing interest in fairness-aware systems, systematic methods for identifying fairness hazards in socio-technical processes and deriving requirements-level mitigations remain limited. To support fairness-by-design during requirements engineering (RE), Fairness Hazard Analysis (FHA) is introduced as a methodology for systematically identifying, analysing, and mitigating fairness hazards. FHA is first assessed through a proof-of-concept validation conducted via two focus groups. Then, a qualitative multiple-case study involving two organisations examines its applicability in real-world settings. The proof-of-concept validation highlighted the benefits derived from the structured nature of the method, and suggested the need to include iterative, dialogic reflection with domain experts. In the multiple case-study where FHA was applied, the practitioners involved were positively impressed by the results and confirmed the relevance of the identified fairness hazards (spanning up to 27% of the process elements), as well as the appropriateness of most of the proposed mitigations, while noting that contextual factors might hinder their implementation. The evaluation also highlighted mitigation patterns, such as independent review and collective decision-making, which can be transferred to different organisations. This paper contributes a structured and empirically validated methodology for integrating fairness considerations in RE and preventing systemic bias in socio-technical processes.

0.19

Specification-Driven Governance for Fairness-by-Design in Agentic Software Engineering: How Far Can We Go?

cs.SE ·2026-11-02 ·Alfonso Cannavale, Alessandra Parziale, Valeria Pontillo, Patrizio Pelliccione, Andrea De Lucia +1 more
similarity 0.63
Abstract

Agentic AI is shifting developers' responsibilities from software implementation to software governance. While this paradigm improves productivity, it also raises challenges in ensuring that autonomous coding agents preserve developers' intentions. Among these, operationalizing fairness-by-design is largely unexplored, as existing approaches provide limited support for representing and propagating fairness requirements throughout agentic development workflows. In this paper, we investigate whether specification-driven development can operationalize fairness-by-design in the agentic development of machine learning systems. We study three coding agents on four machine learning applications, using four prompts that progressively operationalize fairness through nine Fairness Specification Elements, and analyze the traceability of fairness requirements across the generated artifacts, the fairness-accuracy trade-off of the resulting models, and the agents' design decisions and rationales. The results show that when fairness is operationalized, agents capture almost all fairness requirements in the specification (97.22%) and preserve most of them in the code (82.41%). However, the resulting models are not significantly fairer, as agents implement fairness as a post-hoc audit that neither shapes training nor prevents model delivery. Specifications thus make fairness traceable, but not enforceable, calling for fairness specifications that state what agents should do when requirements are not met, and for workflows that make the fairness decisions agents delegate to developers explicit.

0.23

Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning

cs.SE cs.CY ·2026-09-24 ·Qusay H. Mahmoud
similarity 0.62
Abstract · 22 pages, 1 figure, 3 tables

Generative artificial intelligence has moved from a disruptive novelty to a recurring part of software-development and computing-education workflows, while software agents are beginning to act across repositories, command lines, browsers, tests, and other tools. The educational problem is no longer whether students should be allowed to generate code, but whether software-engineering (SE) programs can preserve and assess human understanding while preparing students to work responsibly with increasingly capable AI systems. This paper presents a structured integrative review of research and practice from 2023 through 23 September 2026, supplemented by established work on AI literacy, technical debt, and human-AI collaboration. The evidence supports a conditional conclusion: GenAI can improve access to explanations, feedback, practice, and short-term task completion, but learning outcomes depend on prior knowledge, scaffolding, verification, task design, and assessment. We examine student help-seeking and authorship, faculty assessment and policy demands, and the wider SE risks created when generated artifacts persist across repositories, teams, architectures, and deployed systems. We extend the concept of comprehension debt: the deferred learning and maintenance cost that arises when AI-assisted production outpaces a learner's or team's ability to explain, test, modify, and justify the resulting software. We then refine the AI-Augmented Software Engineering Education (AASEE) framework into five non-linear integration levels and four cross-cutting evidence obligations: explain, verify, modify, and account. The framework links delegation to evidence, governance, and recovery mechanisms appropriate to its consequences.

0.33

Applications of Risk Science to AI Fairness Evaluation: Principles, Challenges, and Best Practices

cs.CY cs.AI ·2026-08-30 ·Kyra Wilson, Sabrina Kang, Saloni Dash, Aylin Caliskan
publishedhot direction similarity 0.61
Abstract · Accepted at AIES 2026

Scholarly work which aims to describe potential societal impacts (e.g., risks) of proliferating technology (especially related to artificial intelligence or other algorithmic systems) is likely to have an impact beyond the scientific communities it was written for, given that general society itself is a primary object of study. However, it is an open question whether the current practices of AI evaluation scholarship follow the principles and best practices established by risk science, which aims to systematically generate knowledge related to understanding, assessing, communicating, managing, and governing risk. In this work, we examine this in depth by conducting a literature review of scholarly works purporting to evaluate the bias or fairness of technological systems used for tasks related to hiring and employment. Through analysis of 22 common fairness evaluation metrics and studies using them, we find that most characterize the severity of bias- or fairness-related consequences but do not follow best practices to characterize the uncertainty around either the occurrence of these consequences or severity estimates. Next, we conduct a case study of fairness evaluation for an AI-mediated resume screening task and demonstrate how principles of risk science can be incorporated into such an evaluation. Finally, we propose the AI Risk Report Card, which facilitates the reporting and communication of risk assessment results to stakeholders in positions to act based on the predicted risks. The outcomes of these activities suggest that further research at the convergence of risk science and AI evaluation can lead to advancements in AI assessments of societal impact by enabling shared frameworks to evaluate and discuss AI risks both within and outside of the scientific community.

0.36

Level Up! AI-Driven Challenges and Adaptations to Code Review Practices

cs.SE ·2026-11-13 ·Kaia Newman, Christian Bird, Jenna Butler, Emerson Murphy-Hill
survey similarity 0.61
Abstract

AI can write code faster than humans can review, causing a bottleneck and necessitating changes to code review practices. This bottleneck is hotly debated, giving rise to two extreme and orthogonal imagined futures: one where humans continue to be firmly entrenched in the review process, and one where AI agents merge pull requests entirely without human involvement. Yet, these futures remain rooted in folk wisdom. Through a mixed-methods study of interviews ($n = 24$) and a survey ($n = 2{,}590$) at a large, multinational software company with widespread access to AI, we identify AI-driven code review challenges and strategies reported as effective to meet them. We found that AI enables, necessitates, and complicates a shift in the human reviewer's role toward high-level concerns (e.g., design and intent). We found that AI-assisted authoring results in increased burden for reviewers due to challenges such as a lack of author stewardship and PR volume and size, potentially leading to downstream debt. Our survey showed AI review usage was not correlated with less difficulty keeping up with PR volume, indicating using AI for review alone is not sufficient for solving AI-driven review challenges. Developers found the most useful strategies to be organizations supporting and resourcing review, pushing review burden upstream, and prioritizing reviewer attention on the concerns that matter most.

0.25