CS Paper Radar

"It Comes in Notebooks": Changes and Challenges when Operationalizing ML Prototypes

Arumoy Shome, Luís Cruz, Diomidis Spinellis, Arie van Deursen

cs.SE ·submitted 2026-09-19 09:22 UTC · v1
top venue

Abstract

Machine learning practitioners commonly prototype models in computational notebooks before transitioning them to automated production systems. Despite its prevalence, the concrete engineering work involved in this transition and the software quality concerns that motivate it remain insufficiently characterized. We report on a qualitative study based on semi-structured interviews with 13 ML practitioners from industry and academia. Using reflexive thematic analysis, we identify 23 engineering changes organized into five themes: code restructuring, data pipeline development, testing & validation, pipeline automation, and monitoring & observability. We also identify 20 software quality attributes across the ML development lifecycle and map them to the engineering changes. A recurring pattern in our findings is that computational notebooks externalize oversight to the human practitioner, and defer costs that become obligatory at operationalization time. Operationalization constitutes the repayment of this technical debt accumulated during prototyping, which we refer to as oversight debt. Practitioners do not merely restructure notebook code, but repay this debt by constructing automated substitutes for the interactive oversight that notebooks provide. We further present seven quality trade-offs showing that these tensions are properties of the notebook-to-production transition, rather than symptoms of poor engineering practice. Our findings structure operationalization effort, establish empirical links between engineering changes and software quality concerns, and provide implications for practitioners, tool designers, and researchers working on ML-enabled software systems.

Closest papers

Nearest neighbours by meaning across the whole index.

From Prompting to Engineering: A Research Agenda for Prompt Engineering in Software Engineering

cs.SE ·2026-09-02 ·Vincenzo De Martino, Giovanna Broccia, Fabiano Pecorelli, Jennifer Horkoff, Riccardo Coppola +4 more
top venue similarity 0.65
Abstract

Prompt engineering is increasingly used across Software Engineering (SE) activities, including requirements analysis, coding, testing, documentation, repository analysis, and planning. Yet prompts and related instruction artifacts are often created and evolved through task-specific and informal practices, with limited support for their systematic evaluation, management, traceability, and governance. To examine how SE can contribute to the maturation of these practices, we organized a structured community discussion at the First International Workshop on Empirical Prompt Engineering for Software Engineering (PROMPT-SE), co-located with EASE 2026. Participants discussed current prompting practices, challenges to their adoption and evaluation, and future directions for integrating prompt engineering into software development. We synthesized these discussions into five areas: prompt artifacts and standardization; evaluation and benchmarking; lifecycle integration; human-AI collaboration and skills; and governance, privacy, and technical debt. Based on these areas, we outline a research agenda to move prompt engineering from predominantly ad hoc interactions toward more systematic, maintainable, evaluable, traceable, and governable SE practices.

0.69

Operationalizing Responsible AI in MLOps and ML Production: A Systematic Literature Review

cs.SE ·2026-11-04 ·Hinduja Nirujan, Siddhartha Paudel, Gouri Ginde
survey similarity 0.65
Abstract · 17 pages, 6 figures, 4 tables. Systematic literature review of 44 primary studies

Artificial intelligence (AI)-enabled systems are increasingly deployed in operational environments where machine learning (ML) models interact with changing data, users, infrastructure, and organizational processes. While Responsible AI principles are widely discussed, less is known about how these principles are translated into concrete mechanisms within Machine Learning Operations (MLOps) and ML production workflows. This paper reports a systematic literature review of studies that operationalize Responsible AI in ML production. The review synthesizes evidence across monitoring and runtime assurance, audit and compliance, ML lifecycle integration, and traceability/provenance/documentation. Across the final study set, the literature shows a shift from abstract ethical principles toward engineering mechanisms such as drift monitoring, runtime checks, audit logs, provenance records, evidence bundles, governance gates, policy-as-code, model passports, and compliance-oriented documentation. However, the evidence base remains uneven: many studies propose architectures or prototypes, while fewer evaluate the mechanisms in realistic production settings, longitudinal operations, or organizational governance workflows. The review identifies a central gap around integrated, evidence-producing Responsible MLOps: current mechanisms often detect, document, or govern isolated parts of the ML lifecycle, but rarely connect runtime signals, ML lifecycle provenance, human accountability, and compliance evidence into closed operational loops. These findings motivate future software engineering research on traceable, evaluable, and operationally integrated Responsible AI support for ML production.

0.30

Reshaping the SDLC for Data- and AI-Centric Systems

cs.SE ·2026-08-18 ·Mamdouh Alenezi
similarity 0.64
Abstract

The traditional Software Development Lifecycle (SDLC) assumes that system behavior is determined primarily by source code, allowing correctness to be specified, implemented, and verified through code-centric practices. Data-intensive and AI-enabled systems challenge this assumption because their behavior emerges from the interaction of code, data, and learned models, while performance may degrade as real-world conditions drift from training data. This paper examines how integrating data engineering and software engineering practices, operationalized through DataOps, MLOps, and LLMOps, reshapes the SDLC for these systems. We make four contributions. First, we synthesize literature across software engineering, data management, machine learning systems, and human-centered computing into a phase-structured account of lifecycle transformation spanning requirements, architecture, development, testing, deployment, monitoring, governance, and organization. Second, we provide a lightweight formalization in which system behavior is defined over code, data, and model configurations; requirements become evaluation-led specifications with probabilistic acceptance regions; and promotion is controlled through statistically grounded validation gates. Third, we develop an adaptive five-layer lifecycle framework comprising artifact, contract, gate, control, and governance layers, positioning maintenance as a closed-loop control problem under configuration drift. Fourth, we propose a conceptual research model linking data engineering integration to measurable lifecycle outcomes and critically assess the evidence base. While the direction of transformation is increasingly established, its magnitude remains insufficiently quantified. We conclude with a research agenda for an empirically grounded, adaptive SDLC for data- and AI-centric systems.

0.30

CLEAR: Structured Specifications for Controllable Long-Horizon ML Research Engineering

cs.SE ·2026-11-04 ·Xin Liu, Hongru Song, Ruqing Zhang, Jiafeng Guo, Xueqi Cheng
similarity 0.63
Abstract · 11 pages, 4 figures

Large language model (LLM) agents are increasingly being used to automate machine learning (ML) research workflows, ranging from code development and experiment execution to research reproduction and scientific discovery. Recent systems demonstrate that LLMs can perform substantial portions of ML research engineering with limited human intervention. These advances suggest a shift from automating individual research tasks toward automating long-horizon workflows that transform research artifacts into executable ML systems. However, ML research engineering remains substantially more difficult than isolated code generation. A research artifact typically specifies a system only implicitly through a mixture of natural-language descriptions, equations, pseudo-code, implementation details, datasets, and evaluation protocols. An agent must first infer the intended computational structure, translate it into executable components, coordinate their dependencies, and repeatedly diagnose and repair failures during execution. Consequently, the quality of the final system depends not only on code generation, but also on whether the agent can maintain a consistent representation of what should be implemented throughout this long-horizon process.

0.19

Spec-Driven Development as Prescribed and as Practiced

cs.SE cs.AI ·2026-08-25 ·Shyam Agarwal, Anmol Singhal, Andrew Macvean, Bogdan Vasilescu, Travis Breaux
highly cited similarity 0.61
Abstract

AI coding agents implement software from naturallanguage instructions, but the intent behind that work is lost when stated only in chat sessions. Spec-driven development (SDD) addresses this by having developers and agents write specifications that guide implementation, yet evidence on its use comes largely from tool vendors. We study SDD in practice through a thematic analysis of Reddit discussions, interviews with 21 practitioners, including eight tool makers, and 266 active and popular repositories in SpecMine, a corpus of 470,795 spec files from 73,030 public GitHub repositories that we release. Practitioners agreed that specs should capture intent, but many also included implementation details, used SDD selectively and mainly for new features, and reported more reliable agent behavior alongside greater planning and review costs. Our findings show that implementation detail may help agents while making specs prone to decay, that tool-specific record structures shape what people and agents see, and that participants described spec-code checks mainly when changes are completed rather than over time. We also find that individual benefits may come from clarifying intent before delegation, while team benefits may depend on SDD becoming a shared process. We derive implications for tool builders, practitioners, and researchers.

0.36

Technical Debt Risks in AI-Assisted Software Development: Insights from Practitioners

cs.SE ·2026-11-11 ·Nazia Jahan, Nafisa Alam, Sourov Debnath, Arafat Rahman, S. M. Waesh Ahmed +2 more
similarity 0.61
Abstract

AI coding assistants increasingly support software development, raising concerns about technical debt associated with AI-generated code and development practices. We investigate these concerns through a sequential mixed-methods study involving 137 survey respondents and 20 follow-up interviews with software professionals in Bangladesh. Drawing on ten established technical debt dimensions, we examine perceived risks, contributing circumstances, maintenance consequences, validation practices, and developer understanding. Respondents expressed the strongest concerns about People Debt (73.0\%), Management Debt (70.1\%), and Architecture Debt (68.6\%), suggesting that risks extend beyond code quality. Participants associated ambiguous requirements, insufficient project context, and unrequested AI-generated changes with architectural inconsistencies, rework, and maintenance difficulties. Their experiences revealed contrasting outcomes: AI assistance supported refactoring, maintainability improvements, and testing, but also introduced rework and context misalignment. Although 89.8\% reported regularly verifying AI-generated code, participants expressed concerns about AI dependence, reduced code understanding, and review effort. They also emphasized the need for stronger architectural awareness, requirements clarification, and validation support in coding assistants. Our findings suggest that managing technical debt risks requires attention to project context, implementation decisions, and developer understanding alongside generated code quality.

0.11

Cognitive Debt in the GenAI Era - Listening to Practitioners' Voices on the Loss of Understanding

cs.SE ·2026-10-27 ·Theocharis Tavantzis, Bianca Trinkenreich, Michaela Greiler, Júlia Condé Araújo, Marina Condé Araújo +4 more
similarity 0.61
Abstract · 12 pages, 3 figures. Manuscript under review at ICSE-SEIP

Fast-paced AI-assisted software development risks reducing developers' understanding of the systems they build. This growing gap between what the system does and what developers understand about it, described as ``cognitive debt'', is not new, but AI has exacerbated it. This has led to a recent surge in blog posts and online discussions, in which practitioners recognize and write about cognitive debt, and share their experiences well before academic studies can capture them. To understand their concerns and inform future research and practice, we conduct a grey literature review of 76 sources, focusing on how cognitive debt is conceptualized and manifests, its contributing factors, its consequences, and the practices to prevent, detect, and mitigate it. We find that cognitive debt can accumulate unnoticed behind working code and passing tests, particularly when developers accept AI-generated solutions without building understanding. Practitioners report consequences such as harder maintenance and debugging, fragmented knowledge in the team, and reduced productivity. They propose practices such as documenting design rationale, review-sized changes, comprehension gates, and knowledge-sharing rituals, along with signals like reliability metrics to detect it. Our findings give software teams a structured view of emerging practical approaches in AI-assisted development and stress the need to preserve human understanding alongside faster code production.

0.55

Helpful but Fallible: Developer Experiences of AI Tools Under a Coordinated Industrial Roll-out

cs.SE ·2026-09-17 ·Andreas Bexell, Rushali Gupta, Lo Gullstrand Heander, Emma Söderberg, Per Runeson +3 more
similarity 0.60
Abstract

AI-enabled software development tools (AI-devtools) are being industrially adopted under strong expectations of productivity gains, yet developers' experiences of such roll-outs are underexplored. Organizations commit budgets, evaluate staff, and revise practice on a partial picture, since the evidence base is mainly tool evaluations, productivity metrics, and surveys, with few qualitative in-situ accounts of ongoing, coordinated roll-outs. We report a case study of a coordinated roll-out of AI-devtools at a large Swedish telecommunications company, investigating how developers experience the roll-out and how they anticipate their profession will change. We conducted semi-structured interviews with 12 software professionals across three sites, analyzed with process coding and thematic analysis, and interpreted through the extended Technology Acceptance Model (TAM2) as a post-hoc analytical lens. Our findings on use cases, productivity, frustrations, and tool limitations corroborate prior survey work. Beyond corroboration, the interviews surface a management-developer expectation gap that maps onto the TAM2 constructs of subjective norm and voluntariness, and show that participants weigh perceived risk heavily, a factor that TAM2 and similar acceptance models do not represent. AI-devtools emerge as helpful but fallible assistants whose value is shaped by organizational expectations, system scale, and developers' skills.

0.05