CS Paper Radar

ADEMM: A Longitudinal Method for Monitoring Developer Efficiency in Industry

Danilo Ribeiro, Breno Alves, Gabriel Souza, César França, Alberto Souza

cs.SE ·submitted 2026-08-17 13:42 UTC · v1

Abstract

Context: Developer efficiency is influenced by technical, organizational, cognitive, and communication-related factors. However, most studies rely on one-time assessments or fixed instruments, limiting the ability to monitor how barriers emerge and change over time, especially in consulting and professional education contexts. Objective: This study proposes and evaluates the Adaptive Developer Efficiency Monitoring Method (ADEMM), an adaptive longitudinal method for monitoring developer efficiency when the monitoring organization does not directly employ the developers. Method: Following Design Science Research and Action Design Research, we conducted a mixed-method longitudinal study with 27 software developers over twelve survey cycles. ADEMM was designed and refined through five iterative cycles, combining recurring surveys, 18 semi-structured interviews, and joint evaluation with a problem owner. Results: The study resulted in ADEMM, a method that supports continuous data collection, mixed-methods integration, and iterative redesign of monitoring instruments. The evaluation produced three design principles: prioritization with the problem owner based on actionability, combination of closed and open data collection, and adaptation of items based on low variance and emerging qualitative signals. Conclusions: ADEMM provides a transferable approach for adaptive longitudinal monitoring of developer efficiency. It helps balance comparability, contextual sensitivity, and practical utility in environments where organizations need to support developers without directly controlling their work contexts.

Closest papers

Nearest neighbours by meaning across the whole index.

Factors Impacting Developer Efficiency: Results from an Adaptive Longitudinal Study

cs.SE ·2026-08-17 ·Danilo Ribeiro, Breno Alves, Gabriel Souza, César França, Alberto Souza
similarity 0.79
Abstract

Context: Developer efficiency is driven by technical, organizational, and personal factors, yet few longitudinal studies explore how these factors evolve over time. Objective: This study investigates the primary factors hindering the perceived efficiency of developers in a consulting and professional development context, analyzing how these factors vary across recurring data collection cycles and how they are described qualitatively. Method: We conducted a mixed-methods longitudinal case study applying the Adaptive Developer Efficiency Monitoring Method (ADEMM) to 27 external software developers, combining twelve waves of periodic surveys with eighteen semi-structured interviews, analyzed through statistical and thematic analysis. Results: The most frequent bottlenecks were organizational dependencies and waiting for external validation, which stayed structurally stable, followed by technical knowledge gaps, which declined as developers adapted. A generative AI usage barrier emerged qualitatively nine waves into the study, was incorporated into the survey instrument, and became the most frequently coded interview theme. Interviews corroborated the quantitative findings, with insufficient requirements documentation and organizational dependencies as the most recurrent themes alongside AI-related challenges. Conclusions: Perceived developer efficiency is highly dynamic and cannot be accurately captured through a single cross-sectional measurement. Adaptive monitoring via ADEMM identified an emerging factor, generative AI usage barriers, that a fixed instrument would have missed, and informed a concrete organizational intervention during the study. For organizations managing external developers, actions should target external dependencies, communication channels, and developers' evolving use of AI tools.

0.22

Characterizing Post-Incident Software Evolution

cs.SE ·2026-10-19 ·Matt Pope, Jonathan Sillito
similarity 0.55
Abstract

BACKGROUND: After a software incident, organizations attempt to improve the reliability of their software systems by conducting post-incident analysis and identifying actions to prevent or mitigate similar future incidents. Despite the importance of this activity, there is a limited understanding of the nature and characteristics of the software evolution triggered by the analysis process. AIMS: This study seeks to bridge this gap by providing the first empirical characterization of post-incident software evolution. We aim to establish an understanding of how these changes manifest, specifically focusing on how their characteristics (such as effort, location, and timing) differ from standard maintenance and evolutionary changes. METHOD: We conducted a mixed-methods analysis of a 10-year longitudinal dataset from Wikimedia, examining 1,302 tasks and 2,203 code reviews. We categorized these changes across five dimensions: target, effort, abandonment rate, prioritization, and time-to-resolve. RESULTS: Our results show that the post-incident evolution is distinct from other change types. Post-incident changes are more time-consuming, with a median completion time over twice that of other changes and are disproportionately concentrated in a server configuration repository. CONCLUSION: By identifying a few of the characteristics of post-incident evolution, this work provides a foundation for future research into reliability engineering and learning from failure. We highlight further avenues for exploration necessary to fully understand post-incident evolution. Lastly, our work enables researchers to move beyond high-level analysis and begin developing specialized methods and tooling tailored to the nature of post-incident evolution.

0.21

Navigating Complexity: A User-Centered Evaluation of Project Management Tools

cs.SE cs.HC ·2026-11-06 ·Yasin Ghadimi Zaker, Hamid Reza Ghaeini
similarity 0.55
Abstract · 11 Pages

In the age of digital transformation, companies are increasingly relying on specialized software to manage the complexity of projects. However, when selecting tools, features and costs often take precedence over user experience (UX), leading to low adoption rates, user frustration, and project delays. This study empirically evaluates the user experience of leading project management tools (Jira, Trello, Monday.com, Asana, and Microsoft Project) across various project scales using a mixed methods approach. The System Usability Scale (SUS) was used as the primary quantitative instrument, with data collected from 141 experienced professionals via online questionnaires. The results reveal a significant trade-off between functionality and usability. Trello achieved the highest SUS score (77.5), indicating a superior user experience for small projects and agile teams, while Microsoft Project scored the lowest (51.2), suggesting significant usability challenges despite its robust features. Jira demonstrated a balanced profile, offering powerful features with manageable complexity. Statistical analyses confirmed that the choice of the optimal tool is highly context dependent, with team size and project complexity influencing UX outcomes. The findings underscore that UX is not a secondary feature but a critical success factor for software acceptance and project effectiveness. This study provides an evidence-based framework for aligning tool selection with the project context, offers practitioners actionable insights, and identifies design opportunities for software developers.

0.33

Vibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art Review

cs.SE ·2026-08-20 ·Dominik L. Michels, Mutaz Abu Ghazaleh, Francois Lazzari, Nabil Kassem, Jonathan Klein
surveyposition similarity 0.54
Abstract

Vibe coding - AI-assisted software development in which the developer describes intent in natural language and validates results by running rather than reading the generated code - was named by Andrej Karpathy in February 2025 and produced its first body of empirical evidence within seventeen months. This state-of-the-art review assembles that evidence across a cross-disciplinary corpus spanning software engineering, human-computer interaction, labour economics, security research, governance, and education. We survey the model landscape, the tool ecosystem, and the performance record by task type, finding the early benchmarks saturated but task-level capability uneven: reliable code generation alongside weak fault detection and hard-to-audit documentation. The productivity record is at first contradictory: peer-reviewed field experiments report +26% more tasks per week, independent randomised trials measure a 19% slowdown, and team-level telemetry shows code-review time up +441%. We argue these readings are consistent once measurement method, scope, and time horizon are held constant, and identify six patterns behind the dispersion, among them effect-shrinkage under broader measurement, self-report diverging from independent measurement, output volume conflated with productivity, and bold claims walked back once tested over longer horizons. We further document security failures in deployed applications, code-quality degradation visible in large-scale code and developer telemetry, unsettled copyright exposure, and evidence of skill atrophy. The review closes with the open research questions and one falsifiable conjecture: that the gains are real on new code and shrink or reverse on mature codebases, which would account for most of the disagreement in the record.

0.20

Helpful but Fallible: Developer Experiences of AI Tools Under a Coordinated Industrial Roll-out

cs.SE ·2026-09-17 ·Andreas Bexell, Rushali Gupta, Lo Gullstrand Heander, Emma Söderberg, Per Runeson +3 more
similarity 0.54
Abstract

AI-enabled software development tools (AI-devtools) are being industrially adopted under strong expectations of productivity gains, yet developers' experiences of such roll-outs are underexplored. Organizations commit budgets, evaluate staff, and revise practice on a partial picture, since the evidence base is mainly tool evaluations, productivity metrics, and surveys, with few qualitative in-situ accounts of ongoing, coordinated roll-outs. We report a case study of a coordinated roll-out of AI-devtools at a large Swedish telecommunications company, investigating how developers experience the roll-out and how they anticipate their profession will change. We conducted semi-structured interviews with 12 software professionals across three sites, analyzed with process coding and thematic analysis, and interpreted through the extended Technology Acceptance Model (TAM2) as a post-hoc analytical lens. Our findings on use cases, productivity, frustrations, and tool limitations corroborate prior survey work. Beyond corroboration, the interviews surface a management-developer expectation gap that maps onto the TAM2 constructs of subjective norm and voluntariness, and show that participants weigh perceived risk heavily, a factor that TAM2 and similar acceptance models do not represent. AI-devtools emerge as helpful but fallible assistants whose value is shaped by organizational expectations, system scale, and developers' skills.

0.05

On the Prospects of Dynamic LLM Conversations in Software Development

cs.SE cs.AI ·2026-08-31 ·Annemarie Wittig, Alina Mailach, Janet Siegmund, Norbert Siegmund
similarity 0.54
Abstract

Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.

0.22

"It Comes in Notebooks": Changes and Challenges when Operationalizing ML Prototypes

cs.SE ·2026-09-19 ·Arumoy Shome, Luís Cruz, Diomidis Spinellis, Arie van Deursen
top venue similarity 0.54
Abstract

Machine learning practitioners commonly prototype models in computational notebooks before transitioning them to automated production systems. Despite its prevalence, the concrete engineering work involved in this transition and the software quality concerns that motivate it remain insufficiently characterized. We report on a qualitative study based on semi-structured interviews with 13 ML practitioners from industry and academia. Using reflexive thematic analysis, we identify 23 engineering changes organized into five themes: code restructuring, data pipeline development, testing & validation, pipeline automation, and monitoring & observability. We also identify 20 software quality attributes across the ML development lifecycle and map them to the engineering changes. A recurring pattern in our findings is that computational notebooks externalize oversight to the human practitioner, and defer costs that become obligatory at operationalization time. Operationalization constitutes the repayment of this technical debt accumulated during prototyping, which we refer to as oversight debt. Practitioners do not merely restructure notebook code, but repay this debt by constructing automated substitutes for the interactive oversight that notebooks provide. We further present seven quality trade-offs showing that these tensions are properties of the notebook-to-production transition, rather than symptoms of poor engineering practice. Our findings structure operationalization effort, establish empirical links between engineering changes and software quality concerns, and provide implications for practitioners, tool designers, and researchers working on ML-enabled software systems.

0.30

Beyond the Prompt: Linking What Developers Ask, Do, and Understand with Coding Agents

cs.SE ·2026-09-27 ·Yunhan Qiao, Summit Haque, Christopher Hundhausen
similarity 0.53
Abstract

Coding agents can now change code for developers, who describe goals, supply context, and respond to the agent's work. Yet prompts, screen activity, and task success each tell only part of this story. We present Say, Do, Understand, an end-to-end workflow for analyzing what developers write to an agent, what they do while it works, and what they can explain afterwards. The workflow has five stages (Capture, Prepare, Analyze, Integrate, and Interpret) and three instruments: a prompt codebook, a scheme for coding screen-recorded activities and events, and separate rubrics for explaining the process and the solution. We applied it in an observational study of ten experienced developers who used GitHub Copilot on an unfamiliar codebase. Crossing task performance with understanding produced four personas. The two measures agreed for eight developers but split for two: one passed most tests but could not explain the solution, and another passed few tests but explained it well. In this sample, the personas that most often asked the agent to check its work spent the least time testing on their own. These patterns are descriptive and do not generalize beyond the sample. We recommend the workflow to computing educators, industry practitioners, and researchers to adapt and evaluate human--AI communication in software engineering.

0.19