CS Paper Radar

Software Project Management with LLM-Based Automation: Coordination, Validation, and Governance in Practice

Ronnie de Souza Santos, Cleyton Magalhaes, Italo Santos

cs.SE ·submitted 2026-09-02 04:18 UTC · v1

Abstract

As software engineering has evolved, development environments have increasingly integrated automated support for tasks such as testing, analysis, and code generation, requiring project management to coordinate both human work and automated workflows. In this paper, we investigate how software project managers perceive changes in their work and learning demands associated with LLM-based automation. Motivated by a gap in software engineering research, which has largely focused on task-level and developer-centered uses of LLMs, we adopted an exploratory case study approach to capture managerial perspectives on automation in practice. Based on the experience of software project managers working in a large, multi-project software organization, our analysis indicates that LLM-based automation influences planning, estimation, coordination, monitoring, and governance activities rather than introducing new formal management practices. Participants described LLMs as becoming embedded in everyday project work, producing uneven effects on productivity, increasing the need for review and validation, and reducing visibility into task execution. These effects contribute to greater reliance on managerial judgment and coordination. Learning demands were perceived as experiential and incremental, centered on understanding LLM capabilities and limitations, critically assessing generated artifacts, and guiding responsible use within teams. The findings provide empirical evidence on how software project management work is adapted in contexts where LLM-based automation is integrated into ongoing software development practice.

Closest papers

Nearest neighbours by meaning across the whole index.

Understanding LLM Usage Among Early-Career Software Engineers in Practice

cs.SE ·2026-08-24 ·Julia Alencar, Ronnie de Souza Santos, Italo Santos, Cleyton Magalhaes, Danilo Monteiro Ribeiro
similarity 0.66
Abstract

Despite the rapid adoption of Large Language Models in professional software engineering, limited research has investigated how early career professionals develop effective AI assisted work practices during their transition into industry. We report findings from a mixed methods survey with 75 novice software engineers who actively use LLM supported tools in their daily work. Our results show that LLMs are embedded in routine software engineering activities, including coding, debugging, testing, documentation, and problem solving. Effective use depends on traditional software engineering competencies, such as debugging, testing, and architectural reasoning, together with critical thinking, output verification, prompt engineering, and continuous human oversight. We also identify a gap between workplace expectations and university preparation, with most participants reporting limited formal education on practical LLM use. These findings have implications for software engineering education, organizational onboarding, and workforce development in AI assisted software engineering.

0.31

Developer Task Delegation to LLMs and the Trust Factor: A Mixed Methods Study

cs.SE cs.AIcs.HC ·2026-10-25 ·Samuel Ferino, Rashina Hoda, John Grundy, Christoph Treude, Hashini Gunatilake
top venuesurvey similarity 0.61
Abstract · Accepted for publication in the Proceedings of the 49th IEEE/ACM International Conference on Software Engineering (ICSE'27)

With the rise of Large Language Models (LLMs), software developers are presented with a powerful new approach: delegating software development tasks to LLMs. Existing studies have explored task delegation for a subset of tasks and factors. Critically, what tasks developers delegate to LLMs, to what extent, and why they delegate them, is not yet well understood. We explore these questions through 22 interviews and a survey with 160 software practitioners. Developers reported employing four modes of task delegation to LLMs: full delegation, partial delegation, self-delegation, and task avoidance. Drawing on our qualitative findings, we discuss the rationales behind these choices and explore how task difficulty, task risk, developer motivation, and trust appear to be associated with software development task delegation to LLMs. Our statistical analysis reveals that trust is a significant factor associated with task delegation to LLMs. Finally, we share actionable implications for human-AI collaboration in software research, and for software practitioners to make the most of their newly found powers of delegation.

0.43

When Review Alone No Longer Scales: Layered Supervision in AI-Assisted Software Engineering

cs.SE ·2026-08-26 ·Markus Stolze, Mirco Strässle
top venue similarity 0.59
Abstract · Accepted for publication at the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), Software Enginee…

AI-assisted development tools enable software engineers to generate implementations at substantially higher speed and volume than in traditional workflows. Software teams have long relied on guardrails -- standing control mechanisms such as code review, linting, testing, and CI/CD pipelines -- to maintain quality and coordination. High-throughput AI-assisted generation increases pressure on these guardrails -- straining their capacity to keep pace with the volume and rate of generated changes -- and reshapes how organizations supervise development workflows, yet relatively little is known about how existing guardrails evolve in response. We conducted a qualitative interview study with five software engineering practitioners, situated within a broader practitioner survey. Our findings indicate that organizations distribute the work of supervision across multiple guardrail layers: preventive guardrails (produced by externalizing architectural intent and conventions into machine-interpretable form), executable guardrails (linting, testing, and CI/CD repurposed as scalable supervision infrastructure), and human oversight (shifting from line-by-line inspection toward supervisory interpretation focused on architectural reasoning, explainability, and long-term maintainability). We characterize this as a transition from review- centric guardrails toward layered supervision, in which no single guardrail carries the supervision load alone.

0.35

''You Can't Open an LLM With a Screwdriver'': The De-Democratization of Software

cs.SE ·2026-08-25 ·Zixuan Feng, Italo Santos, Kostadin Damevski, Anita Sarma
publishedposition similarity 0.58
Abstract · Accepted to the ACM AI Leadership Summit 2026

Claims that generative AI will soon write all of the code have led to predictions that programming is nearing its end. In this vision paper, we argue against this assumption that broader access to code generation necessarily democratizes software development, i.e., everyone can code but we have to distinguish between access and control: by access, we mean the ability of more people, including non-experts and less-experienced developers, to generate code-like artifacts with AI; by control, we mean the capacity to inspect, evaluate, integrate, maintain, and govern those artifacts as dependable software. While AI may broaden access to code production, control may become more concentrated among those who own or understand the code, software practices, infrastructure, evaluation practices, and deployment pipelines. Grounded in an expert panel, our vision paper argues that AI does not eliminate software engineering expertise but shifts where that expertise becomes most critical. The locus of software engineering expertise is shifting toward intent specification: orchestrating and governing AI behavior, evaluating software behavior, and integrating software systems. We conclude this paper by identifying research opportunities for education, tools, and policy that can help the software engineering community respond to the AI era with greater agency, accountability, and adaptability.

0.17

From Prompting to Engineering: A Research Agenda for Prompt Engineering in Software Engineering

cs.SE ·2026-09-02 ·Vincenzo De Martino, Giovanna Broccia, Fabiano Pecorelli, Jennifer Horkoff, Riccardo Coppola +4 more
top venue similarity 0.58
Abstract

Prompt engineering is increasingly used across Software Engineering (SE) activities, including requirements analysis, coding, testing, documentation, repository analysis, and planning. Yet prompts and related instruction artifacts are often created and evolved through task-specific and informal practices, with limited support for their systematic evaluation, management, traceability, and governance. To examine how SE can contribute to the maturation of these practices, we organized a structured community discussion at the First International Workshop on Empirical Prompt Engineering for Software Engineering (PROMPT-SE), co-located with EASE 2026. Participants discussed current prompting practices, challenges to their adoption and evaluation, and future directions for integrating prompt engineering into software development. We synthesized these discussions into five areas: prompt artifacts and standardization; evaluation and benchmarking; lifecycle integration; human-AI collaboration and skills; and governance, privacy, and technical debt. Based on these areas, we outline a research agenda to move prompt engineering from predominantly ad hoc interactions toward more systematic, maintainable, evaluable, traceable, and governable SE practices.

0.69

"It Comes in Notebooks": Changes and Challenges when Operationalizing ML Prototypes

cs.SE ·2026-09-19 ·Arumoy Shome, Luís Cruz, Diomidis Spinellis, Arie van Deursen
top venue similarity 0.57
Abstract

Machine learning practitioners commonly prototype models in computational notebooks before transitioning them to automated production systems. Despite its prevalence, the concrete engineering work involved in this transition and the software quality concerns that motivate it remain insufficiently characterized. We report on a qualitative study based on semi-structured interviews with 13 ML practitioners from industry and academia. Using reflexive thematic analysis, we identify 23 engineering changes organized into five themes: code restructuring, data pipeline development, testing & validation, pipeline automation, and monitoring & observability. We also identify 20 software quality attributes across the ML development lifecycle and map them to the engineering changes. A recurring pattern in our findings is that computational notebooks externalize oversight to the human practitioner, and defer costs that become obligatory at operationalization time. Operationalization constitutes the repayment of this technical debt accumulated during prototyping, which we refer to as oversight debt. Practitioners do not merely restructure notebook code, but repay this debt by constructing automated substitutes for the interactive oversight that notebooks provide. We further present seven quality trade-offs showing that these tensions are properties of the notebook-to-production transition, rather than symptoms of poor engineering practice. Our findings structure operationalization effort, establish empirical links between engineering changes and software quality concerns, and provide implications for practitioners, tool designers, and researchers working on ML-enabled software systems.

0.30

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

cs.SE ·2026-08-21 ·Jacek Dąbrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari, Mohammad Amin Zadenoori, Yijun Yu
code/data similarity 0.56
Abstract

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

0.48

The Comprehension Gap in LLM-Assisted Software Development: An Experiment

cs.SE ·2026-11-18 ·Julian Oertel, Jil Klünder, Regina Hebig
similarity 0.56
Abstract

When LLM tools are pushed beyond their capabilities for software development tasks, developers must be able to step in. However, the comprehension that developers acquire during development might be negatively impacted due to excessive code generation. This could create a cognitive debt that needs to be resolved before tasks can be solved efficiently again. We explore how well developers understand code they generated by prompting in comparison to self-written and unseen code and how they reflect on their comprehension. We conduct a crossover experiment and interviews with 35 participants consisting of students, academic personnel, and software developers. After completing two programming tasks with or without LLM chat usage, respectively, participants are tasked with finding introduced mutations in the produced and additional unseen code. We use insights from interviews and screen recordings to supplement our results. Regarding the experiment, 36% of participants report a better understanding of self-written code compared to their generated code. On average, participants solved the comprehension task 20% faster for self-written code than for generated code and found 11% more mutations. Participants needed significantly less time to solve programming tasks when working with the LLM. However, participants with high reliance on the LLM chat and low task engagement showed significantly decreased comprehension. Our qualitative results show great variability in individual perspectives on comprehension differences and strategies to mitigate issues. Strategies for dealing with comprehension differences range from accepting comprehension differences to adapting their LLM interactions or avoiding the tools altogether. Our results warrant further investigation into cognitive debt in larger systems and developer practices to mitigate issues.

0.24