CS Paper Radar

Developer Task Delegation to LLMs and the Trust Factor: A Mixed Methods Study

Samuel Ferino, Rashina Hoda, John Grundy, Christoph Treude, Hashini Gunatilake

cs.SE cs.AIcs.HC ·submitted 2026-10-25 14:28 UTC · v1
top venuesurvey

Abstract

With the rise of Large Language Models (LLMs), software developers are presented with a powerful new approach: delegating software development tasks to LLMs. Existing studies have explored task delegation for a subset of tasks and factors. Critically, what tasks developers delegate to LLMs, to what extent, and why they delegate them, is not yet well understood. We explore these questions through 22 interviews and a survey with 160 software practitioners. Developers reported employing four modes of task delegation to LLMs: full delegation, partial delegation, self-delegation, and task avoidance. Drawing on our qualitative findings, we discuss the rationales behind these choices and explore how task difficulty, task risk, developer motivation, and trust appear to be associated with software development task delegation to LLMs. Our statistical analysis reveals that trust is a significant factor associated with task delegation to LLMs. Finally, we share actionable implications for human-AI collaboration in software research, and for software practitioners to make the most of their newly found powers of delegation.

Comment
Accepted for publication in the Proceedings of the 49th IEEE/ACM International Conference on Software Engineering (ICSE'27)

Closest papers

Nearest neighbours by meaning across the whole index.

(Don't) Trust, but (Don't) Verify: Developers' Attention to Security in AI-Generated Code

cs.CR cs.SE ·2026-09-17 ·Hamza Khalid, Ronald E. Thompson, Alejandra Sabater, Perucy Mussiba, Kelsey R. Fulton +1 more
top venue similarity 0.70
Abstract · To appear at IEEE Security and Privacy (S&P) '27

AI coding assistants are rapidly transforming software development, but are known to produce insecure code. Prior work has measured whether AI-assisted developers produce secure code, but less is known about how they evaluate AI-generated code: whether they can identify vulnerabilities, what cues they use, and how trust shapes their decisions. This evaluation step is foundational to secure development with AI, whether using auto-complete, chat tools, or AI agents. As a first step, we conducted a remote observational study with 100 participants isolating this evaluation stage. Participants were tasked with producing secure and functional code for four C linked-list tasks. For each, participants were able to cycle through five AI-generated suggestions varying in security and functionality, select one, and edit their choice into a final submission. Participants also completed a post-study survey about their decision-making and perception of AI-generated code's security and 23 completed a more in-depth interview.

0.44

The Comprehension Gap in LLM-Assisted Software Development: An Experiment

cs.SE ·2026-11-18 ·Julian Oertel, Jil Klünder, Regina Hebig
similarity 0.68
Abstract

When LLM tools are pushed beyond their capabilities for software development tasks, developers must be able to step in. However, the comprehension that developers acquire during development might be negatively impacted due to excessive code generation. This could create a cognitive debt that needs to be resolved before tasks can be solved efficiently again. We explore how well developers understand code they generated by prompting in comparison to self-written and unseen code and how they reflect on their comprehension. We conduct a crossover experiment and interviews with 35 participants consisting of students, academic personnel, and software developers. After completing two programming tasks with or without LLM chat usage, respectively, participants are tasked with finding introduced mutations in the produced and additional unseen code. We use insights from interviews and screen recordings to supplement our results. Regarding the experiment, 36% of participants report a better understanding of self-written code compared to their generated code. On average, participants solved the comprehension task 20% faster for self-written code than for generated code and found 11% more mutations. Participants needed significantly less time to solve programming tasks when working with the LLM. However, participants with high reliance on the LLM chat and low task engagement showed significantly decreased comprehension. Our qualitative results show great variability in individual perspectives on comprehension differences and strategies to mitigate issues. Strategies for dealing with comprehension differences range from accepting comprehension differences to adapting their LLM interactions or avoiding the tools altogether. Our results warrant further investigation into cognitive debt in larger systems and developer practices to mitigate issues.

0.24

From Delegation to Action Approval: An Exploratory Study of Developers' Practices in Collaborating with AI Coding Agents

cs.SE ·2026-11-06 ·Yinan Wu, Dominik Wermke, Ze Shi Li, Kathryn Thomasset Stolee, Bowen Xu
similarity 0.66
Abstract

Developers now collaborate with AI coding agents that write code and take actions on their behalf. Beyond reviewing this code, developers decide what to delegate and which agent actions to approve. Some of these actions, such as running shell commands or reading files, can have effects that never appear in the resulting code. How developers make these decisions and how much effort this requires are not well understood. To better understand this collaboration, we surveyed 203 developers who use AI coding agents. We found that careful code review does not necessarily extend to action approval: 61% of developers who carefully reviewed most or all generated code still often approved at least one type of agent action without close review. Developers aware of security or privacy incidents or concerning practices did so more often than those who were not. We also found that autonomy varied more between developers than between development activities. Although developers reported a lighter workload, supervising agents remained mentally demanding. When asked what would make them grant more autonomy, developers wanted more than reliability: more than half of their responses called for agents that are bounded, observable, or reversible. Our findings suggest that as agents take on more of the coding, the challenge shifts toward supporting the developers who supervise them.

0.10

When Review Alone No Longer Scales: Layered Supervision in AI-Assisted Software Engineering

cs.SE ·2026-08-26 ·Markus Stolze, Mirco Strässle
top venue similarity 0.65
Abstract · Accepted for publication at the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), Software Enginee…

AI-assisted development tools enable software engineers to generate implementations at substantially higher speed and volume than in traditional workflows. Software teams have long relied on guardrails -- standing control mechanisms such as code review, linting, testing, and CI/CD pipelines -- to maintain quality and coordination. High-throughput AI-assisted generation increases pressure on these guardrails -- straining their capacity to keep pace with the volume and rate of generated changes -- and reshapes how organizations supervise development workflows, yet relatively little is known about how existing guardrails evolve in response. We conducted a qualitative interview study with five software engineering practitioners, situated within a broader practitioner survey. Our findings indicate that organizations distribute the work of supervision across multiple guardrail layers: preventive guardrails (produced by externalizing architectural intent and conventions into machine-interpretable form), executable guardrails (linting, testing, and CI/CD repurposed as scalable supervision infrastructure), and human oversight (shifting from line-by-line inspection toward supervisory interpretation focused on architectural reasoning, explainability, and long-term maintainability). We characterize this as a transition from review- centric guardrails toward layered supervision, in which no single guardrail carries the supervision load alone.

0.35

Spec-Driven Development as Prescribed and as Practiced

cs.SE cs.AI ·2026-08-25 ·Shyam Agarwal, Anmol Singhal, Andrew Macvean, Bogdan Vasilescu, Travis Breaux
highly cited similarity 0.64
Abstract

AI coding agents implement software from naturallanguage instructions, but the intent behind that work is lost when stated only in chat sessions. Spec-driven development (SDD) addresses this by having developers and agents write specifications that guide implementation, yet evidence on its use comes largely from tool vendors. We study SDD in practice through a thematic analysis of Reddit discussions, interviews with 21 practitioners, including eight tool makers, and 266 active and popular repositories in SpecMine, a corpus of 470,795 spec files from 73,030 public GitHub repositories that we release. Practitioners agreed that specs should capture intent, but many also included implementation details, used SDD selectively and mainly for new features, and reported more reliable agent behavior alongside greater planning and review costs. Our findings show that implementation detail may help agents while making specs prone to decay, that tool-specific record structures shape what people and agents see, and that participants described spec-code checks mainly when changes are completed rather than over time. We also find that individual benefits may come from clarifying intent before delegation, while team benefits may depend on SDD becoming a shared process. We derive implications for tool builders, practitioners, and researchers.

0.36

AI Policies: Help or Hindrance? A Software Developer's Perspective

cs.SE cs.AI ·2026-09-15 ·Samuel Ferino, Rashina Hoda, John Grundy, Christoph Treude, Hashini Gunatilake
similarity 0.63
Abstract

AI policies introduced by software organisations to mitigate LLM-related risks such as sensitive information leaks and unauthorised usage are not useful if software developers do not engage with them. We draw on 19 software developer interviews to show how AI policies help and hinder developers. We suggest approaches to support managers and decision makers with a developer-centric approach to introducing AI policies.

0.22

Deliberate Pacing for Human Oversight in Agentic Software Development

cs.SE ·2026-11-06 ·Tobias Reichel, Michèle Stassen, Anamaria Mojica-Hanke, Maria Teresa Baldassarre, Sebastian Baltes +2 more
position similarity 0.63
Abstract

Agentic AI increasingly enables software development activities to proceed without continuous human involvement. While this autonomy may accelerate development, it also raises questions about when developers can meaningfully oversee software development done by agents. We propose deliberate pacing: intentionally adjusting when and how autonomous development proceeds to preserve opportunities for meaningful oversight. Drawing on an exploratory multivocal literature review of 20 sources, we identify three factors motivating this vision: changing relative costs of human involvement, the timing of oversight opportunities, and interdependencies between pacing decisions across the software development life cycle. We argue that neither uninterrupted autonomy nor frequent human intervention is universal. Instead, pacing should balance opportunities to understand and influence consequential decisions against costs in human attention, development time, and rework. We outline a research agenda to investigate when deliberate pacing is beneficial, how intervention timing affects oversight, and how pacing decisions interact across development activities.

0.11

You can contribute if you... An Empirical Framework of AI Contribution Policies in OSS

cs.SE ·2026-09-07 ·Gregorio Robles, Daniel M. German
code/data similarity 0.63
Abstract

Artificial intelligence is reshaping open source software (OSS) contribution by lowering the cost of producing code, documentation, issue reports, and review interactions. This creates opportunities for broader participation, but also disrupts how maintainers assess contributor effort, competence, and accountability. In response, OSS projects are beginning to regulate AI-mediated contribution through contribution guidelines and other project documentation. This paper presents an empirical study of these emerging policies. We analyze project policies on AI-mediated contributions by evaluating their underlying rationales, rules, and expectations. Our analysis shows that these policies seek to protect scarce maintainer attention, preserve accountability, sustain meaningful review interactions, address legal and quality concerns, and maintain pathways for newcomer learning. Based on these findings, we introduce the AI Contribution Governance Framework, which organizes recurring concerns and governance mechanisms across projects. The framework helps OSS communities develop AI contribution policies and provides researchers with a vocabulary for studying how AI is changing collaborative software production.

0.46