CS Paper Radar

Beyond Similarity-Based Generation: Guiding Commit Message Generation with Issue-Commit Traceability

Chunying Zhou, Gong Chen, Xiaoyuan Xie, Yi Song, Yuxuan Yang, Peng He

cs.SE ·submitted 2026-10-18 14:36 UTC · v1
code/data

Abstract

Commit messages provide records of code changes and help developers understand and maintain software. To reduce the manual effort involved in writing them, commit message generation (CMG) approaches automatically generate commit messages that describe code changes in natural language. Existing CMG approaches generally treat CMG as a standalone task and optimize generated messages primarily for similarity to developer-written references. However, this objective may not properly reflect the practical usefulness of generated commit messages. This limitation matters in current AI-assisted software engineering workflows, where commit messages are used as shared information to serve different software engineering activities. To fill this gap, we propose TraceMsg, which uses issue-commit traceability, a representative software engineering activity, to guide commit message generation. TraceMsg is a collaborative multi-agent framework that coordinates message generation, traceability reasoning, and feedback-guided refinement, allowing generated messages to serve traceability while using reasoning feedback to guide issue-conditioned refinement. We evaluate TraceMsg on six open-source projects. The experimental results show that TraceMsg achieves an average Hit@1 of 0.893 and an MRR of 0.911, improving Hit@1 and MRR by 13.16% and 38.75%, respectively, over the best baseline for each metric.

Comment
16 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Tracybot: Preserving Developer Intent by Tracing Agent-Generated Code to Prompts

cs.SE cs.HC ·2026-11-06 ·Răzvan Albu, Aliaksei Khval, Maxine Orlova, Danis Music, Filipe Araújo Veras Rosa +2 more
surveycode/data similarity 0.75
Abstract

Large language models (LLMs) are now widely used in software development, and LLM-based coding agents are given increasing autonomy. With coding agents, many design decisions are made in the conversation between the developer and the agent. Typically, the developer starts a conversation with an agent and states their intent, the agent proposes options if needed, and the developer plans the change accordingly. Then, the resulting code is committed. However, in such scenarios, the decisions and the intent behind the changes remain in the conversation, with no link to the code they produced. In this paper, we present Tracybot, an open-source tool that integrates with VS Code, Git, and coding agents to keep track of relevant conversations in an LLM-assisted development process. Tracybot captures each change as a Tasklet which consists of optional planning prompts used to plan a change, together with a build prompt that produces it. Tracybot has two main features: (i) it maps line-level code changes made by coding agents to the Tasklets that produced them, and (ii) it allows the developer to export a Tasklet as a decision record when needed, for example when the decision concerns the architecture of the project. We developed the tool over two iterations of a design science study, while focusing the evaluation on the usability and user-friendliness of the tool. In a focus group (7 participants) and a survey (24 participants), we found that Tracybot was perceived as useful for activities like code review, and most participants found our Tasklet design and presentation to be a clear way to represent the developer's intent. Tool: https://github.com/TracyTeam/tracybot. Video: https://youtu.be/Vg_a1F5eO04.

0.21

Rethinking Requirement-to-Code Traceability Through Single-Agent and Multi-Agent Approaches

cs.CE cs.SE ·2026-11-06 ·Sarthak Banerjee, Sallam Abualhaija, Domenico Bianculli, Lionel Briand
similarity 0.73
Abstract

Requirements-to-code traceability supports essential software engineering activities, including change impact analysis, maintenance, and regulatory compliance. Existing approaches typically rely on a retrieval-and-classification pipeline, which suffers from an inherent retrieval bottleneck: relevant source files that are not retrieved cannot be recovered during subsequent analysis. In this paper, we revisit requirements-to-code traceability in light of the substantially larger context windows offered by modern large language models, which make repository-scale analysis feasible without a separate retrieval step. We propose three approaches: a single-agent approach that analyzes the entire code repository in one pass, and two multi-agent approaches based on self-verification and multi-agent debate. Our evaluation shows that the single-agent approach substantially improves recall, achieving 80.9%, a gain of 38.6 percentage points (pp) over the state-of-the-art retrieval-based baseline, albeit with a 25.8 pp reduction in precision. By leveraging collaborative reasoning, the multi-agent debate-based approach achieves the best overall precision-recall balance, with 24.5% precision and 71.0% recall, yielding an F2 score of 49.5%. This represents an improvement of 7.5 pp over the state of the art and 2.7 pp over the single-agent approach. While the quantitative evaluation establishes the overall effectiveness of the approaches, the aggregate metrics alone explain little due to the considerable variability across results. Importantly, qualitative analysis provides key insights into highly variable quantitative results: the retrieval approach suits requirements that echo the code's vocabulary; single-agent, full-context reasoning suits well-documented, consistently named code; and multi-agent debate suits indirect, conflicting, or distributed evidence.

0.15

WiseSpec: Requirements-Driven Agents for Code Generation

cs.SE cs.AI ·2026-09-01 ·Zhao Tian
top venue similarity 0.71
Abstract · Accepted by ASE 2026 (SRC)

Code generation aims to automatically generate source code from task requirements and has attracted significant attention with the rapid advancement of large language models (LLMs). Despite remarkable progress, LLMs often struggle to generate correct code for complex software engineering tasks because task descriptions are frequently incomplete, ambiguous, or lack critical contextual information. Existing approaches primarily improve the capabilities of coding agents through more sophisticated tools, skills, and workflows, while largely overlooking the quality of the task requirements themselves. To address this limitation, we draw inspiration from software requirements engineering and propose WiseSpec, a novel requirements-driven agent framework for repository-level code generation. WiseSpec automatically constructs structured and information-rich requirements, assesses their quality through execution-based evaluation, and iteratively refines them to better guide code generation. Experimental results show that WiseSpec consistently outperforms all baselines, achieving an average improvement of 13.17% in %Resolved.

0.36

ComBeautify: Leveraging Code Change Summaries for Conventional Commit Generation

cs.SE cs.AI ·2026-10-25 ·Rishi Raj, Aashray Tandon, Nikolay Matyunin, Jibesh Patra
code/data similarity 0.71
Abstract

Modern software development relies on version control systems such as Git to track changes and facilitate collaboration. While commit messages are crucial for conveying developer intent, they do not affect program functionality; as a result, developers often lack motivation to write meaningful ones. The Conventional Commits Specification (CCS) addresses this with a structured format and guidelines that help developers express the intent of a change, improving clarity and enabling automation. Yet, developers frequently struggle to express intent while adhering to the format. In particular, they may misclassify commits due to the large number of possible categories or the lack of project-specific guidance. We introduce ComBeautify, an approach that automatically generates CCS-compliant commit messages. Our hybrid approach leverages context-preserving summarization of Git diffs with large language models (LLMs) to improve commit classification accuracy under CCS. Rather than relying on manually curated category definitions, we learn them from historical commits. We also propose a method for automatically generating the CCS description component. Our empirical evaluation across 13 languages, 6,500 open-source repositories, and over 20 million commits provides long-term insights into CCS adoption, revealing increasing compliance. Our evaluation on twenty popular repositories reveals the limitations of approaches guided by generic category definitions, and shows that ComBeautify, when trained on project-specific data, outperforms baselines by up to 3x. Additionally, using a dataset of 2,000 manually curated commits, we validate our summarization approach, achieving up to 95.25% accuracy and surpassing baselines. These findings emphasize the need for adaptable, context-aware AI methods for commit classification and CCS-compliant commit message generation.

0.25

A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

cs.SE cs.AI ·2026-08-10 ·Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu +9 more
benchmarkcode/datahighly cited similarity 0.71
Abstract · 9 pages

Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.

0.69

Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows

cs.SE ·2026-09-23 ·Jasem Khelifi, Issam Oukhay, Ali Ouni, Mohammed Sayagh, Mohamed Aymen Saied
similarity 0.71
Abstract

Agentic workflows shift software development from prompting AI agents for individual tasks to defining recurring work that agents execute automatically. GitHub Agentic Workflows (gh-aw) enables this approach through Markdown files that combine natural-language instructions with configuration and compile into executable GitHub Actions workflows. Unlike conventional workflows that primarily prescribe scripted operations, these files delegate tasks requiring interpretation to AI agents. They also couple agent instructions with execution triggers, making those instructions operational specifications for repeated repository activities. However, how developers structure and maintain these specifications, and which execution requirements and safeguards they express, remains insufficiently understood. In this paper, we examine the structure, evolution, and instruction content of gh-aw Markdown files to inform how practitioners define and maintain agent-run work. We analyze 1,248 files from 276 repositories, 20,841 commit-file events, and 288 resolved instruction-label sets from a sample of 294 files. Our results show that workflow instructions extend beyond short prompts, with a median of 556.5 words and code blocks in 62.1% of files. Among files with at least 120 days of observed activity, 78.2% still receive updates in month 4, while size-normalized churn decreases after the first month. Tasks, outputs, constraints, and process instructions each appear in over 93% of labeled workflows, yet only 9.4% explicitly address prompt-injection defense. LLM classification achieves an F1 score of 0.818 and Cohen's Kappa of 0.715 against the resolved human labels. These findings suggest that developers should account for the evolution of workflows copied or referenced across repositories and consider adding prompt-injection defenses, resource budgets, and evidencecredibility checks where applicable.

0.34

LLM-Enhanced Commit Message Generation via Issue Information: An Exploratory Study

cs.SE ·2026-08-22 ·Zongen Ren, Wei Shi, Bo Xiong, Chong Wang, Peng Liang
dataset similarity 0.71
Abstract · 28 pages, 7 images, 11 tables, Manuscript submitted to a journal (2026)

Commit messages help developers understand code changes, support collaboration, and improve long-term maintenance. However, the use of issue information alone as the external context for LLM-based CMG has not been systematically studied. We propose an ISsue-Augmented framework for Commit message generation (ISAC) by combining code diffs with issue information as LLM input. To support the evaluation, we construct ApacheCM-Issue, a commit-issue aligned dataset built upon ApacheCM by linking commits with issues from GitHub and Apache Jira. Using samples from Scala, Java, and C++ projects, we evaluate four input configurations using two representative LLMs, GPT-5.5 and DeepSeek-V4-Flash in different reasoning configurations. The results show that incorporating issue information consistently improves LLM-based CMG across all evaluated model configurations and metrics, with the largest gains observed for CIDEr. Incorporating a similar historical commit further improves automatic metric scores, while replacing full issue information with a structured issue summary decreases them. ISAC also outperforms the four reproduced state-of-the-art (SOTA) CMG baselines across all five automatic metrics on the experimental dataset. The human evaluation further shows that structured issue summaries may improve perceived completeness, although replacing the original issue information can sacrifice contextual details and lead to worse results on automatic metrics.

0.26

How Should Agents See Code? Rethinking Code Representations for Agentic Software Engineering

cs.SE cs.AI ·2026-11-05 ·Melika Sepidband, Jiho Shin, Song Wang, Hadi Hemmati
similarity 0.71
Abstract

LLM agents are increasingly used for repository-level software engineering tasks. However, most agents still access repositories through raw files, lexical search, and shell commands, treating code primarily as text. This paper investigates map-based code-representation workflows as a design variable in agentic software engineering, examining how different repository views change agents' retrieval behavior, task success, token use, and latency. In this paper, we study five code-representation workflows: abstract syntax trees, call graphs, module dependencies, documentation-enriched search, and a hybrid workflow that combines them. Each workflow exposes a repository map that the agent can query during a task. We evaluate Claude Code with Haiku 4.5 against a raw-tool baseline on four repository-level tasks: fault localization with LinuxFLBench, repository-level code generation and repair with SWE-bench Pro, test generation with SWT-bench, and security patching with SecBench. The results show that representation workflows are most useful as token-saving tools when localization is the bottleneck, while their success gains narrow when the decisive step is patch or test authoring. On LinuxFLBench, all five workflows improve recall@10 over the baseline (0.757-0.829 vs. 0.717) and reduce token usage by 25-39%. On SWE-bench Pro, resolution remains close to the baseline (44.3-50.6% vs. 49.0%) while token usage drops by 17-26%. On SecBench, Callgraph has the highest patch-resolution rate in this sample (34.3% vs. 28.9%) and uses fewer tokens, but all workflows increase latency. On SWT-bench, Hybrid and Docs have the highest reproducing-test rates in this sample (47.8% and 47.2% vs. 44.7%), but all workflows use more tokens and take longer. Overall, map-based workflows change both what the agent inspects and how much it spends, but their benefits are task-dependent rather than universal.

0.33