CS Paper Radar

Automatic Assistance to Reduce Information Overload during Software Issue Report Understanding

Salma Begum Tamanna, Gias Uddin, Junaed Younus Khan, Lan Xia, Longyu Zhang

cs.SE ·submitted 2026-11-04 20:24 UTC · v1

Abstract

Modern issue tracking systems are used to log an issue and track it from its inception to closure. Issue reports with diverse discussions from stakeholders can be large and verbose, making them challenging to analyze quickly. Our developer interviews revealed that they struggled with the diverse information in issue reports and demanded automated techniques to classify and summarize information. We designed a tool, iSum (issue summarizer), that helps with the automatic classification and summarization of information types discussed in issue reports. The summarizer offers two views: 1. Developer View takes an issue report as input and outputs visualized summaries of information in the report. 2. Manager View takes a suite of all issue reports under a given software project as inputs and shows the prevalence and trends of different information types across the reports. We evaluated iSum through three studies. 1. In an empirical study over two benchmarks, iSum offers, on average, 11-13% higher F1-scores than state-of-the-art baselines to detect information types. 2. We conducted a user study in task-based setup with 30 developers to determine iSum's effectiveness in issue exploration scenarios. Results showed increased productivity and accuracy with iSum compared to manual inspection. 3. We assessed iSum's feasibility for development and management tasks. It proved useful to both developers and project managers for issue comprehension and trend monitoring.

Closest papers

Nearest neighbours by meaning across the whole index.

LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling

cs.SE ·2026-09-03 ·Liam Johnston, Shayan Noei, Maram Assi, Ying Zou
hot direction similarity 0.67
Abstract · 38 pages, 11 figures

Software users often submit issue reports to a product's issue tracking system to report defects, suggest enhancements, or raise other product-related concerns. Labeling these issue reports supports effective planning and improves community engagement. However, many issue reports remain unlabeled due to the substantial manual effort required to design an appropriate label taxonomy, then assign suitable labels from this taxonomy to new issue reports. Existing automated labeling approaches attempt to mitigate these challenges. However, they suffer from key limitations, such as extensive manual intervention, the assignment of generic labels, and a dependence on existing labeled datasets. To address these limitations, we propose LabelMate, a novel Large Language Model (LLM)-driven framework that (1) derives a comprehensive, project-specific label set from historical issue reports and (2) automatically assigns relevant labels to new issue reports without requiring any pre-labeled training data. We evaluate LabelMate on 16,500 issue reports from 30 popular and diverse GitHub repositories. Based on this dataset, our approach generates a coherent list of 275 labels and achieves an average labeling accuracy of 89.84%, a statistically significant improvement over existing generic label assigning approaches. These results demonstrate that LabelMate offers an efficient, domain-adaptive solution to streamline the issue labeling process.

0.34

An Empirical Study on Improving LLM-Based Issue Classification through Domain-Informed Analysis

cs.SE ·2026-11-10 ·Anubhav Gupta, Tanner Wright, Gema Rodríguez-Pérez
similarity 0.64
Abstract · 12 pages, 2 figures

Issue reports that describe Intrinsic Bugs, Extrinsic Bugs, and Not a Bug cases often use similar language, but they might point to different causes and require different maintenance actions. This makes three-way issue classification difficult for Large Language Models (LLMs), particularly when prompt-based refinements begin to plateau. We study whether domain-informed analysis of LLM outputs can improve classification beyond prompt modification alone. Using 428 manually labeled issue reports, we evaluate six LLMs across 18 prompt configurations and analyze 7,704 model-generated rationales to identify recurring failure patterns in how models justify predictions, use evidence, and handle conflicting information. We translate these findings into a multi-stage pipeline that separates bug validity from bug origin. The pipeline improves macro F1 and Extrinsic Bug recall for five of the six models. The strongest pipeline configuration reaches 0.827 accuracy and 0.757 Extrinsic Bug recall, improving over the best-accuracy prompt-only baseline by +1.2 and +7.5 pp, respectively. The decomposition further reveals that bug origin classification is more reliable than validity, indicating that the bug-validity decision accounts for most of the remaining error. These results show that expert failure analysis can support more reliable LLM-based software engineering classifiers.

0.32

IssueLighter: An Issue Type-Aware LLM Agent for Issue Report Validity Assessment

cs.SE cs.AI ·2026-11-02 ·Jiawei Ran, Yuxia Zhang, Ziyang Zhao, Hui Liu
top venuecode/data similarity 0.62
Abstract · Accepted to ICSE 2027. 12 pages, 2 figures

Invalid issue reports are prevalent in open source software (OSS) projects, imposing substantial burdens on maintainers who already face a high risk of burnout. Prior research has primarily focused on identifying invalid reports for specific issue categories (e.g., bug and wontfix), leaving the broader landscape of OSS issue types largely unexplored and limiting the generalizability of existing approaches. Moreover, existing methods provide little support for explaining or validating their predictions, reducing their practical usefulness for maintainers. With the rise of LLM-based agents, it remains unclear whether a comprehensive understanding of the causes of invalidity across all issue types can improve validity assessment performance while producing well-grounded rationales. To this end, we first manually annotated the validity status of 520 issue reports from three OSS projects, i.e., pandas, scikit-learn, and mypy, resulting in 257 invalid reports and 263 valid ones. Then, we conducted a thematic analysis on these invalid reports and identified 13 causes of invalidity that are closely specific to issue types. Based on these insights, we propose IssueLighter, an agent-based framework with a type-aware validity assessment strategy that dynamically executes commands. Experimental results show that IssueLighter consistently and substantially outperforms all baseline methods, with an average F1 improvement of over 20 percentage points. Ablation studies confirm that both the type-aware strategy and the taxonomy of causes of invalidity contribute to the performance gains. Qualitative analysis further demonstrates that IssueLighter generates rationales grounded in accurately identified causes of invalidity and supported by evidence. Moreover, leveraging these rationales, we challenged maintainers' original decisions in four disputed cases, and all received approval.

0.53

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

cs.SE ·2026-08-21 ·Jacek Dąbrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari, Mohammad Amin Zadenoori, Yijun Yu
code/data similarity 0.62
Abstract

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

0.48

Noisy Issues, Noisy Results: Mislabeling in Information Retrieval-based Bug Localization

cs.SE ·2026-11-03 ·Moumita Asad, Rafed Muhammad Yasir, Yirui He, Ziyao He, Sam Malek
benchmarkcode/data similarity 0.61
Abstract

Information Retrieval-based Bug Localization (IRBL) leverages bug reports to identify faulty source files. Evaluations of IRBL approaches primarily rely on labeled issues from issue tracking systems. However, prior research has shown that issue labels are frequently incorrect in open-source projects, threatening the validity of IRBL evaluations. Existing studies have primarily examined traditional IRBL techniques on small datasets, overlooking modern approaches and the partial overlap between noisy and cleaned datasets. In this paper, we conduct a large-scale empirical study to investigate the prevalence and impact of mislabeled issues in IRBL benchmarks. We analyze five benchmark datasets comprising over 55,000 issue reports and find that, on average, 34.82% of issues are mislabeled. We further evaluate six IRBL techniques, spanning traditional, deep learning-based, and large language model-based approaches using the widely adopted Ye et al. benchmark and statistical methods that account for dataset overlap. Our results show that removing mislabeled issues from evaluation datasets significantly reduces measured performance across all six techniques, with Mean Reciprocal Rank decreasing by 4.68%-21.74%. Although mislabeling does not change the relative ranking of techniques, it affects the magnitude of their performance differences. These findings highlight the importance of reliable evaluation data for ensuring valid and meaningful comparisons of IRBL techniques.

0.29

Context Matters: Improving LLM-Based Code Summarization with Hierarchical Information

cs.SE cs.AIcs.LG ·2026-11-13 ·Alexandre Petrachini, Breno Bernard Nicolau de França, Guido Araujo
published similarity 0.60
Abstract · 11 pages, 4 figures, 10 tables. Accepted and presented at the Brazilian Symposium on Software Quality (SBQS 2026), Technical Track. Replicat…

Source code documentation plays an important role in software quality activities, such as code review, program comprehension, maintenance, and bug fixing. However, it is often fragmented across individual code elements and focuses on local implementation details, providing little information about their surrounding context. This limits both developers' understanding of software systems and the effectiveness of Large Language Models (LLMs), whose reasoning is constrained by finite context windows. Enriching documentation with hierarchical context has the potential to improve software comprehension and support downstream, quality-related tasks for both developers and LLM-based tools. In this paper, we present an in silico study that showcases the impact of augmenting context on the generation of code-level documentation, using a variety of LLMs. By varying the function, class, and module hierarchy visible to the LLMs, we assess whether this variation produces a significant difference in the resulting code documentation compared with the existing documentation, which serves as a proxy for developer intent. Our results show that Documentation-Aware prompts with class or module-level context consistently yield summaries that are closest to the original documentation. Adding surrounding documentation yields statistically significant improvements over the baseline across models, with effect sizes ranging from negligible (DeepSeek) through weak (GPT-4o-mini) to medium (Gemini). We restrict the study to this immediate hierarchical context; call-graph traversal and cross-module dependency context fall outside our scope and are left to future work. These findings provide a clear path to improve automated documentation tools and enhance LLMs' ability to reason about codebases for downstream tasks, such as code retrieval.

0.39

How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community

cs.SE ·2026-09-04 ·Panida Rumriankit, Akito Monden, Hiroki Inayoshi, Pattara Leelaprute, Bundit Manaskasemsak +2 more
publishedcode/data similarity 0.60
Abstract · 6 pages, 2 figures, 4 tables. Accepted at 34th IEEE/ACIS SNPD 2026

Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development, yet little is known about how mainstream open source communities discuss them in practice. This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository, using 43,806 candidate issues created between January 2021 and June 2026. To improve corpus quality, we combined keyword retrieval with semantic relevance filtering, yielding a filtered corpus of 25,227 AI-related issues. We applied BERTopic to the retrieved corpus to identify discussion topics, using the filtered corpus for theme validation and a robustness re-clustering, and analyzed their evolution over time using monthly prevalence and Mann-Kendall trend tests. The results show that developer discussions are dominated by practical concerns regarding the operation of AI-assisted development environments, including agent management, configuration, reliability, authentication, and billing, whereas risks frequently emphasized in survey-based studies, such as hallucination and licensing, rarely surface in this venue. This suggests that discussions of generative AI in the VS Code issue tracker primarily focus on operational aspects of AI-assisted software development. Furthermore, discussions evolved from AI-assisted code completion toward conversational and agent-based development, reflecting the increasing integration of generative AI into software development workflows. These findings suggest that GitHub Issues provide a practical, workflow-oriented perspective on generative AI that complements survey-based studies of developer perceptions.

0.38

ComBeautify: Leveraging Code Change Summaries for Conventional Commit Generation

cs.SE cs.AI ·2026-10-25 ·Rishi Raj, Aashray Tandon, Nikolay Matyunin, Jibesh Patra
code/data similarity 0.60
Abstract

Modern software development relies on version control systems such as Git to track changes and facilitate collaboration. While commit messages are crucial for conveying developer intent, they do not affect program functionality; as a result, developers often lack motivation to write meaningful ones. The Conventional Commits Specification (CCS) addresses this with a structured format and guidelines that help developers express the intent of a change, improving clarity and enabling automation. Yet, developers frequently struggle to express intent while adhering to the format. In particular, they may misclassify commits due to the large number of possible categories or the lack of project-specific guidance. We introduce ComBeautify, an approach that automatically generates CCS-compliant commit messages. Our hybrid approach leverages context-preserving summarization of Git diffs with large language models (LLMs) to improve commit classification accuracy under CCS. Rather than relying on manually curated category definitions, we learn them from historical commits. We also propose a method for automatically generating the CCS description component. Our empirical evaluation across 13 languages, 6,500 open-source repositories, and over 20 million commits provides long-term insights into CCS adoption, revealing increasing compliance. Our evaluation on twenty popular repositories reveals the limitations of approaches guided by generic category definitions, and shows that ComBeautify, when trained on project-specific data, outperforms baselines by up to 3x. Additionally, using a dataset of 2,000 manually curated commits, we validate our summarization approach, achieving up to 95.25% accuracy and surpassing baselines. These findings emphasize the need for adaptable, context-aware AI methods for commit classification and CCS-compliant commit message generation.

0.25