CS Paper Radar

Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking

Yongjian Tang, Linhan Li, Thomas Runkler

cs.SE cs.AI ·submitted 2026-10-07 11:23 UTC · v1
benchmarkdatasetcode/data

Abstract

Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaboration, under-utilizing the full potential of multi-agent systems. Meanwhile, available datasets focus on isolated subtasks, such as requirements extraction, classification, and completeness detection, leaving the absence of an end-to-end RE benchmark that spans from requirements elicitation to generation. We present Agent4RE - a self-refining multi-agent RE system that orchestrates specialized agents and incorporates two iterative improvement loops. To support evaluation, we construct RE-E2E - a real-world dataset built from human-written requirement specifications, enabling end-to-end assessment of RE workflows. Building on this foundation, we further propose two enhanced Agent4RE versions that incorporate either autonomous self-refinement or structured human feedback, and analyze their strengths and limitations across different scenarios. Evaluation on 8 Large Language Models (LLMs) demonstrates that all three Agent4RE variants consistently outperform a domain-context-augmented prompting baseline by average 8% in text-based metrics. The two enhanced variants achieve the highest LLM-as-a-judge and human ratings, surpassing two RE baselines by approximately 0.8 points on a four-point scale. This consistent performance establishes Agent4RE as a practical end-to-end RE solution for industrial environments.

Comment
Accepted to the ASE@POVC track; The E2E requirements engineering benchmark is available https://zenodo.org/records/22815961
DOI
10.1145/3843779.3844634

Closest papers

Nearest neighbours by meaning across the whole index.

WiseSpec: Requirements-Driven Agents for Code Generation

cs.SE cs.AI ·2026-09-01 ·Zhao Tian
top venue similarity 0.79
Abstract · Accepted by ASE 2026 (SRC)

Code generation aims to automatically generate source code from task requirements and has attracted significant attention with the rapid advancement of large language models (LLMs). Despite remarkable progress, LLMs often struggle to generate correct code for complex software engineering tasks because task descriptions are frequently incomplete, ambiguous, or lack critical contextual information. Existing approaches primarily improve the capabilities of coding agents through more sophisticated tools, skills, and workflows, while largely overlooking the quality of the task requirements themselves. To address this limitation, we draw inspiration from software requirements engineering and propose WiseSpec, a novel requirements-driven agent framework for repository-level code generation. WiseSpec automatically constructs structured and information-rich requirements, assesses their quality through execution-based evaluation, and iteratively refines them to better guide code generation. Experimental results show that WiseSpec consistently outperforms all baselines, achieving an average improvement of 13.17% in %Resolved.

0.36

The Impact of GenAI on the Future of Requirements Engineering

cs.SE ·2026-09-04 ·Travis Breaux, Anmol Singhal
top venuesurvey similarity 0.77
Abstract

Recent advances in artificial intelligence (AI), particularly large language models (LLMs), are transforming how we design and build systems by increasing access to domain knowledge and by providing automation support to software engineering (SE). As implementation becomes less expensive through generalist SE agents, engineering effort shifts away from writing correct code and toward expressing, curating, verifying, and evaluating requirements. In this paper, we survey the state of the art in AI for requirements engineering (RE) research leading up to the transformation, before reviewing advances in LLMs. We survey two subsequent research areas: prompt programming, which treats LLM instructions as a program in SE vernacular, and generalist SE agents, which combine multiple LLM advances to yield semi-autonomous processes that complete SE tasks. Finally, we explore the future of requirements engineering along two axes: matters changing how we interact with requirements through the SE process, and matters changing how requirements are experienced by software developers and stakeholders more broadly, including end-users. This article aims to inform how RE researchers can navigate this transformation in the selection of future research priorities.

0.27

DRILS: Dependency-based Requirement Change Impact Analysis Using LLMs

cs.SE ·2026-10-24 ·Romina Etezadi, Ikram Darif, Lionel Briand, Ramesh S, Parkish Peranandam
similarity 0.75
Abstract · 40 pages, 8 figures

Change is inevitable in software systems, where requirements must continuously adapt to evolving stakeholder needs, regulations, and project scopes. Managing such changes is a fundamental software engineering challenge, as manually identifying the impact of a change request, given high requirements volume, Natural Language (NL) complexity, and intricate interdependencies, is error-prone and time-consuming. Large Language Models (LLMs) offer promising solutions for requirements change impact analysis (CIA), yet their effectiveness when enhanced with advanced techniques remains underexplored. We present DRILS, an LLM-based agentic framework for requirements Change Impact Analysis, employing a multi-agent architecture to generate candidate impact sets, with an LLM-as-a-judge promoting diversity during selection. DRILS features two key components: RE-RAG, a carefully engineered RAG technique providing focused domain context, and a requirement dependency graph (RDG) as an additional knowledge source. Evaluation across three manually curated datasets demonstrates superior performance over state-of-the-art baselines, achieving an average recall of 87.1%, precision of 76.3%, and F1 score of 81% at a lower cost. Precision gains are particularly significant, 41.1%, 83.4%, and 149.8% over the closest baseline per dataset, as higher precision directly reduces false positives and manual validation effort. We also contribute three annotated CIA datasets to support future research.

0.11

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

cs.SE ·2026-08-21 ·Jacek Dąbrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari, Mohammad Amin Zadenoori, Yijun Yu
code/data similarity 0.74
Abstract

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

0.48

AREAs-Lab: An Interactive Environment for AI-driven Requirement Elicitation for AI Systems

cs.HC ·2026-08-29 ·Pengshan Cai, Zihao Zhang, Ting Jin, Chenyang Zhu, Kushal Chawla +6 more
top venuebenchmark similarity 0.74
Abstract · Accepted to Findings of EMNLP 2026. 45 pages, 13 figures

Building effective AI systems increasingly depends on writing high-quality task requirements, yet users often struggle to articulate the constraints, preferences, and edge cases that determine success. This problem is especially acute in AI development, where behavior is shaped not only by human expectations but also by data characteristics. We present AREAs-Lab, an interactive environment for AI-driven Requirement Elicitation for AI systems. In AREAs-Lab, an assistant iteratively refines an initially incomplete requirement by analyzing the underlying dataset and asking targeted clarification questions to uncover the user's latent intent. To study this setting systematically, we construct a synthetic benchmark grounded in 16 public datasets spanning diverse domains and task types. Each benchmark instance includes a user profile, a complete reference requirement, and an intentionally underspecified version that serves as the assistant's starting point. We further introduce an automated evaluation pipeline based on an AI-simulated user that reveals hidden information only when appropriately prompted, enabling scalable and reproducible assessment of interactive elicitation quality. AREAs-Lab provides a controlled testbed for studying how AI assistants can transform vague user goals into actionable requirements for AI systems.

0.30

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

cs.AI ·2026-08-18 ·Liya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang +33 more
benchmarkhighly cited similarity 0.73
Abstract

Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce \textbf{StartupBench}, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30\% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.

0.38

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

cs.SE cs.AIcs.CL ·2026-10-29 ·Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao +1 more
benchmark similarity 0.73
Abstract

Large language models (LLMs) have shown strong performance on software engineering (SE) tasks, driving the need for high-quality datasets for reliable training and evaluation. SWE-bench-style datasets (SWE datasets) are widely used to evaluate LLM-based coding agents on real-world software issues, where task clarity and test coverage directly affect whether instances are solvable and verifiable. However, quality issues persist in existing SWE datasets, and manual annotation is costly and labor-intensive, motivating LLM-based annotation. Existing approaches typically rely on static, overly general rubrics, limiting their ability to handle diverse and nuanced scenarios and often yielding high false positive rates. In contrast, human annotators draw on accumulated experience to make context-aware judgments. Inspired by this, we propose ExLabel, an LLM-based, self-evolving, experience-guided assistant for task clarity and test coverage assessment in SWE datasets. ExLabel maintains an evolving experience bank that distills cross-instance knowledge into stable yet adaptive judgment criteria and reuses relevant experience to guide the assessment of new instances. For systematic evaluation, we construct SWE-TQ, a new benchmark spanning eight programming languages, by cleaning human annotations from SWE-bench and Multi-SWE-bench. Experiments with three backbone LLMs show that ExLabel achieves the best F1 across all settings, with relative improvements of 44.2%/49.9% and 24.0%/18.4% over the best baseline for task clarity and test coverage assessment on the Python/Multilingual subsets, respectively. Moreover, a coding model fine-tuned on SWE-rebench data filtered by ExLabel resolves 66.2% of issues on SWE-bench Verified, outperforming the model trained on data filtered by the best baseline by 14.5%.

0.13

CHATDIFY: Automating End-to-End Dify Workflow Synthesis from Natural Language via Multi-Agent Collaboration

cs.AI ·2026-11-01 ·Tao Fan, Guoqiang Ma
benchmark similarity 0.73
Abstract

The proliferation of Large Language Models has catalyzed powerful application development platforms like Dify, which enable users to build sophisticated LLM-powered workflows through modular node orchestration. However, manually configuring these workflows requires deep platform expertise and remains time-consuming and error-prone, creating a significant barrier for non-expert users. To address this challenge, we propose \textsc{ChatDify}, a novel multi-agent framework that automates end-to-end Dify workflow synthesis from natural language descriptions. \textsc{ChatDify} employs an Actor-Critic architecture comprising three specialized agents: the \textit{Planner Agent} that decomposes user requests into structured execution plans, the \textit{Executor Agent} that performs context-aware parameter synthesis and schema-augmented DSL compilation, and the \textit{Reviewer Agent} that ensures reliability through hybrid verification with closed-loop refinement. We evaluate \textsc{ChatDify} on a curated benchmark of 100 production-inspired tasks across three state-of-the-art LLM backends, achieving 0.96, 0.91, and 0.75 success rates with Qwen-3.7, GLM-5.1, and DeepSeek-v4, respectively. These results demonstrate its robustness and practical applicability for lowering the technical barrier to creating sophisticated LLM applications.

0.10