CS Paper Radar

Assessing the Construct Validity of Object-Oriented, Class-Level Code Quality Metrics

Hera Arif, Miikka Kuutila, Paul Ralph

cs.SE ·submitted 2026-09-17 13:55 UTC · v1
top venuecode/data

Abstract

Background: Code quality metrics are intended to measure latent properties of software source code. Although numerous code metrics have been proposed and used, their construct validity is rarely evaluated. Thus, the extent to which code metrics actually measure what they claim to measure is often unclear. Aim: Drawing from modern measurement theory, we investigate the construct validity of common class-level, object-oriented code quality metrics by identifying their factor structure using Exploratory Factor Analysis (EFA). The metrics were extracted from the Apache Maven project by three software tools: Designite, JHawk, and Understand. The factor structure was later verified using Confirmatory Factor Analysis (CFA) on 22 randomly selected open source projects meeting a predetermined eligibility criteria. Results: 24 code quality metrics that correspond to six constructs: Cohesion, In-Coupling, Out-Coupling, Size, Sub-Inheritance (related to subclasses), and Sup-Inheritance (related to superclasses) were revealed in the underlying factor structure. Ten metrics did not correspond to any known dimension of software quality and were removed in the EFA. Ten additional metrics exhibited low loadings in the CFA, suggesting their removal from the final measurement model. Size, Cohesion, Inheritance, and Coupling were the constructs retained, with subcategories identified for Inheritance and Coupling. Conclusions: Our results strongly support the construct validity of 24 code quality metrics. Coupling and Inheritance are revealed as multidimensional constructs, since they require measuring two different concepts, revealed as sub-categories in our analysis, and Complexity may be better explored in a multilevel model. Overall, our study demonstrates the value of applying modern measurement theory and latent variable modeling in validating software code quality metrics.

Comment
Accepted to the ESEIW 2026 ESEM - Technical Track

Closest papers

Nearest neighbours by meaning across the whole index.

DCI: Dependency Confidence Index for Assessing Open-Source Dependency Trustworthiness

cs.SE cs.CR ·2026-08-17 ·Clemens Albrecht, Stefan Reitmann
surveycode/datahot direction similarity 0.63
Abstract

Selecting trustworthy open source software dependencies remains a major challenge in software supply chain security. We present the Dependency Confidence Index (DCI), a composite formative index that combines nine empirically weighted trust factors into a single normalized composite score for dependency selection. DCI's trust factors combine insights from a systematic literature review and an exploratory Analytic Hierarchy Process (AHP) survey of ten software developers, highlighting security, source code quality, and project health as the most influential dimensions. Following Goal-Question-Metric methodology, we implemented 12 automated measurements using SonarQube, GitHub APIs, and OpenSSF Scorecard data, deployed in a containerized evaluation platform. We conducted a pilot evaluation of the normalized DCI on 92 popular PyPI packages, observing moderate agreement with OpenSSF Scorecard scores and perfect test--retest reliability. Analysis reveals process-based factors (dependency management, CI) dominate scores on high-quality packages, while security metrics saturate---suggesting DCI's complementary role to existing tools. Our publicly available implementation provides a foundation for open source software trustworthiness research and practical dependency auditing.

0.48

Evaluation of Software Quality Through Refactoring Commit-Impact Analysis

cs.SE ·2026-10-09 ·Md Safayat Hossen, Sheikh Moonwara Anjum Monisha, Shadid Intisar Quasem
top venuecode/data similarity 0.62
Abstract

The aim of this study is to conduct an empirical analysis of the correlation between code modifications and software quality. This will be achieved by examining the effects of changes made before and after each refactoring commit, using software metrics such as size, complexity, code smells, vulnerability, and bug detection. The objective is to conduct static code analysis on open-source software from GitHub repositories using various repository mining and static code analysis tools. Our approach involves formulating research-based questions and utilizing the specified tools to obtain results that will enable us to resolve the case study. The obtained results have been summarised and further analyzed using tables and graphs to achieve the objective.

0.50

Design, development, and preliminary validity and reliability evidence of the Software Engineering Self-Efficacy Scale (SESES)

cs.SE ·2026-09-24 ·Albert D. Ritzhaupt, Jhonathan Sora-Cardenas, Priya Ganapathy Prasad, Kai A. Hackney, Neha Rani +1 more
similarity 0.58
Abstract

The purpose of this research is to design, develop, implement, and provide preliminary validity and reliability evidence of the Software Engineering Self-Efficacy Scale (SESES). Framed by a conceptual framework using guidance in software engineering curriculum and concepts along with the notion of self-efficacy, we generated an initial item pool of n = 87 items to operationalize and measure software engineering self-efficacy among undergraduate computing students. The conceptual framework traces five dimensions: 1) Requirements Engineering, 2) Teamwork and Collaboration, 3) Software Quality Management, 4) Software Design and Architecture, and 5) Software Agile Methodologies. We pilot tested the SESES with n = 527 undergraduate computing students who had completed a software engineering course in the current semester or a previous academic semester. We employed Exploratory Factor Analysis (EFA) with the Principal Axis Factoring method and an oblique (Promax) rotation to examine the underlying structure of the SESES, resulting in the same five internally consistent latent constructs in the conceptual framework with minimal cross-loading and a simple structure in the pattern matrix, explaining approximately 57% of the variability in these data. Our findings suggest that software engineering self-efficacy is a multidimensional construct of five theorized and correlated, yet distinct latent factors. We unpack the limitations and delimitations of the research while exploring undergraduate computing students' software engineering self-efficacy using necessary domain-specific measurements.

0.26

From Metrics to Improvement: A Lifecycle-Aware LLM Feedback Framework for Research Software Quality

cs.SE ·2026-08-24 ·Nafis Tanveer Islam, Nafiseh Soveizi, Yutong Li, Zhiming Zhao
code/data similarity 0.56
Abstract

Research software is increasingly central to scientific workflows, yet it is often developed by researchers with limited software engineering expertise. This can lead to quality issues that hinder maintainability, reproducibility, reuse, and sustainability. Existing static analysis tools can identify such issues, but their outputs often require expert interpretation and provide limited support for translating quality assessments into actionable improvements. To address this gap, we propose a lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement. The framework comprises two stages. First, a lifecycle-aware Quality Model is developed from established software quality standards and practitioner requirements. The model defines five quality dimensions and 25 candidate metrics, of which 14 are operationalized using existing analysis tools and custom measurements. Second, the resulting quality diagnostics are used as structured feedback within an iterative LLM-based refinement process, enabling generated improvements to be repeatedly reassessed against the Quality Model. We evaluate the framework on notebook-centric research software using multiple LLMs and compare iterative structured feedback with single-step feedback and unstructured prompting. The results show improvements in specific quality attributes, particularly code duplication and structural quality, while also revealing trade-offs among maintainability, code size, documentation, and complexity. These findings demonstrate the potential of metric-driven LLM feedback for research software quality improvement while highlighting its inherently multi-objective nature \footnote{The source code and experimental data are publicly available at https://github.com/QCDIS/Software_Quality_Control_LLM . }

0.20

PreMaQ: Predicting Maintainability-Related Quality of LLM-Generated Code Before Generation

cs.SE ·2026-10-05 ·Taiki Wakamatsu, Yuta Ishimoto, Masanari Kondo, Yasutaka Kamei
benchmark similarity 0.56
Abstract · 10 pages, 4 figures

As large language models (LLMs) become increasingly capable of code generation, adopting generated code in software development requires assessing not only its functional correctness but also its maintainability-related quality. If such quality could be estimated before generation, developers could avoid the cost of generating, reviewing, and discarding low-quality code. Although prior work has shown that the functional correctness of the LLM-generated code can be predicted in advance, it remains unclear whether maintainability-related quality is similarly predictable. We introduce Pre-Generation Maintainability-Related Quality Prediction (PreMaQ), which predicts the Code Smell Score (CSS) and Maintainability Index (MI) of generated code from the internal representations of LLMs before generation. Our evaluation covers four open-weight LLMs and four Python code generation benchmarks, comprising 2,695 tasks in total. Our results show that predicted CSS and MI consistently correlate with their observed values across all 16 model-benchmark combinations, achieving mean Spearman rank correlations of 0.57 and 0.65, respectively. When used for model selection, PreMaQ achieves 59.5% of the maintainability-related quality improvement attainable by an ideal maintainability-based selector over random selection on tasks for which multiple models generate functionally correct code. Combining predictions from PreMaQ and prompt embeddings increases this proportion to 60.7%, indicating that the two signals are complementary. These findings suggest that PreMaQ can be used to predict and improve the maintainability-related quality of LLM-generated code.

0.20

What Is Under the Hood? A Cross-Ecosystem Study of Vulnerable Dependencies in Construction Software

cs.SE ·2026-10-19 ·Mahzabin Tamanna, S M Jamil Uddin
code/data similarity 0.55
Abstract

The construction industry increasingly relies on software built from third-party components, yet the dependency composition and known-vulnerability exposure of construction-related software remain poorly analyzed. In this study, we conduct an empirical, cross-ecosystem investigation of 904 construction-related GitHub repositories spanning six construction functions and a multifunction category. Using commit-pinned repository snapshots and OSV-Scanner, we examine dependency assessability, known-vulnerability prevalence, severity, weakness classifications, and shared upstream exposure. Our results reveal that only 494 repositories (54.6%) completed vulnerability assessment, revealing substantial gaps in dependency visibility. Among successfully scanned repositories, 395 (80.0%) contained at least one dependency version associated with a published vulnerability advisory, and 97.2% of affected repositories contained a High or Critical severity risk. Additionally, Common Weakness Enumeration (CWE) mapping identified recurring resource-consumption, path-traversal, information-exposure, and code-injection weaknesses. Vulnerability burden was uneven, with a median of 50 distinct advisory clusters per affected repository, while just two affected upstream package families collectively reached ~70% of affected repositories. These findings establish an empirical baseline for dependency assessability and known-vulnerability exposure in construction-related open-source software. Our findings underscore the importance of documenting assessment coverage, interpreting advisory matches in their deployment context, and examining shared upstream dependencies when evaluating construction software supply-chain security.

0.20

Evaluating Code Recommender Systems: A Review

cs.SE ·2026-09-24 ·Daniel Borst, Stefan Sobernig
survey similarity 0.55
Abstract

Context: Code recommender systems (CRSs) are specialized software systems operating on source artifacts to provide automatically generated recommendations to software developers in all phases of development. The goal is to improve software quality while enhancing developers' efficiency, effectiveness, and experience. Problem: Despite the growing importance, little is known about the state of evaluating these systems in a human-centric manner. Research Approach: We conducted a systematic literature review to identify and to synthesize primary studies evaluating CRSs (2017-2024). Ninety-two publications were included and subjected to a systematic content analysis. Results: Our study confirms that Offline Evaluations are the most common, system-centric evaluation type, whereas user-centric evaluations (Online Evaluations, User Studies) are rarely reported. Evaluations are concentrated on the Software Construction phase. Most studies explicitly report threats to validity, with External threats to study Materials being the most common. Conclusion: The state of evaluations on CRSs is narrowly focused on single or a few system qualities. There is an imbalance between a majority of system-centric evaluations and a minority of user-centric evaluations. Combined, multi-type evaluations are rarely reported.

0.21

Does AI-Assisted Development Homogenize Software? An Empirical Study of Code Similarity, Quality, and Evolution

cs.SE cs.AI ·2026-10-16 ·Luciano Marchezan, Romain Lefeuvre, Kevin Delcourt, Benoit Combemale, Houari Sahraoui
code/data similarity 0.54
Abstract

AI-assisted software development is evolving from interactive LLM assistance toward agentic systems that autonomously perform software engineering tasks, yet the effects of this shift remain insufficiently understood. We present an empirical study of 50 long-lived Java and Python open-source projects across three eras: pre-LLM, potentially LLM-assisted, and detected agentic development. Using snapshots from Software Heritage, we examine code similarity, software quality, and software evolution. Similarity is measured semantically through embedding-based cosine similarity and syntactically through CodeBLEU*. Quality is assessed with SonarQube metrics covering maintainability, complexity, reliability, and security. Evolution is characterized by annualized growth rates of LOC, functions, classes, files, and statements, and by commit activity in the three months preceding each snapshot. Results show no consistent evidence of increased similarity as semantic and syntactic similarity decreased significantly in Java for both the LLM and agentic eras relative to the pre-LLM era, while Python showed no significant differences. Software quality remained largely stable, although the agentic era showed significantly lower code-smell and SQALE index densities. Structural growth was generally lower in the agentic era, particularly for files, whereas development activity increased moderately in Java, especially in commits and added lines. These findings challenge the assumption that LLM- and agent-assisted development necessarily homogenizes code, degrades quality, or accelerates growth.

0.27