CS Paper Radar

On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study

Marius Mignard, Steven Costiou, Anne Etien

cs.SE cs.LG ·submitted 2026-09-08 12:52 UTC · v1

Abstract

Context: Computational notebooks are the standard environment for machine learning (ML) development. Within the ML community, model performance is often the primary considered metric, and code quality is treated as a secondary concern. This prioritization relies on a largely untested assumption that code quality and ML performance are unrelated. Practitioners also reuse existing code that may come from notebooks selected through social signals (popularity, author expertise) whose reliability as quality proxies has never been assessed. Objective: We empirically investigated the relationship between code quality and ML performance in notebooks, and evaluated whether popularity and author expertise give indication on code quality or performance. Method: We conducted a large-scale empirical study of 265,363 Python notebooks submitted to Kaggle competitions. We assessed code quality with two static analysis tools: Pylint, capturing general Python code quality, and SonarQube, configured with a profile of 34 rules targeting data-science and ML-specific practices. Results: The relationship between code quality and performance depends on the notion of quality considered. General Python code quality is decoupled from ML performance, showing negligible or non-significant correlations across all observations. In contrast, ML-specific violations exhibit a consistent, small negative association with performance that persists across all observations. The popularity of a notebook does not give information on the code quality or performance. Code expertise provides no information on quality or performance, but competition expertise correlates with better performance, fewer ML-specific violations, and slightly more Python errors and refactoring violations.

Closest papers

Nearest neighbours by meaning across the whole index.

Using machine learning approaches to classify false-positives generated by static analysis tools - An empirical study using small datasets

cs.SE cs.LG ·2026-11-14 ·Lakmal Deshapriya, Sherlock A. Licorish, Brendon J. Woodford
similarity 0.66
Abstract

Static Code Analysis (SCA) tools are vital for software development to reduce the cost and time required for manual code reviews. It was established in our previous study that the best tools reported in the software engineering community return high false-positive and false-negative rates when used for code analysis. Several approaches have been taken in existing studies to identify false-positive alarms generated by SCA tools, in helping to enhance the utility of these tools. However, existing studies reveal several limitations, chiefly among which is the tendency to focus on a limited number of alarm types or failing to evaluate a large breadth of Machine-Learning (ML) techniques comprehensively. This study has addressed this opportunity. We identified the best Code Representation Learning (CRL) techniques and ML algorithms for identifying false-positives, where results show that performance outcomes vary based on the characteristics of the training datasets and the goal of the model building. Given the evidence established in this study, we recommend further investigations targeting specific configurations of advanced CRLs and ML Algorithms, including simple ML models.

0.22

Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code

cs.SE ·2026-10-13 ·Jue Huang, Tarek Mahmud, Meiru Che, Guowei Yang
publishedbenchmark similarity 0.65
Abstract · Accepted for publication in ICECCS2026

Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that code is maintainable, reliable, or well-structured. This paper {presents a large-scale static quality profiling of} LLM-generated Python code, with a focus on maintainability and reliability. Using 65{,}663 programs from 73 models in LiveCodeBench, we evaluate generated code with established static analysis tools, including Pylint, Radon, and SonarQube. Our results show systematic differences across models that are not captured by correctness alone, with maintainability emerging as the clearest source of variation. We further find that correctness often comes with structural trade-offs rather than uniformly simpler solutions. Overall, our findings motivate evaluation frameworks that assess LLM-generated code beyond pass rates alone.

0.25

PreMaQ: Predicting Maintainability-Related Quality of LLM-Generated Code Before Generation

cs.SE ·2026-10-05 ·Taiki Wakamatsu, Yuta Ishimoto, Masanari Kondo, Yasutaka Kamei
benchmark similarity 0.64
Abstract · 10 pages, 4 figures

As large language models (LLMs) become increasingly capable of code generation, adopting generated code in software development requires assessing not only its functional correctness but also its maintainability-related quality. If such quality could be estimated before generation, developers could avoid the cost of generating, reviewing, and discarding low-quality code. Although prior work has shown that the functional correctness of the LLM-generated code can be predicted in advance, it remains unclear whether maintainability-related quality is similarly predictable. We introduce Pre-Generation Maintainability-Related Quality Prediction (PreMaQ), which predicts the Code Smell Score (CSS) and Maintainability Index (MI) of generated code from the internal representations of LLMs before generation. Our evaluation covers four open-weight LLMs and four Python code generation benchmarks, comprising 2,695 tasks in total. Our results show that predicted CSS and MI consistently correlate with their observed values across all 16 model-benchmark combinations, achieving mean Spearman rank correlations of 0.57 and 0.65, respectively. When used for model selection, PreMaQ achieves 59.5% of the maintainability-related quality improvement attainable by an ideal maintainability-based selector over random selection on tasks for which multiple models generate functionally correct code. Combining predictions from PreMaQ and prompt embeddings increases this proportion to 60.7%, indicating that the two signals are complementary. These findings suggest that PreMaQ can be used to predict and improve the maintainability-related quality of LLM-generated code.

0.20

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

cs.SE cs.AI ·2026-09-11 ·Cristina Improta, Pietro Liguori, Domenico Cotroneo
benchmarkcode/data similarity 0.62
Abstract · Preprint. This manuscript is currently under peer review

AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whether their code differs from human code in the quality dimensions dominating lifecycle cost. We compare human-written and AI-generated code at scale: 787,562 function pairs across Python, Java, and C, each human function mined from open-source repositories paired with implementations generated from its docstring by three AI assistants (OpenAI GPT models, DeepSeek-Coder, Qwen2.5-Coder). We characterize structural complexity and statistical naturalness, and map static-analysis findings onto Orthogonal Defect Classification for defects and the Common Weakness Enumeration for vulnerabilities, making authors and languages directly comparable. AI-generated code is structurally compressed and stylistically templated: roughly half the size and branching of human code, clustering apart at the style level. Defect profiles differ in kind: human code concentrates issues of mature codebases, AI code repetitive boilerplate; security is language-dependent, with LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C. Once size is controlled for, complexity metrics carry little signal, while naturalness separates authors. Finally, we release CQBench, a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline for quality assurance and security testing.

0.55

A Comprehensive Study of Native Code Bugs in Python Applications

cs.SE cs.CR ·2026-08-30 ·Haoran Yang, Haipeng Cai
similarity 0.61
Abstract

The impact of Python applications has been evidenced by their widespread presence in some of the most impactful software domains, such as machine learning frameworks and scientific computing platforms. These applications often integrate native code components written in a lower-level programming language like C. This multilingual construction brings various benefits such as greater performance efficiency and easier interoperability with diverse runtime environments. However, bugs in the native code (i.e., native code bugs), which are usually stealthy, also constitute a major additional challenge to the quality of the Python applications as a whole. Yet despite existing relevant studies, there remains a lack of comprehensive understanding of native code bugs in Python applications. In this paper, we aim to mitigate this knowledge gap through the first in-depth study of such bugs, dissecting their common symptoms, introducing locations, manifestation characteristics, root causes, and fixes. Based on our extensive automated and manual analyses of 216 native code bugs in real-world Python projects on GitHub, we obtained novel findings about and new insights into the occurrence mechanisms and resolution strategies of those bugs.

0.09

Characterizing Feedback Statements in Machine Learning Jupyter Notebooks

cs.SE ·2026-09-19 ·Arumoy Shome, Lu\is Cruz, Diomidis Spinellis, Arie van Deursen
top venue similarity 0.61
Abstract

Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use it to decide what to do next. We call these feedback statements and identify two forms: exploratory statements that display values for visual inspection, and validation statements that enforce conditions programmatically through assertions. Many ML failures do not surface as exceptions and thus escape the crash-based analyses that dominate prior work on ML notebooks. This study examines what practitioners check to catch failures that would otherwise pass silently, by characterizing feedback statements that encode the practitioner's mental model of what the code should do and what could go wrong. We mine 297,851 public Python Jupyter notebooks from GitHub and Kaggle and extract 1,092,780 feedback statements. We sample 816 statements through proportional stratified sampling from semantic clusters obtained from CodeBERT embeddings, and apply grounded theory and open coding to label and analyze each one. We contribute a taxonomy of feedback statements in ML notebooks, organized along the functional intent of the statement and the ML pipeline stage in which it appears. The taxonomy reveals that feedback is overwhelmingly exploratory, and that the two platforms host qualitatively different modes of ML work. Mapping our taxonomy to an existing crash taxonomy shows that it captures defensive practices against silent failures that crash analysis cannot observe. Our findings indicate that notebook source should be treated as a confounder in studies of ML developer practice, surface opportunities for notebook tooling, and motivate empirical study of silent ML failures. We release the corpus of 1,092,780 feedback statements and the codebook to support replication and tooling research.

0.45

PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns

cs.SE cs.LG ·2026-08-13 ·Syrym Abdikhan, Mazhar Hameed
datasetcode/data similarity 0.60
Abstract

Pandas has emerged as the de facto library for data processing and machine learning, widely used for tasks, such as data loading, transformation, and analysis. Despite its ubiquity, there has been limited systematic investigation into how Pandas is used in real-world projects and how typical workflows are composed in practice. To address this gap, we introduce PandasCorpus, a dataset curated from GitHub repositories that captures real-world Pandas workflows at scale. In this work, a workflow refers to Pandas-based code contained in Jupyter notebooks, a prevalent medium for writing, executing, and sharing data analysis code. The dataset comprises 139k notebooks from approximately 100k repositories and captures more than 4M Pandas API calls spanning 136 distinct operations. Beyond dataset construction, we characterize workflows using structural and Pandas-specific features and analyze notebook evolution between 2015 and 2025. Our study examines code executability, notebook size, and recurring sequences of Pandas operations, providing empirical insights into how Pandas is used in practice. The resulting corpus offers a reusable resource for studying data analysis workflows, Pandas usage patterns, and library-aware code composition. Both the dataset and the extraction pipeline are publicly available via GitHub and Zenodo.

0.20

Code Health in LLM-Based Test Generation: Effectiveness and Token Efficiency

cs.SE ·2026-08-19 ·Freya Wirdemann, Markus Borg, Nadim Hagatulah, Adam Tornhill
published similarity 0.60
Abstract · Accepted at the Engineering Track of the 26th IEEE International Conference on Source Code Analysis and Manipulation (SCAM 2026)

Coding agents powered by Large Language Models (LLMs) are now prominent in software engineering. Previous work has shown that AI tools perform better on high-quality source code that is easy to maintain. In this study, we investigate how the effectiveness of LLM-generated unit tests varies across maintainability levels measured by CodeScene's CodeHealth (CH). We assess test effectiveness using traditional coverage metrics and mutation score across Python, Java, and C++. Moreover, we study how code with different levels of CH translates into input tokens using common industrial tokenizers. Our results suggest that CH provides a weak but consistent signal of LLM-generated test effectiveness and is negatively correlated with input-token count. These findings provide further evidence for a relationship between maintainability and LLM-based software development.

0.39