CS Paper Radar

Data Quality Assessments: A Theoretically Structured Overview of Approaches and Methods

Ralph Foorthuis

cs.DB stat.OTstat.ME ·submitted 2026-08-23 15:53 UTC · v2
publishedsurveyunusual

Abstract

The quality of data is crucial for both practice and academia, and this holds for descriptive statistics, AI and advanced analytics alike. This study addresses an apparent gap in the literature and as such presents a full and theoretically grounded overview of data quality assessment approaches and methods, as well as their defining characteristics and inter-relationships. For this purpose a broad typology is introduced that employs two theoretical dimensions, namely the evaluation logic (formal versus informal) and the assessment driver (norms versus data). This yields four high-level approaches and 15 methods for assessing data quality. The research results are relevant for academia, as they provide a theory-based overview and definition of the ways that data quality can be evaluated. The study is also relevant for practice because it allows professionals to make informed decisions on using these methods, e.g. as part of an audit or broad data quality assessment strategy.

Comment
9 pages, 4 figures, 1 table. Accepted for presentation

Closest papers

Nearest neighbours by meaning across the whole index.

Rethinking Data Quality for AI-Driven Systems: Evidence from Practitioner Interviews

cs.SE cs.AI ·2026-09-25 ·Hariharan Gopinath, Jan Bosch, Helena Holmström Olsson
similarity 0.61
Abstract · This is a preprint version and the final version will appear in the proceedings of PROFES 2026

Data quality research has usually treated data as an input that is stored, processed, and validated. In AI-driven software-intensive systems, data also shapes model behavior, evaluation, and lawful use. Empirical evidence remains limited on how practitioners define, assess, and manage quality under these conditions. We interviewed 16 practitioners from nine organizations and analyzed the transcripts using reflexive thematic analysis and developed six themes from participants' accounts. In AI systems, traceability shifted from modular debugging to attributing model behavior, while using models as quality assessors introduced circularity. Agent context and memory became data objects, and synthetic and pseudo-labeled data made authenticity a quality concern. In foundation-model development, lawfulness became a gate for training data, while representativeness was judged through coverage of situations in which the system must behave safely. Prior ML research examines many of these problems separately. Our study provides a practitioner-grounded account of how they are encountered together as an engineering and organizational concern. We also interpret five recurring conditions as helping explain how the themes relate to reduced trust in data and AI outcomes. We synthesize these findings through lifecycle assurance: a conceptual framing focused on producing evidence that data can support a specific AI claim when its influence may be embedded in model behavior, model-based judgments, or agent actions.

0.28

Knowledge Graphs Quality Assessment in the Era of LLMs: A Conceptual Framework

cs.CL ·2026-10-03 ·Elwin Huaman
similarity 0.60
Abstract · 8 pages, 1 figure, KGSWC 2026

Knowledge Graphs (KGs) have been popularized during the last few decades and are increasingly constructed and consumed by Large Language Models (LLMs). Various applications therefore rely on KGs to provide concise, complete, accurate, faithful, fresh, and traceable answers. However, what is the quality of those KGs? In which cases should a KG be used, and can it be trusted to ground an LLM? How might it be evaluated? We reviewed the literature on quality assessment (QA) of data, information, linked data, KGs, and the intersection of KGs and LLMs. We extended the current state-of-the-art frameworks to a catalog of 24 quality dimensions (QDs) and various quality metrics (QMs). Furthermore, we propose a general-purpose, customizable to a domain or task, and practical QA framework for assessing the quality of KGs.

0.20

Implementation Guidelines for Data Quality Metrics

cs.DB cs.LG ·2026-10-07 ·Philipp Jung, Katinka Becker, Felix Biessmann, Valerie Restat, Martin Seyferth +2 more
code/data similarity 0.57
Abstract · 14 pages, 7 figures, 5 tables. Submitted to EDBT 2027

Despite decades of data quality (DQ) research, a gap remains between DQ dimensions, such as accuracy or completeness, which the literature defines in textual form, and DQ tools, which typically implement low-level checks that are not aligned with these dimensions. ISO/IEC 25024 and ISO/IEC 5259 attempt to bridge this gap by defining DQ metrics for each dimension. However, these DQ metrics are hardly used, because the standards leave open how to implement them: for example, the metric for syntactic accuracy counts syntactically accurate values, but does not state how to decide that a value is syntactically accurate. This simply moves the problem to another level without solving it. As a result, DQ assessment currently cannot build on the standards. In this paper, we make the ISO DQ metrics executable. We classify all data-level metrics of both standards into (i) generalizable metrics that need no input beyond the data, (ii) parameterized metrics whose parameters can be learned from clean reference data or set by an expert, and (iii) non-generalizable metrics that need qualitative judgment and cannot be automated. For the metrics that can be automated, i.e., categories (i) and (ii), we propose implementation guidelines that resolve what the standards leave open. We realize the guidelines in dqmeasure, an open-source library of 20 metrics that learns these parameters from reference data instead of relying on manually defined rules. Our experiments on real-world and synthetic datasets show that the metric scores decrease monotonically with an increasing number of injected errors, decline together with downstream ML performance, and scale linearly with the number of rows, which enables automated DQ monitoring based on the standards.

0.28

How Do Data Collection Strategy and Data Quality Influence the Outcomes of Digital Technology Adoption?

cs.DB ·2026-09-11 ·Xuejiao Li, Cheng Yang
similarity 0.56
Abstract

In the era of Industry 4.0 (I4.0), data has become the essential foundation for digital transformation, yet many organizations still struggle to link data practices with digital performance outcomes. This study investigates how data collection strategy and data quality jointly influence the success of digital technology adoption (DTA) in manufacturing firms. Drawing on survey data from 86 firms, the research employs Partial Least Squares Structural Equation Modeling (PLS-SEM) to examine the relationships among data collection strategy, data quality, implementation performance, and operational performance. The results show that both data collection strategy and data quality significantly influence implementation and operational performance. However, the effect of data collection strategy on implementation performance is indirect, fully mediated by data quality. The study also finds that data collection strategy influences data quality. These findings demonstrate that data quality acts as a critical bridge between upstream data practices and downstream digital outcomes. The study contributes to the digital transformation and data management literature by empirically validating the central role of data quality and offering practical insights for managers to design and govern data processes strategically. It also sets a foundation for future research on data governance frameworks that integrate data quality assurance, standardization, and lifecycle management to sustain data-driven and digital transformation.

0.25

Data Patching

cs.HC ·2026-09-23 ·ATM Mizanur Rahman, Syed Ishtiaque Ahmed, Sharifa Sultana
similarity 0.56
Abstract

To provide data-driven quality services to their citizens, every institution determines the acceptability of the citizen data and datafied mechanisms through institutional protocols, including preset standards and policies. However, data meeting standards at one institution might fail to meet a different institution's standards in subsequent phases of the intended citizen services due to mismatched protocols, leading to data devaluation in the citizen service ecosystem. Terming this phenomenon cross-institutional data devaluation, we investigate its causes and workarounds through interviews with 41 Bangladeshi participants. We found that traditional data auditing mechanisms cannot solely address data devaluation; hence, we draw on our findings and theory of cross-institutional AI audits to propose the Citizen-centered Cross-institutional Data Audit (CCDA). We also discuss design and policy implications of CCDA in HCI and datafied citizen services.

0.35

Challenges and Contributions in Quality of AI-Based Software: A Systematic Mapping Study

cs.SE ·2026-08-26 ·Maryum Hamdani, Mateen Ahmed Abbasi, Marko Jäntti, Markku Tukiainen
published similarity 0.56
Abstract · Accepted in SEAA 2026

Artificial Intelligence (AI) is increasingly embedded in modern software systems, raising important questions about how its quality should be defined, assessed, and assured. This paper presents a Systematic Mapping Study (SMS) on the quality of AI-based software. The study synthesizes primary studies published between January 2020 and January 2026 and selected from five electronic data sources. A total of 33 primary studies were included after automated search, screening, and snowballing. The results identify six recurring challenge categories, with the most prominent being limitations in existing quality assessment models, followed by issues in non-functional requirement management, quality-aware development, and quality assurance. The findings suggest a call for collaboration of researchers and industrial practitioners with standardization organizations, that could possibly devise comprehensive quality assessments and their measurement methods.

0.32

Don't Blame the Model, Verify the Data: An Evaluation of SMT-based Dataset Verification (Extended Version)

cs.LO ·2026-09-17 ·Sehee Park, Dominik Geißler, Andrei Aleksandrov, Kim Völlinger
similarity 0.54
Abstract

The EU AI Act mandates that datasets for high-risk machine learning (ML) systems meet strict quality criteria such as soundness and bias mitigation. While Satisfiability Modulo Theory (SMT) solving offers a formal approach to verifying these properties, its scalability in realistic ML settings remains unexplored. To bridge this gap, this work presents the first large-scale empirical study of SMT-based dataset verification on two real-world ML datasets. We systematically evaluate how solver performance is shaped by three key dimensions: the type of data-quality property, the specification style, and the dataset encoding strategy. Our findings demonstrate that SMT-based verification is feasible for practical scenarios, but each dimension shapes it: the property type sets the tractability limit, the specification style drives scalability (exceeding $2{,}000{\times}$ for aggregate properties), and the encoding strategy has a systematic effect, with extracted feature columns performing best.

0.36

The Data Hospital: A Workflow-Based Concept for Explainable Research Data Quality Assistance

cs.HC ·2026-09-17 ·Lennard Scheurer, Robert Porzel, Vinicius Carrillo Beber, Rainer Malaka
code/data similarity 0.52
Abstract · Concept paper, 16 pages, 13 figures, 6 tables

Research data quality is multidimensional and purpose-dependent: it emerges from the interplay of data, intended use, contextual knowledge, documentation, intervention decisions, and traceability. This concept paper presents the Data Hospital, a human-in-the-loop control and interaction model for research data quality. Using a hospital metaphor, datasets are admitted, contextualized, assessed, reviewed in specialized stations, modified only through approved interventions, validated, documented, and made replayable where interventions are sufficiently specified. The concept combines deterministic profiling and inspectable evidence with optional evidence-bound explanation by Dr. Data and explicit user decisions. Preserved Raw Data and controlled working states separate observation from intervention. The contribution is not a new cleaning or imputation algorithm, but a ten-stage workflow that makes assessability, uncertainty, intervention authority, provenance, and process reproducibility visible. The prototype is an implementation-backed demonstrator rather than a released research artifact and illustrates selected parts of the concept through representational standardization, imputation, Patient File documentation, and replay. The paper concludes with a staged agenda for subsequent technical and user-centered evaluation.

0.38