CS Paper Radar

ArtifactBench: Lineage-Aware Evaluation of AI-Generated Music Detectors under Distribution Shift

Heewon Oh

cs.SD eess.AS ·submitted 2026-09-20 11:03 UTC · v1
benchmarkhot direction

Abstract

AI-generated music detectors are commonly compared using aggregate scores on benchmarks whose training overlap, generator lineage, source provenance, and audio-transformation history are only partially observable. This paper introduces ArtifactBench, a lineage-aware evaluation suite for measuring detector behavior across generator families and versions, real-music domains, collection-cohort shift, and inference coverage. The benchmark groups source recordings and their derived variants by content identity, separates calibration from final testing, records inference failures independently from classification errors, and reports source-level performance with uncertainty in addition to aggregate metrics. We evaluate multiple publicly available detectors under a version-pinned common protocol and examine how leakage control, cohort availability, threshold policy, and model-specific missingness alter measured performance and model ranking. On the 562-track common-success test intersection, ArtifactNet obtains 0.982 AUROC and 0.918 balanced accuracy, compared with 0.761/0.776 for the public Deezer detector; SpecTTTra and CLAM fall below 0.30 AUROC under this shifted cohort. These results also expose substantial generator- and real-domain shifts that aggregate scores alone conceal.

Comment
9 pages, 1 figure, 6 tables. Related to ArtifactNet (arXiv:2604.16254); this work contributes the benchmark and evaluation protocol rather than a new detector

Closest papers

Nearest neighbours by meaning across the whole index.

MUSEBENCH: A Benchmark for AI-Generated Music Detection and Artifact Analysis

cs.SD ·2026-10-14 ·Yi Chen Liu, Syed Irfan Ali Meerza, Jiawei Yu, Xi Guo, Samuel Bright +1 more
benchmarkcode/data similarity 0.65
Abstract

Advances in generative AI have made it possible to produce music that is nearly indistinguishable from human compositions, raising the question of whether such works can be reliably distinguished from human compositions. We present MUSEBENCH, a large-scale benchmark for AI-generated music detection and artifact analysis, comprising over 361K tracks. The dataset includes human compositions, stylistically matched music generated by ten state-of-the-art open-source and commercial models, such as Suno and Udio, as well as human-AI co-creations and variants processed with adversarial perturbations, compression, and acoustic post-processing. Using MUSEBENCH, we benchmark 11 automated detectors and evaluate their accuracy, generalization, and robustness, under default, out-of-domain, and transformed-variant experiment settings. We further conduct a comprehensive artifact analysis through acoustic feature inspection, identifying recurring signatures such as non-idiomatic harmonic progressions, upsampling induced spectral periodicities, and violations of physical performance constraints. Our findings reveal key strengths and vulnerabilities of existing detectors and provide practical insights for building more reliable AI music detection systems.

0.19

Building a Dataset for Music Sample Identification

cs.SD ·2026-09-14 ·R. Oguz Araz, Xavier Lizarraga, Xavier Serra, Dmitry Bogdanov
code/datahot direction similarity 0.64
Abstract · Extended Abstracts for the Late-Breaking Demo Session of the 27th Int. Society for Music Information Retrieval Conf., 2026

Sample identification (SI) is the task of matching an element of a musical work to its musically transformed versions used to create new works. The task has received little attention and lacks large-scale publicly available data. In this work, we mine sampling annotations from a music database and split them for training and evaluation. The resulting dataset is nearly three orders of magnitude larger than the existing SI benchmarks, with training, validation, and test sets of 114 k, 6 k, and 10 k tracks. We find that naively splitting the annotations places the same tracks in different sets. To avoid this, we construct a graph from the annotations and split it over connected components. We further find that a single mega-component contains half of the annotations, making component-wise splitting incompatible with balanced splits; we trim it, yielding a leakage-aware pipeline. We share the dataset for non-commercial scientific research purposes only and make the data-analysis and splitting code publicly available. We hope that our work fosters research on SI.

0.61

Distinguishing AI-Generated Music from Edited Audio as a Hard-Negative Robustness Task

cs.SD cs.AIcs.LG ·2026-08-14 ·Alexandru-Stefan Morosanu, Valerian Cecan, Stefan-Daniel Achirei, Laura Erhan
published similarity 0.62
Abstract · Accepted at the RobustifAI 2026 Workshop @IJCAI-ECAI 2026, Bremen

AI-generated music detectors are commonly evaluated against original songs, but real-world uploads are often remixed, re-encoded, pitch-shifted, or otherwise edited. These edited versions form a difficult negative class: they are not generated by AI, yet they may introduce spectral artifacts that resemble synthetic audio fingerprints. We study this problem as a hard-negative robustness setting for AI-generated music detection, focusing on AI-generated and edited variants derived from the same anchor songs. We compile a YouTube-based dataset of AI, edited, and original variants, using the original tracks only as references, and train a binary AI versus edited detector. Audio is processed as 10-second clips and passed as raw waveforms to a pretrained PaSST spectrogram transformer. To reduce leakage, all splits are performed by anchor song. On the held-out test set, the final video-level system achieves 0.811 balanced accuracy. At clip level, AI-generated clips reach an F1-score of 0.836, while edited clips reach a lower F1-score of 0.720. The results suggest that AI-generated music retains detectable fingerprint-like spectral cues beyond ordinary editing, but the lower edited-class F1-score shows that these cues can still overlap with artifacts from edited audio. Grad-CAM visualizations are used to inspect whether high-confidence predictions rely on localized time-frequency regions.

0.20

Tracing Inputs, Verifying Outputs: Validating Attribution in Music Generation

cs.SD cs.LGeess.AS ·2026-10-07 ·Taejun Kim, Wonil Kim, Jongmin Jung, Hyeongseok Wi, Sangeun Kum +8 more
similarity 0.60
Abstract · 15 pages, 4 figures. Audio examples: https://neutune.github.io/attr2027demo/

How can we verify whose music contributed to an AI-generated output? This paper demonstrates how input-based attribution can provide verifiable evidence of which audio sources were used in a generation and whether they shaped the output. To do so, we condition the generation solely on audio without any text input, then trace the inputs behind each output, and establish their musical effect. In prompt adherence tests and controlled input swaps, the stems generated by our generator, MixAudio, follow the prompt audio in timbre and the context audio in harmony. Yet these outputs may still reproduce training data not supplied as inputs. We therefore audit memorization with our musical version identification model, musicDNA, and find few reproductions outside the input records. On human-judged cases within the flagged pool, it achieves higher precision and recall than the other tested memorization detectors. The two evaluations suggest that input records and output analysis provide complementary evidence for attribution, on which rights-holder reporting and compensation can draw as the AI music economy takes shape. Audio examples are available at https://neutune.github.io/attr2027demo/

0.32

Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution

cs.SD cs.LG ·2026-09-07 ·Gang Shi
similarity 0.60
Abstract · 13 pages, 5 figures

Audio provenance attribution - which system produced a synthetic utterance - is reported at near-ceiling accuracy on clean benchmarks, yet audio reaching an analyst has usually been transcoded. We report a prospectively registered measurement of closed-set attribution after single-stage codec transport, with the analysis region fixed from fidelity metadata before any attribution model was trained. On two corpora, in-support losses reach 53.5 [43.5, 63.6] and 70.3 [63.0, 77.5] Macro-F1 points for WavLM-Base+, and 61.0 [56.8, 65.1] and 49.8 [41.6, 57.9] for W2V2-BERT 2.0, under simultaneous component-level bands. Degradation is strongly condition- and representation-dependent: within one in-support grid WavLM losses run from -0.4 to +53.5 points, and the two encoders differ beyond a prespecified +/-5-point margin at six of twelve conditions. A clean-qualified ECAPA-TDNN and a Proxy-Anchor head degrade comparably, so the effect is not confined to one representation family or a weak linear head. The registered matched-fidelity comparison was not estimable on this grid, and waveform and perceptual measures order the conditions differently: MP3 at 8 kbit/s ranks mid-grid on SI-SDR but last on PESQ-WB while causing the largest loss. For the tested tasks, corpora, representations and codec grid, a clean accuracy figure does not by itself characterise deployment robustness.

0.22

GlobalTap: Globally Diverse Beat Tracking Benchmark via Crowdsourced Tapping

cs.SD eess.AS ·2026-11-08 ·William T. Botticelli-Wells, Manuel Anglada-Tort, Elif Celen, Rena Far, Harin Lee +2 more
publishedbenchmarkcode/data similarity 0.59
Abstract · Accepted at ISMIR 2026. 13 pages, includes supplementary appendix

Existing beat-tracking benchmarks draw from a narrow range of musical cultures, predominantly Western, and are difficult to scale due to reliance on expert annotation. We address both limits with GlobalTap, a benchmark of 200 popular-music excerpts spanning 59 countries, annotated by 746 participants recruited across ten countries. The benchmark is created using a scalable online tapping pipeline that records taps via audio for high temporal precision, aggregates pooled tap times into consensus beat annotations, and computes intrinsic quality measures that flag perceptually difficult excerpts without requiring a reference. In a separate study on a different 240-excerpt validation set, tapped by an additional 733 participants, an independent group of 130 listeners rated how well the resulting beat tracks lined up with the music, and we detected no overall difference between our annotations and existing reference annotations. Crowd-derived annotations occasionally diverged from the references in phase and metrical interpretation, yet listeners did not consistently prefer the reference versions, suggesting that crowd-derived and existing reference annotations can reflect different perceptual or structural interpretations. The disagreements were often systematic rather than random, and metrical-level choices differed across the ten recruitment countries. We release GlobalTap together with open-source pipeline code as a listener-derived resource for evaluating beat-tracking systems on culturally diverse music.

0.36

A Stem-Agnostic Approach to Hybrid AI Music Detection

cs.SD eess.AS ·2026-09-22 ·Richa Namballa, François Rigaud, Romain Hennequin
similarity 0.58
Abstract · 4 pages + references, 5 figures, submitted to the 2027 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

The inclusion of generative audio in the music production process has led to an increase in hybrid music tracks that blend authentic human performances with AI-generated stems, challenging traditional AI music detectors which operate in a binary setting. In this work, we propose a stem-agnostic framework for identifying synthetic audio sources within hybrid musical mixtures. We introduce the inspectrogram, a novel time-frequency representation that maps localized probabilities of synthetic content across the audio spectrum. By combining the inspectrogram with a Wiener filter estimating target stem energy dominance, a single CNN model evaluates whether the specific stem is generated. Trained on rendered hybrid mixtures and evaluated across various stem classes, our model achieves strong performance on high-frequency sources such as vocals, drums, and guitar, but struggles on the low-frequency, narrow-band bass. We conclude that the quality of separation impacts the detection accuracy and identify source separation as a primary bottleneck and a crucial direction for future research.

0.29

Unified Music Identification for Tracks and Versions

cs.SD ·2026-08-20 ·R. Oguz Araz, Joan Serrà, Yuki Mitsufuji, Xavier Serra, Dmitry Bogdanov
publishedbenchmark similarity 0.58
Abstract · Accepted to ISMIR2026

Given a music database, track identification (TI) retrieves the exact track matching an audio excerpt, whereas version identification (VI) retrieves its musical versions. Traditionally, the two tasks have been addressed separately. However, as every track is its own closest version, we investigate whether VI can subsume TI. This requires VI systems to be robust to both signal manipulation and audio degradation. We therefore propose a unified benchmark that evaluates accuracy and robustness on each task. Comparing seven existing models on this benchmark, we show that none of them are both accurate and robust on both tasks. We then train a baseline model targeting both tasks and show that a unified system is possible with 10 s TI queries. Lastly, we characterize the two retrieval constraints that limit our model's TI performance. We envision extending this unification to other music identification tasks.

0.42