CS Paper Radar

Perceptually Motivated Alignment and Interpolation of Pitch-Aligned Time-Frequency Representations

Shahan Nercessian, Jeff Sontag, Alejandro Koretzky

eess.AS cs.SDeess.SP ·submitted 2026-09-25 20:15 UTC · v1
published

Abstract

We propose a framework for the alignment and interpolation of pitch-aligned time-frequency representations. Building on the tonal interval vector, we introduce a series of extensions that reformulate it as an invertible operator, culminating in a new feature extractor that embeds perceptual consonance priors within a pitch-aligned representation. Accordingly, we develop methods for aligning and interpolating between musical structures. For alignment, we cast the problem as a permutation search under perceptually weighted distances, enabling robust matching without explicit key detection, which is inherently ambiguous in music information retrieval. For interpolation, we employ optimal transport to generate musically meaningful transitions between pitch distributions, proposing a circular formulation over the Circle of Fourths/Fifths to respect harmonic structure. Finally, we learn a low-dimensional geometric representation of scale structure aimed to factorize scale color and density. Experimental results demonstrate the effectiveness and musicality of the proposed methods. Together, our contributions provide a principled, purely signal processing approach to modeling pitch structure in audio and symbolic music signals.

Comment
8 pages, 7 figures. Accepted to the 29th International Conference on Digital Audio Effects (DAFx26)

Closest papers

Nearest neighbours by meaning across the whole index.

Geometric Representations for Transformed Pattern Matching in Music

cs.SD ·2026-10-06 ·David Meredith
similarity 0.71
Abstract · 35 pages, 8 tables, 15 figures. Draft of first chapter of book currently in preparation for publication by Springer

We review the notion of representing music using point sets and argue that such representations are better adapted than sequential representations for matching patterns in unvoiced, polyphonic music, such as keyboard music. Musical transformations such as transposition, inversion, diminution, augmentation and retrograde can be modelled by geometric transformations in pitch-time representations that combine translation with scaling parallel to and reflection in the time axis. We identify eight types of geometric pitch-time representation that use chromatic pitch, morphetic pitch, morph or chroma to represent pitch and either onset time or midtime to represent time. We illustrate how these types of representation allow us to characterise different types of musical transformation. For example, by using midtime instead of onset time, we can precisely characterise certain retrograde relationships; and by using morph and chroma pitch representations we can characterise transformations involving octave displacements and duplications. We present the concept of a transformation class and consider the three specific classes, $F_{\mathrm{2STR}}$, $F_{\mathrm{2STRMod7}}$ and $F_{\mathrm{2STRMod12}}$. We introduce the notion of an inter-pattern transformation graph for a set of patterns, $S$, and a transformation class, $F$. Each vertex in such a graph represents a pattern in $S$ and there is an edge in the graph from pattern $P_1$ to pattern $P_2$ if and only if $P_1$ can be mapped onto $P_2$ by a transformation in $F$. We show, with the aid of such graphs, that the musical relationships between the occurrences of the HAYDN theme in Ravel's Menuet sur le nom d'Haydn can be precisely described in terms of transformations in $F_{\mathrm{2STR}}$, $F_{\mathrm{2STRMod7}}$ and $F_{\mathrm{2STRMod12}}$ within the pitch-time representations considered.

0.22

Dissonance Spectrum explicitly models perceptual frequency interactions for better music understanding

cs.SD cs.AI ·2026-08-26 ·Tianle Wang, Xinyi Tong, Liangke Zhao, Jishang Chen, Sirui Zhang +5 more
similarity 0.66
Abstract

Conventional music representations describe acoustic energy over time and frequency but do not explicitly expose relations among simultaneous frequency components. We introduce the \emph{Dissonance Spectrum} (DS), a nonnegative time--frequency representation that applies a tolerance-based rational pitch-relation kernel with logarithmic harmonic distance to a constant-Q spectrum and attributes aggregate pairwise interactions back to individual frequency bins. Controlled music-theory tests show strong ordinal agreement for intervals, harmonic-function connections, and church modes, and weaker but significant agreement across diverse chord voicings. DS is then encoded by a lightweight parallel branch whose zero-initialized residual projection preserves the baseline function at initialization. Across six paired training seeds in open-ended music question answering and categorical and dimensional music emotion recognition, DS obtains the highest mean on every reported endpoint relative to the unchanged baseline, a parameter-matched Gaussian-input branch, and an architecture-matched magnitude-CQT branch. These results support DS as an interpretable, complementary representation, while listener-specific perception and broader task coverage remain open problems.

0.34

A Multidimensional Model for Quantifying Tonal Strength: A Continuous Framework of Analyzing Tonal Evolution Beyond Tonal-Atonal Binary Classification

cs.SD cs.MM ·2026-10-04 ·Yuliang Li, Nan Nan, Meilian Gu, Xiaohong Guan
dataset similarity 0.63
Abstract

Tonality is a fundamental organizing principle of Western music. However, the gradual transition from common-practice tonality to atonality around the turn of the twentieth century cannot be captured by binary labels. To characterize intermediate tonal states and trace this evolution, we propose a multidimensional framework for quantifying tonal strength, grounded in tonal theory and mechanisms underlying tonal dissolution. This framework operationalizes the global statistical regularity, temporal coherence, and local perceptual alignment of tonal organization through three complementary features: pitch-class distribution uniformity, temporal stability of tonal centers, and local tonal clarity, respectively. To validate the model, we introduce two new datasets: a binary dataset comprising 196 tonal and atonal pieces, and a historical MIDI corpus of 1,561 compositions by 21 composers spanning the Baroque era to the early twentieth century. The proposed model achieves high discriminative performance on the binary classification task (F_1=0.985), and quantitatively reveals a continuous historical trajectory from established tonality through progressive weakening to eventual dissolution. Computational analysis of representative composers further confirms significant declines in tonal strength throughout the careers of late-Romantic and modernist figures. Additionally, analysis of EMOPIA dataset reveals a weak but statistically significant positive correlation between higher tonal strength values and positive emotional valence. This work establishes a unified quantitative model for analyzing tonal evolution and provides a continuous measure for future behavioral and neurophysiological investigations of the relationships among tonal organization, musical emotion, and cognition.

0.26

Automatic Transcription of Microtonal Free-Rhythm Vocal Music: A Case Study in Iranian Classical Music

cs.SD ·2026-08-17 ·Sepideh Shafiei, Shapour Hakam, Harsh Dange, Joel Rodriguez Caraballo
similarity 0.63
Abstract

This paper introduces a computational workflow for automatically transcribing microtonal, free-rhythm vocal music, with Iranian classical music as a case study. Our approach is based on performances by the renowned vocalist Karimi and ground truth transcriptions by the prominent ethnomusicologist Masoudieh [14], which were subsequently incorporated into the IRMA Audio-MIDI dataset [20]. To accurately extract melodies, we employ pitch histograms in conjunction with Dynamic Time Warping (DTW). Additionally, we introduce specialized musical notations to capture the intricate ornamentations characteristic of the genre, with particular emphasis on the vocal technique tahrir. The transcription process is implemented in Python using the music21 library for symbolic music representation [5]. This study not only advances the field of computational ethnomusicology but also highlights the potential of computational methods in preserving and analyzing complex musical traditions. The transcription system also generates a combined visualization of the audio pitch contour and the DTW-aligned MIDI representation, enabling users to inspect the correspondence between the performance and the generated transcription. A companion visual editor supports expert-in-the-loop correction of the resulting notation.

0.18

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

cs.SD cs.AI ·2026-10-01 ·Liwei Lin, Gus Xia
position similarity 0.63
Abstract

How can we understand what a music foundation model has learned \textit{internally}? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as \textit{structured relations} rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, \textbf{we shift from feature identification to structure-based analysis}, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.

0.22

Sonic Palette: Predicting Dynamic Color Themes from Music

cs.MM cs.SD ·2026-11-19 ·Wen-Li Wei, Kenneth Yang, Jen-Chun Lin
similarity 0.61
Abstract

We present Sonic Palette, a music-driven framework for predicting time-varying color themes from audio. Rather than treating music-to-palette generation as a static problem, our method formulates music--color correspondence as a cross-modal sequence prediction task that models how palette-level visual attributes evolve over time in response to music. Specifically, Sonic Palette uses a music-conditioned autoregressive Transformer to predict frame-wise color themes, each represented by five dominant colors, from input audio features. To support this task, we curate a dataset of paired audio--color sequences derived from 265 full-length official music videos. We further visualize the predicted color themes through a palette-guided recoloring process that preserves image structure while revealing temporally varying color appearance associated with the music. Quantitative results show that Sonic Palette predicts color themes that more closely match the ground truth, while subjective results indicate more natural temporal color transitions and stronger perceived music--color atmosphere alignment, highlighting its potential for music-driven visual design, computational art, and creative visual authoring.

0.22

Pitch Contour Tokenization using VQ-VAE and Its Application on Korean Traditional Music Analysis

cs.SD ·2026-08-11 ·Seonguk Ju, Seola Cho, Sooin Chung, Danbinaerin Han, Dasaem Jeong
published similarity 0.61
Abstract · 8 pages, 3 figures, 2 tables. Accepted at ISMIR 2026

Computational analysis of music often relies on discrete representations, yet many musical traditions are organized around continuous pitch movement that resists segmentation into note-like units. For such traditions, the discrete units that analysis would build on are not given in advance. We address this gap by learning a vocabulary of local pitch-contour patterns directly from unlabeled audio, using a VQ-VAE that quantizes fixed-length contour segments into a finite codebook. To make the learned tokens stable across segmentation positions and small variations in timing and pitch range, we train the model with a reconstruction objective evaluated under the best alignment among a set of candidate temporal and pitch-domain transformations. Applied to Korean traditional music, the learned tokens recover information about expert-defined sigimsae categories without supervision, and in pansori individual tokens align with the two principal modes, Gyemyeonjo and Ujo, supporting their use as units for corpus-level analysis of contour-centric traditions.

0.25

A learning-based method of automatic music transcription

cs.SD ·2026-10-29 ·Phu-Anh-Tai Do, Huu-Hoang Nguyen, Duc-Hoang Nguyen, Thanh-Hai Le
similarity 0.60
Abstract · 19 pages, 14 figures

Automatic Music Transcription (AMT) seeks to convert acoustic music recordings into symbolic musical notation, a process that remains remarkably difficult for polyphonic piano due to concurrent polyphony, harmonic partial interference, expressive timing, and severe pitch activation sparsity. In this paper, we formulate polyphonic piano transcription as a frame-level multi-label classification problem across the standard 88 piano pitches (MIDI notes 21 to 108). Raw audio waveforms are processed using the Constant-Q Transform (CQT) to construct a geometrically spaced log-frequency time-frequency representation that aligns with the chromatic musical scale across seven octaves (252 frequency bins per frame). We systematically investigate recurrent sequence models, contrasting unidirectional Long Short-Term Memory (LSTM) with Bidirectional LSTM (BiLSTM) backbones, and propose two architectural adaptations tailored to recurrent temporal representations: a modified additive attention module and a modified multihead attention mechanism equipped with hyperbolic tangent activation and head-wise summation. Furthermore, to mitigate the severe class imbalance between inactive and active notes (approximately 41:1), we evaluate Weighted Binary Cross-Entropy (WBCE) alongside standard Binary Cross-Entropy (BCE). Rigorous empirical evaluations on the MAPS and MAESTRO datasets demonstrate steady, compounding gains from bidirectionality and attention mechanisms. Our proposed Multihead Attention-BiLSTM architecture attains an F1-score of 88.20%, an accuracy of 78.90%, a precision of 0.89, and a recall of 0.86, substantially outperforming the baseline unidirectional LSTM (F1-score of 71.20%) while maintaining a compact parameter budget of 4.43M. Qualitative piano-roll visualizations confirm that our model generates faithful transcriptions even in complex harmonic passages.

0.19