CS Paper Radar

Structure Across Voices: Comparing acoustic-event type accumulation and sequence dependence across four vocal corpora using frozen audio encoders

Mudit Sinha, Sanika Chavan

cs.SD cs.AI ·submitted 2026-09-15 04:19 UTC · v2

Abstract

Comparing acoustic structure across very different vocal corpora is difficult because the recordings use different native event units, contain different amounts of sequence, and were collected under different conditions. We develop a matched procedure for comparing such corpora without forcing them into the same event definition. Frozen audio encoders provide a common acoustic representation for recorded events, while matched sampling gives each corpus the same amount of evidence for acoustic and sequence analysis. We apply this procedure to sperm-whale codas, human speech phones, Bengalese-finch syllables, and common-marmoset calls. The comparison reveals different patterns across the four corpora. Sperm-whale recordings stand out broadly: new acoustic forms continue to appear, while their sequences also show strong organization and repetition. Finch recordings show the strongest organization of sounds over time, especially in how neighboring sounds and repeated patterns are arranged. Direct temporal and spectral measurements show that these differences are also present in the recorded sounds themselves. The comparison provides a practical way to measure acoustic diversity and temporal organization across very different vocal corpora by combining a common acoustic representation with matched observation opportunity, without treating their native sounds as equivalent.

Comment
12 pages, 4 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Music of the Deep: An acoustic blueprint of sperm whale codas

cs.SD ·2026-10-25 ·Mudit Sinha, Sanika Chavan
similarity 0.72
Abstract · 10 pages, 3 figures, 4 tables

Sperm-whale codas are usually described by the number and timing of their clicks. Earlier acoustic studies showed that individual clicks and longer coda sequences contain additional organization. We ask how that organization compares with music, whose smaller sounds also form larger units and ordered sequences. We compare codas with performed violin music, drum grooves, and sung melodies. Their sequences resemble music in some measures of predictability, yet codas show a different relationship between the smaller sounds within a unit and the order of units across a sequence. This combination was not observed in the musical references tested. Coda timing also varies coherently and carries information about individual callers. The recordings show another striking property: broad, reproducible variation along acoustic dimensions associated with human ratings of emotion, with a range comparable to human audio in most eligible tests. These findings motivate an acoustic blueprint that describes natural codas beyond their click timing. Its importance is apparent in synthetic audio: WhAM produces recognizable coda-like recordings, yet the tested clips remain distinguishable from natural codas in five of six acoustic comparisons after strict matching. The blueprint provides a fuller way to study natural codas and to evaluate how faithfully synthetic recordings represent them.

0.23

OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations

cs.CL ·2026-09-28 ·Faadil Mustun, Chiara Semenzin, Roberto Dessi, Pablo Robin Guerrero, Pierre Orhan +5 more
top venuebenchmark similarity 0.67
Abstract · Accepted as a Spotlight at the NeurIPS 2026 Datasets & Evaluations Track

Recent advances in bioacoustics have been driven by large-scale corpora and standardized benchmarks, yet existing resources are overwhelmingly bird-centric and shallow per species, limiting their use for studying the structure of a single species' communication system. This gap is particularly acute for cetaceans: despite bottlenose dolphins (Tursiops truncatus) being a compelling case of complex vocal communication among non-human mammals, existing dolphin datasets are small, fragmented, and largely closed. We introduce OpenWhistle, the largest publicly available dataset of dolphin vocalizations. It comprises approximately 180,000 whistles (114 hours) recorded over five years from a stable pod of five individuals in a semi-natural environment, paired with a curated subset of 8,354 expert-annotated whistles and reproducible evaluation protocols for whistle-type detection and classification. We further release the full processing pipeline for whistle detection, segmentation, and categorization. To demonstrate its utility, we pretrain a Wav2Vec2.0 model adapted to dolphin acoustics on the OpenWhistle corpus and show that it learns effective representations, outperforming general-purpose bioacoustic models such as AVES and BioLingual on both tasks while leaving meaningful headroom for future work. By releasing the dataset, pipeline, and evaluation protocol, we provide the first open dolphin whistle dataset tailored for training self-supervised models, laying the groundwork for advancing dolphin communication research and developing models that capture fine-grained acoustic structure within species.

0.29

Interpreting Dolphin Vocal Sequences via Multiple Sequence Alignment

eess.AS ·2026-09-08 ·Daniel Kohlsdorf, Denise Herzing, Thad Starner
unusual similarity 0.67
Abstract

Dolphin communication understanding is essential for uncovering the linguistic complexity and social structures of wild pods. We adapt the ClustalW bioinformatics algorithm to analyze continuous acoustic data, treating vocalizations as high-dimensional spectral feature vectors. By replacing discrete scoring with a continuous Gaussian kernel similarity measure, our framework generates Multiple Sequence Alignment (MSA) visualizations that reveal shared structural patterns. These alignments highlight temporal motifs such as synchronized burst pulses in aggressive contexts that are difficult to detect through standard spectrogram inspection.

0.40

Audiovisual diarization of overlapping click trains in sperm whale (Physeter macrocephalus) vocal sparring using a three-hydrophone array

eess.SP q-bio.QM ·2026-09-11 ·Lara Berkenbaum, Hervé Glotin, François Sarano, Walter M X Zimmer, Véronique Sarano +2 more
similarity 0.64
Abstract

The individual attribution of sperm whale (Physeter macrocephalus) vocalizations during surface interactions constitutes a methodological challenge due to acoustic overlaps, multipath propagation, body shadowing, and indistinguishable inter-pulse intervals among similar-sized individuals. A multimodal audio-visual workflow is presented to deinterleave and attribute uncharacterized click trains produced by immature males during ``vocal sparring'' using a portable three-hydrophone array coupled with synchronized video. The approach combines spectro-temporal tracking, based on inter-click interval dynamics and the Constant-Q Transform, with spatial time-difference-of-arrival modeling projected onto the image plane for optical validation. Analysis of 2,655 manually validated clicks shows that purely acoustic clustering diarization errors remain low, peaking at 25.79% only during extreme temporal superpositions. Integrating the optical modality resolves residual spatial indeterminacies; although near-planar array geometry induces vertical ambiguities, the system achieves up to 100% horizontal visual concordance for primary emitters. Crucially, this framework successfully reconstructed and assigned 11 distinct, intertwined click trains totaling 882 clicks to specific focal individuals despite near-field tactile constraints. Because ethological descriptions remain incomplete without identifying the emitter, explicitly correlating these emissions with physical kinematics provides the fine-scale resolution required to define this socio-acoustic behavior.

0.30

Generating the Unheard: Phylogeny-Guided Latent Generation for Ancestral Sound Reconstruction

cs.SD ·2026-09-14 ·Tianyi Xu, Shrinaath Narasimhan, Evan Gorstein, Santiago Perea, Yunyi Shen +1 more
similarity 0.64
Abstract · 29 pages, 11 figures

What did an ancestral bird species sound like? Existing ancestral state reconstruction methods can infer low-dimensional traits such as morphological characters at internal nodes of a phylogenetic tree, but no one has tried to produce rich perceptual signals such as audio. Some of the challenges include inferred representations that are either too low-dimensional to decode or lie in non-generative feature spaces, so no method to date can produce ancestral audio. We introduce the first framework that generates plausible ancestral vocalizations. Our pipeline encodes bird recordings into a VAE latent space, learns a low-dimensional trait projection aligned with phylogenetic distances, performs ancestral inference in this trait space, and recovers decodable latents through an anchored inverse lift before emitting novel waveforms for each ancestral node. Because the entire pipeline stays within a decodable latent space, every internal node receives a genuinely new audio output representing plausible intermediate ancestral sounds unavailable to retrieval-based alternatives. Experiments on two phylogenetically distant bird clades, 21-species Tyrannidae and 19-species Paridae, show that our method is the only approach that simultaneously achieves genuine generation, phylogenetic consistency, and naturalistic audio quality across both datasets.

0.48

Dual-Channel Contrastive Encoding of Sperm Whale Coda

cs.SD cs.AIcs.LG ·2026-10-21 ·João Victor Quintanilha, Emma Virnelli, Shuo Zhang
code/data similarity 0.63
Abstract

Sperm whales (Physeter macrocephalus) communicate through short click sequences called codas, whose structure includes the rhythmic pattern of inter-click intervals (ICI) and the spectral composition of the clicks. In this paper, we propose the Dual-Channel Contrastive Encoder (DCCE), which explicitly encodes rhythm and spectral information with separate encoders before fusing them. We evaluate DCCE on three coda-understanding tasks: social unit, coda type, and individual identification. We compare DCCE against strong baselines from literature under a common linear-probe protocol and analyze the role of each channel and training objective through ablations. On 1,383 codas, shared partitions and validation-only layer selection support three random splits and three recording-date-disjoint folds. DCCE with unit pairing and type supervision achieves mean random-split macro-F1 of 0.901/0.865/0.597 for unit/type/identity. Raw ICI remains strongest for type (0.987), and ICI plus temporal mel statistics is a strong identity reference (0.724). An auxiliary identity objective improves spectral-branch identity prediction but does not improve the mean fused-identity score in this configuration. Date-grouped evaluation and four year holdouts expose generalization limits. The results support task-dependent combinations of timing and spectral information, without establishing independent biological channels or general architectural superiority. Code, notebooks, and complete results are available at https://dcce-research-site.vercel.app.

0.50

Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription

cs.SD ·2026-10-09 ·Ayush Kumar Bar, Promila Ghosh
code/data similarity 0.63
Abstract · Code is available at https://github.com/theboringai-work/surround-sound and pre-trained weights are hosted at https://huggingface.co/thebori…

Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.

0.26

Behavior-Conditioned Animal Vocalization Generation: Event-Aware Modeling and Multidimensional Evaluation

cs.SD ·2026-10-12 ·Xinlong Deng, Yong Tian, Jian Guan, Qiuqiang Kong, Jie Jiang +1 more
datasetcode/data similarity 0.63
Abstract · Submitted to ICASSP 2027. Project page: https://persimmontian.github.io/BCAVG

Behavior-conditioned animal vocalization generation aims to synthesize vocalizations from natural-language descriptions of animal behaviors, social interactions, and contexts. This task remains largely unexplored, with limited behavioral text supervision and substantial variation in vocalization duration. We construct the Behavior-Vocalization Dataset (BeVo) by integrating public bioacoustic data from six species with natural-language behavioral conditions derived from species-specific ethological and bioacoustic literature, totaling 31,191 events. We then adapt a rectified-flow-based generator with Event-Aware variable-length modeling, which encodes vocalizations at their actual durations and uses masks to prevent attention to padded latent positions and exclude these positions from the training loss. We evaluate vocalization discriminability, acoustic distribution fidelity, and language-aligned representation fidelity. Experiments show that task-specific adaptation improves all evaluated metrics compared with its zero-shot baseline, while Event-Aware modeling further increases vocalization-anchor macro accuracy from 59.86% to 61.99% and reduces Frechet Audio Distance from 1.40 to 1.22. Conditioning analyses further show that behavioral conditions improve target-anchor discriminability, while swapping behavior shifts generations toward the corresponding vocalization anchors. Demos are available on our project page.

0.30