CS Paper Radar

Protocol before progress: leakage-aware evaluation of AIS trajectory prediction

Zobeir Raisi, Vali Mohammad Nazarzehi Had

cs.LG ·submitted 2026-09-22 07:54 UTC · v1

Abstract

Reported gains in vessel-trajectory prediction from Automatic Identification System (AIS) data are credited to new architectures, but the evaluation protocol is rarely measured as a source of error reduction. We build a leakage-aware protocol with vessel-, time- and region-disjoint splits and apply it to two corpora with different traffic: 31 days of Danish national AIS traffic and 30 days of US Gulf coast traffic off Houston and Galveston. On both, we audit TrAISformer, GATransformer, and controlled AISFormer-inspired reconstructions. Three protocol effects appear in both corpora. First, TrAISformer's best-of-16 oracle decoder lowers error by a factor of 2.1-3.2 relative to greedy decoding. Second, a split that shares vessels lowers its greedy error by 23-25% at one hour, against 2% or less for a compact 0.43 M-parameter encoder. Third, a region-disjoint split raises TrAISformer's one-hour error from 2.2 to 24.6 km on the US corpus, because 99.9% of the test contexts fall in longitude bins never seen in training; the encoder built on local offsets is unaffected by this. Architectural mechanisms matter less: GATransformer's graph attention gives no measurable benefit on either corpus, while its waterway feature is worth 12-22%. The effect of a time-disjoint split is not stable across corpora (13% versus 2%). We release the splits and code.

Closest papers

Nearest neighbours by meaning across the whole index.

TempTPI: Informer-Based trajectory prediction for maritime vessels

cs.LG ·2026-09-09 ·Kevin Ferneding, Veronika Lietavcova, Aleksandra M. Blachowiak, Peder Heiselberg
published similarity 0.65
Abstract · Accepted to IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2026. \c{opyright} 2026 IEEE. Personal use of this material …

Accurate long-term trajectory prediction for maritime vessels is essential for safety and logistical efficiency. While deep learning models, particularly Transformers, have shown promise in processing Automatic Identification System (AIS) data, they often struggle with the quadratic computational complexity of self-attention and the loss of accuracy over extended forecasting horizons. This study proposes TempTPI, a novel prediction framework that integrates an Informer-based encoder with a multi-channel temporal encoding mechanism. The Informer architecture leverages a ProbSparse self-attention mechanism to reduce computational overhead and focus on the most significant dependencies, while the temporal encoder utilizes Fourier-like frequency expansions to capture cyclic patterns (hourly, daily, and seasonal) in vessel behavior. We evaluate our model against the state-of-the-art TPTrans architecture using AIS data from Danish waters. Experimental results demonstrate that TempTPI consistently outperforms existing methods across prediction windows of 1 to 5 hours. Notably, at a 5-hour horizon, the proposed model achieves a 55% improvement in Mean Squared Error (MSE), offering a robust solution for long-range maritime situational awareness.

0.28

Pairing AIS with RF: Analysis of a Markovian Approach for Maritime Data Fusion

cs.LG cs.CR ·2026-11-13 ·Daniel Grimm, Kyra Jongman, Renato Cordeiro Ferreira, Giuseppe Cascavilla, Alfredo Cuzzocrea +1 more
published similarity 0.61
Abstract · 10 pages, 5 figures, accepted at the 13th IEEE Special Session Privacy and Security of Big Data of IEEE BigData 2026

Monitoring maritime activities is important for safety and security, particularly for detecting illegal activities, but remains challenging due to the size and complexity of the maritime domain. The Automatic Identification System (AIS) is commonly used for this purpose, despite its known vulnerabilities. In contrast, space-based radio frequency (RF) observations provide an independent source of vessel activity but lack ship identification and therefore cannot identify vessels on their own. Following the CRISP-DM process, this study applies a Pair Hidden Markov Model (PHMM) to align individual RF detections with AIS trajectories, using temporal and spatial information while modeling noise and timing discrepancies. We evaluate it on NOAA AIS data with synthetic RF detections against nearest-neighbor baselines and analyze its errors with bias tests on trajectory length, sampling density, stationarity and AIS--RF distance. The length-corrected Forward score reaches a top-1 precision of $0.7776$ on the validation split of multi-candidate detections, where errors are possible, and an AUC of $0.637$ for detecting dark vessels among the detections that pass the prefilter. With a single RF detection, the score mostly reflects the AIS trajectory itself, and its errors favor longer and more stationary trajectories. A log-odds score, which compares the full model with a null model without match transitions, raises precision to $0.8580$ and AUC to $0.852$. Its length-corrected variant no longer shows a significant bias towards stationary trajectories and short inter-message distances, but still favors candidates farther from the RF detection. A time-weighted nearest-neighbor baseline without fitting remains ahead (precision $0.9013$, AUC $0.904$), so this single-detection, hand-designed PHMM formulation approaches but does not beat it.

0.35

Non-Parametric Spatiotemporal Trajectory Prediction via State-Conditioned Transition Sampling

cs.LG ·2026-08-14 ·Michael Fore, Akshay Jain, Justin Downes, Rohan Pradhan, Duncan Botti
publishedbenchmark similarity 0.60
Abstract · Accepted at the 34th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL '26), Riverside, CA, …

We present a training-free method for multi-modal trajectory prediction that achieves comparable accuracy to a 57M-parameter transformer while requiring no GPU and zero learned parameters. The method builds a transition table of historical state-to-next position pairs and retrieves neighbors using a product kernel over spatial proximity, bearing, speed, and temporal context. Two inference modes operate over this shared representation: diversity-penalized sampling produces trajectories covering distinct plausible routes, while beam search finds the highest-likelihood path. On the TrAISformer benchmark (Danish Maritime AIS), our method achieves competitive accuracy at full data availability and dramatically outperforms the transformer in data-scarce regimes---remaining stable down to 10% of training data where TrAISformer degrades catastrophically. This enables deployment in new geographic regions from an order of magnitude less historical data.

0.29

Trustworthy Geospatial AI Workflows: Reproducible Maritime Analytics and Forecasting on Open-Source AIS Data

cs.AI ·2026-11-19 ·Divya Acharya, Thomas M. Roehr, Arnaud Gotlieb
code/data similarity 0.59
Abstract

Publicly available geospatial data provide increasing opportuni- ties for data-driven maritime analytics and predictive modelling. However, their availability does not necessarily imply that the underlying observations are equally suitable for downstream AI applications. Properties of the data source and observation pro- cess, including provenance, reporting regularity, spatial coverage, observation density, and missingness, can affect how geospatial data are interpreted, processed, and ultimately used by predictive models. At the same time, trustworthy geospatial AI requires ana- lytical workflows in which these processing decisions can be made explicit and reproduced. This paper presents a reproducible mar- itime geospatial AI workflow built around damast, an open-source data-processing and machine-learning library developed at Simula Research Laboratory. Using AIS observations from multiple public data providers as a case study, we demonstrate metadata-aware validation, reusable processing pipelines, maritime trajectory con- struction, and reproducible machine-learning experimentation. The predictive stage is model-agnostic and is instantiated in this study using a Transformer-based vessel-trajectory forecasting model. Building on these implemented capabilities, we further outline an envisioned extension toward trustworthiness-aware geospatial AI. The extension introduces explicit characterization of source- and data-level properties and their relationship with downstream model performance. This distinguishes capabilities demonstrated in the current workflow from trustworthiness-oriented components identified for future development. Overall, the study demonstrates how reproducible open-source infrastructure can provide a practi- cal foundation for developing more transparent and trustworthy geospatial AI workflows.

0.31

Sequential RF Fingerprinting for AIS: Rejecting Unknown Vessels and Insider MMSI Spoofs Days After Enrollment

eess.SP ·2026-10-13 ·Fawaz Abdul Razak, Bora Turgut, Mason Clark, Yasin Yilmaz
similarity 0.56
Abstract · The authors are withdrawing this version to update the experimental results. A revised version will be posted

Any AIS transmitter can claim any vessel's identity (MMSI), because nothing in the protocol authenticates it; only the transmitter's analog hardware reveals who is really sending. We treat AIS traffic as streams of bursts keyed by claimed MMSI and accumulate physical-layer evidence with a sequential test. Each burst is scored by its k-nearest-neighbor distance to enrolled fingerprints in the feature space of a classifier whose weights are fixed after training, calibrated into a tail probability, and accumulated by a geometric-entropy-minimization cumulative-sum (GEM-CUSUM) statistic. The same test rejects unknown transmitters when scored against all enrolled vessels, and exposes insider spoofs when scored against the claimed vessel. A threshold refreshed from genuine traffic alone keeps it at a stated false-alarm rate as fingerprints drift. On AIS in-phase/quadrature (I/Q) recordings captured over 13 days at the Istanbul Strait (Bosphorus), the detector rejects unseen vessels at least 10 days after enrollment. Evaluated over 50-burst trials with every vessel weighted equally, it reaches an area under the ROC curve (AUROC) of 0.956 (95% confidence interval 0.913-0.989), where the best score computed from one burst at a time reaches 0.899. A threshold refreshed from the first quarter of the enrolled vessels' traffic holds 5.3% false alarms (3.0-9.2% across five independently trained encoders) at 88.8% detection. Scored against the claimed MMSI, enrolled transmitters spoofing another vessel are rejected at 0.991, a threat that open-set tests cannot see (AUROC 0.500, chance level). The lightweight Transformer GLFormer, reimplemented from its paper (no code is public) and retrained on the same data, reaches 0.652 in the same detector with its published in-phase input and 0.849 when given the same I/Q input as our encoder.

0.26

UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets

cs.SD cs.CVeess.AS ·2026-09-12 ·Connor Hashemi, Trevor Stout, Anthony Hoogs, Jason Parham
top venuebenchmark similarity 0.56
Abstract · 9 pages paper, 4 pages supplementary at the end, accepted/will be published at Oceans 2026. Note that while this is underwater acoustics, we…

Underwater Acoustic Target Recognition (UATR) of ships is well-suited for machine learning, yet its progress is hindered by the lack of large, diverse, and publicly available labeled datasets. In this work, we introduce UniqueShip, a machine learning-ready benchmark dataset for UATR applications sourced from the open Ocean Networks Canada (ONC) repository. Unlike previous datasets, we explicitly control for "data leakage" between the training and evaluation sets to ensure more reliable and generalizable model evaluation that does not encourage the model to memorize individual ships. We demonstrate that typical, random data partitioning in two prominent UATR datasets leads to falsely optimistic test performance, increasing accuracy by 10-48 percentage points compared to our more careful partitioning. Ablations on UniqueShip further show that doubling the number of unique vessels improves accuracy by 2.4-2.6 percentage points, while doubling total audio duration improves only by 0.8-1.3 points, indicating that vessel diversity should drive dataset curation more than total hours. We provide baselines with convolutional and transformer backbones, and analyze how ship metadata correlates with classification performance, finding that individual vessel characteristics predict classification difficulty far better than distance to the hydrophone alone. Overall, UniqueShip contains 2,460 hours of ship-radiated audio from 4,218 unique vessels (3,437 hours including background). We publish the dataset, code, and easy-to-download splits at uniqueshipdata.org to foster further UATR research.

0.69

When appearance and motion disagree: evaluating class-motion consistency against disguised uncrewed surface vessels

cs.CR cs.CVcs.RO ·2026-10-30 ·Jon Reifschneider
similarity 0.56
Abstract

Defense systems for maritime assets must distinguish approaching benign craft from uncrewed surface vessels (USVs) with sufficient warning to act. Shore cameras using computer vision can separate hostile vessels from benign traffic, but these systems may be attacked using an adversarial hull patch or physical disguise. We evaluate adversarial hull patches and find they suppress detection in up to 29% of attempts but never cause misclassification as benign. Physical disguise, however, can bypass both a camera check and a radar presence check. To evaluate defenses, we develop a hybrid replay protocol from automatic identification system (AIS) tracks that combines recorded benign motion with direct or off-axis attack run-ins. We find that for recreational-labeled vessels attacking via direct run-in within 4 nautical miles, an asset-relative closing-rate check flags 94-95% at 15-20 knots, compared with 12% for a track-only speed check. The checks are complementary on off-axis approaches, constraining the attacker to remain below both thresholds and extending the time required to reach the asset. Flagging on either check has a 4.9% held-out false-alarm rate overall and 12.3% on recreational traffic. The replay-based evaluation protocol supports comparative test and evaluation of defense methods.

0.23

The Split Protocol Decides the Result: Re-Evaluating Encrypted Traffic Classification

cs.LG cs.AI ·2026-11-04 ·Mayan Sharma, Kriti Saini, Devansh Behl
code/data similarity 0.56
Abstract · 12 pages, 4 figures, 6 tables Code and data splits: https://github.com/Mayan10/Flowconx

Encrypted traffic classifiers are routinely reported at 95-99% accuracy. We show that on one of two public corpora, the same model and code score macro-F1 0.992 under the evaluation protocol standard in the literature and 0.567 under a protocol that keeps flows from one packet capture on one side of the split. Nothing changes but the partition (gap 0.426 over eight seeds, p = 0.008 after Holm-Bonferroni correction). The inflation is not a property of random splitting as such. On a month of backbone QUIC traffic, we evaluate five split protocols at eight seeds each: random, temporal and client-disjoint splitting are statistically indistinguishable from session-disjoint, while server-disjoint splitting costs 0.192 macro-F1 (d = 17.0, p = 0.008). Which axis matters depends on how the corpus was collected, in a way invisible from its label space. We give a diagnostic that identifies the right axis before any model is trained: single-identifier probes. A decision tree on the capture identifier alone reaches 0.983 under a random split and exactly 0.000 under session-disjoint splitting. Under this evaluation, a metric-trained flow embedding leads four deep baselines but does not beat gradient-boosted trees on twenty packet sizes. Five of our own pre-registered architectural claims fail: distance-based rejection of unknown applications beats softmax thresholding, but ablating every metric objective makes rejection better, because class separation correlates at r = -0.88 with rejection quality. Few-shot enrollment is flat within a corpus but, across corpora, five labelled flows per class recover 91% of the deficit left by failed zero-shot transfer. All splits, results, code and negative findings are released.

0.34