CS Paper Radar

MINT: A Universal Zero-Shot Predictor for Transaction Data

Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan, Julia Rozanova, David Sutton, Stuart Burrell

cs.LG cs.CL ·submitted 2026-08-14 11:17 UTC · v1

Abstract

Banks analyse sequential financial transaction data to perform many tasks, including fraud prevention, credit risk assessment and offer personalization. To improve the predictive accuracy of these tasks, Payments Foundation Models encode transaction sequence data as rich contextual embeddings, which can then be provided to task-specific models as features. However, these Foundation Models are not designed for flexible zero-shot reasoning across novel downstream prediction tasks, limiting their adaptability and utility. Existing LLM-based approaches to zero-shot prediction often fail to fully exploit the predictive signal within transaction data, while relying on costly text serialization or task-specific architectures that scale poorly. To address these limitations, we present the Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning. We find that MINT achieves state-of-the-art predictive question-answering performance in both in-distribution and out-of-distribution questions, while substantially reducing input tokens, latency, and memory consumption compared to text-serialization baselines. Through comprehensive analyses of representations, alignment strategies, training data, and history length, we establish that compact transaction embeddings are a superior approach to transaction representation than text serialization for multimodal reasoning and zero-shot prediction tasks.

Closest papers

Nearest neighbours by meaning across the whole index.

Large Payments Model

cs.LG cs.AIcs.CE ·2026-11-11 ·Natraj Raman, Abhenandhan Manoharan, Vishwasreddy Thuniki, Dhvani Kansara, Anirudh Som +10 more
similarity 0.73
Abstract

Payment applications such as fraud detection, authorization prediction, and payment optimization draw on overlapping transaction information. This motivates a reusable payments intelligence layer that shares learned representations and predictive signals, rather than developing predictive models independently for each task. Realizing this shared capability requires handling heterogeneous transaction fields, high-cardinality identifiers, lifecycle-dependent attributes, long histories, and rare outcomes while meeting latency and cost constraints. We present Large Payments Model (LPM), a domain-specific foundation model designed for operational use and trained from scratch on large-scale structured transaction data to learn reusable representations for payment prediction tasks. LPM employs field-specific encodings and lifecycle-aware self-supervision in two complementary architectures: the Independent variant uses hybrid masking and supports efficient per-transaction inference, while the Sequence variant incorporates entity-linked histories through compact latent representations for richer temporal context. Optional distillation enables resource-constrained deployment. We pretrain LPM on up to 400 billion tokens and study scaling, downstream performance, and representation reuse. LPM improves over strong task-specific baselines across multiple prediction tasks in Merchant and Treasury Services, demonstrating applicability in distinct payment domains. Its representations also facilitate adaptation to a target not observed during pretraining through compact task-specific components. Together with production deployment, these results position LPM as a practical basis for reusable payments intelligence.

0.07

Transaction Foundation Models for Retrieval-Augmented Generation over Financial Transaction Data: A Comparative Evaluation with Text Embeddings

cs.IR ·2026-08-25 ·Snehal Pal
benchmark similarity 0.65
Abstract · 7 pages, 2 figures. Code available on Kaggle (see links in paper)

Domain-specific tabular and transaction foundation models are increasingly proposed as drop-in representation learners for financial applications, but their value relative to off-the-shelf text embedding models for retrieval-augmented generation (RAG) over structured data remains underexamined. We construct a deterministic, template-generated question-answering benchmark of 1,026 questions with retrieval-level gold-document annotations over IBM's synthetic Anti-Money-Laundering (AML) transaction dataset, and compare NVIDIA's 29M-parameter Transaction Foundation Model (TFM) against five general-purpose and financial-domain text embedding models under a size- and data-matched retrieval setup, as well as against a hand-aggregated summary-text baseline. TFM substantially outperforms every size-matched generic embedder on both retrieval (Recall@5: 0.81 vs. 0.25-0.39) and end-to-end answer accuracy (66.7% vs. 27.5%). However, a simple pipeline that embeds a human-engineered aggregated summary with a generic sentence encoder matches or exceeds TFM overall, and dominates it on several low-cardinality entity categories (bank, currency, payment format). This suggests that the benefit of a transaction-native foundation model over generic text embeddings is category-dependent rather than uniform, and that careful feature aggregation can substitute for domain-specific pretraining in several practically relevant regimes. We release the benchmark generation and evaluation code.

0.22

TAPES: Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences

cs.LG cs.AI ·2026-11-03 ·Ruobing Zuo, Weiyu Han, Huilin He, Haoyu Jiang, Ze Zhao +4 more
similarity 0.64
Abstract

Transaction event sequences support diverse prediction tasks in online commerce and digital financial services. The same transaction records can yield prediction targets with different semantics, including values, changes, and relations to past events. These targets impose different constraints on history representations, but their predictability alone does not establish the downstream utility of a target set. Moreover, target-wise prediction losses do not explicitly model conditional co-occurrence among targets. To address these challenges, we propose TAPES, a task-aligned joint predictive pretraining framework for transaction sequences. TAPES constructs a finite collection of candidate target sets from admissible field-operator combinations. It selects a target set according to the downstream utility of representations learned through short joint pretraining trials under matched budgets. It then reinitializes the encoder and pretrains it on the selected targets through a joint likelihood parameterized by a shared discrete latent variable, before downstream supervised adaptation. Experiments in a real-world transaction-risk setting and on multiple public financial benchmarks spanning diverse downstream tasks demonstrate that TAPES outperforms representative baselines. Ablation studies further validate the contributions of predictive target selection and joint modeling.

0.15

CARTA: Self-Supervised Reusable Transaction Embeddings via Heterogeneous Tokenization and Cross-Attention

cs.LG cs.CL ·2026-10-17 ·Shreyas K. Chandrahas, Jose Munoz, Satish Babu Bellapu
published similarity 0.64
Abstract · 9 pages, 4 figures. Accepted at ICAIF '26 (7th ACM International Conference on AI in Finance), Milan, Italy, November 14-17, 2026

Acquiring fraud labels for payment transactions is expensive and inconsistent across tasks, and high-cardinality identifiers such as merchant names make one-hot feature pipelines infeasible at scale. We present CARTA, a self-supervised encoder that learns a compact 16-dim embedding from raw transactions without labels; the frozen embedding is then reused for label-efficient fraud probing and exploratory unsupervised risk stratification without retraining the encoder. CARTA uses type-specific projections with learned embedding tables for high-cardinality identifiers, trained with token excision and InfoNCE contrastive loss. On IEEE-CIS, its linear probe at 1% of labels exceeds a raw-feature probe at 100%; on IBM TabFormer (2M rows), its frozen embedding reaches GBDT AUPRC 0.508, above frequency encoding (0.411) and the merchant-dropping baseline (0.022), though fully-supervised CatBoost is stronger. CARTA is a label-free, reusable representation that complements task-specific supervised models.

0.37

Grounding Long Financial Text with Semantically-Enriched Tables via Multimodal Learning

cs.CL ·2026-10-27 ·Xiaoyu Hu
similarity 0.63
Abstract · Accepted to Neurips Long-context foundation model workshop

We propose a multimodal transformer framework for efficiently modeling long1 financial documents that combine structured tabular data with noisy, unstructured2 textual disclosures. The framework represents financial tables as semantically3 enriched token sequences and integrates them with long-form text through a cross-4 attention mechanism that explicitly captures cross-modal dependencies. In par-5 ticular, we augment each numerical variable with a semantic embedding derived6 from its textual definition, enabling the model to encode both numerical magnitude7 and economic meaning. This design yields a more expressive and parameter-8 efficient representation of tabular data compared to value-only embeddings. Using9 a large-scale dataset of financial statements and MD&A disclosures, we show that10 structured numerical features remain the dominant source of predictive signal;11 however, textual information provides consistent incremental gains when incor-12 porated via cross-attention rather than simple concatenation. Our results further13 demonstrate that noisy textual disclosures are most useful when conditioned on a14 stable, semantically structured tabular representation. In particular, using frozen15 variable-level tabular representations as cross-attention queries enables the model16 to weight MD&A chunks differently for each financial variable and observation.17 These findings suggest that cross-attention provides an efficient mechanism for18 table-conditioned aggregation of long-form narrative evidence.

0.22

Dynamic Multi-Byte Prediction With Hierarchical Language Models

cs.AI ·2026-08-16 ·Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant, Tomasz Limisiewicz, Sachin Kumar
similarity 0.63
Abstract

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.

0.34

ID2Pattern: Pretraining Generalizable Identity Behavior Patterns for Transaction Risk Foundation Models

cs.AI ·2026-10-30 ·Ze Zhao, Haoyu Jiang, Weiyu Han, Ruobing Zuo, Xiao Tan +2 more
similarity 0.62
Abstract · 13 pages, 6 figures

Large-scale behavioral pretraining offers a promising route to generalizable representations in online payment systems. However, high-cardinality identity fields such as card, device, and account IDs, though rich in behavioral signals, remain underexploited. Their continually growing vocabularies and limited semantic meaning make existing vocabulary-based and semantics-based encodings ill-suited. Their generalizable signal lies not in specific values, but in the identity behavior patterns they exhibit across a transaction sequence. Building on this insight, we propose ID2Pattern, a pretraining framework that brings these patterns into transaction foundation models. ID2Pattern first encodes identity fields with a Local Anonymous ID (LAID) tokenizer, which captures how each identity appears and recurs within the sequence, independent of its specific value. It then pretrains a multi-scale pattern codebook on unlabeled sequences, which abstracts LAID sequences into discrete behavior patterns for downstream prediction. Experiments on an industrial dataset with over 100M sequences and six public datasets show that ID2Pattern consistently outperforms strong baselines, improving industrial by 4.13% and public datasets by 8.83% on average. Notably, ID2Pattern remains equally effective even when all identity values are unseen during training, highlighting its potential to generalize to new users and markets.

0.33

Text as an Index for Zero-Shot Cross-Domain Knowledge Tracing

cs.CL cs.CYcs.LG ·2026-10-30 ·Yin Yi, Haiping Ma, Siyu Song, Ziwen Wang, Shangshang Yang
similarity 0.62
Abstract · 10 pages, 5 figures

Knowledge tracing (KT) predicts a student's next response from their history. New platforms often have question texts but no response logs for training, motivating zero-shot cross-domain KT. Existing transfer and cold-start methods use question semantics, but cross-platform semantic discrepancies limit transferability. We propose TIZSKT, which separates behavioral sequence modeling from domain-local historical evidence construction and uses text as an index rather than a directly transferable predictive representation. The Mastery-Dynamics Encoder learns correctness dynamics without question identifiers or text embeddings. Domain-Local Question Addressing builds fixed question addresses using a frozen text encoder and an unsupervised per-domain projection. The Text-Addressing Kernel uses these addresses to aggregate the current student's previous outcomes into a question-specific historical correctness summary. Cross-Segment Continuation retains access to earlier responses without extending the behavioral encoder's input sequence. Evidence-Volume Read-Out preserves the total kernel mass discarded by normalization and combines its log-compressed value with the behavioral state and historical correctness summary to predict the next response. All trainable parameters are learned from source domains; target inference uses public question texts and the current student's observed history without target-domain training. Leave-one-domain-out experiments on three public datasets show that TIZSKT achieves the highest mean AUC and accuracy among evaluated baselines on all target domains. Ablations confirm the contributions of text addressing and cross-segment continuation.

0.21