CS Paper Radar

MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting

Dong Shu, Yanguang Liu, Huopu Zhang, Saisai Hu, Haiyan Zhao, Hekun Huang, Mengnan Du

cs.LG cs.AI ·submitted 2026-09-29 20:40 UTC · v1
benchmark

Abstract

Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals. However, existing financial benchmarks are often limited to unimodal inputs or single-task settings, making it difficult to evaluate whether multimodal large language models (LLMs) can support real-world financial analysis. In this paper, we introduce MM-FinEval, a novel benchmark designed to evaluate multimodal LLMs across multiple financial tasks. MM-FinEval spans a diverse timeline from 2019 to 2022. The entire proposed dataset contains 2,045 S\&P 500 conference earning calls as inputs and 12 financial task labels as outputs. Each input contains three modalities: a word-to-word text transcript of the earning call, the corresponding presentation slides used during the call, and the entire audio recording. To establish a rigorous evaluation framework, we analyze 19 baseline models across three distinct model categories: Image-Text, Audio-Text, and Any-to-Any configurations. We observe that small-size Any-to-Any models processing all three modalities achieve strong performance, even when compared against larger proprietary models restricted to two-modality inputs. This indicates that our tri-modal dataset design introduces useful, non-redundant information. These results validate that text, audio, and visual data serve as important, complementary signals that mimic the decision-making process of expert human analysts.

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context

cs.LG cs.AI ·2026-09-14 ·Peng Chen, Zhihao Zhuang, Hongzhou Chen, Junhao Huang, Aiping Yang +4 more
benchmark similarity 0.69
Abstract · preprint

Most time series forecasting benchmarks remain numerical-centric and provide limited support for evaluating contextual information that shapes real-world temporal dynamics. Existing multimodal benchmarks also suffer from limited data and context coverage, fragmented evaluation settings, and overreliance on aggregate evaluation. In this paper, we propose \textbf{MUSE-Bench}, a unified benchmark for multimodal time series forecasting with heterogeneous context. It comprises fourteen datasets across eight domains and six types of context: metadata, events, holidays, news, images, and numerical covariates. We evaluate diverse forecasting paradigms, including statistical, data-specific, foundation, multimodal, and general-purpose LLM forecasting methods under shared non-overlapping forecast windows, common target observations, and consistent point and probabilistic metrics. Extensive experiments yield three main findings. First, numerical time series foundation models dominate the overall ranking, while Aurora, the evaluated multimodal foundation model, trails the leading numerical TSFMs but outperforms all evaluated data-specific models. Second, ablations show that external context improves the four evaluated context-aware models, whereas incorrect or temporally misaligned context degrades performance. Third, general-purpose LLMs perform poorly as direct forecasters, and LLM-guided refinement does not yield consistent improvements. MUSE-Bench enables systematic evaluation of how forecasting models utilize context and provides a foundation for future multimodal forecasting research.

0.41

Grounding Long Financial Text with Semantically-Enriched Tables via Multimodal Learning

cs.CL ·2026-10-27 ·Xiaoyu Hu
similarity 0.68
Abstract · Accepted to Neurips Long-context foundation model workshop

We propose a multimodal transformer framework for efficiently modeling long1 financial documents that combine structured tabular data with noisy, unstructured2 textual disclosures. The framework represents financial tables as semantically3 enriched token sequences and integrates them with long-form text through a cross-4 attention mechanism that explicitly captures cross-modal dependencies. In par-5 ticular, we augment each numerical variable with a semantic embedding derived6 from its textual definition, enabling the model to encode both numerical magnitude7 and economic meaning. This design yields a more expressive and parameter-8 efficient representation of tabular data compared to value-only embeddings. Using9 a large-scale dataset of financial statements and MD&A disclosures, we show that10 structured numerical features remain the dominant source of predictive signal;11 however, textual information provides consistent incremental gains when incor-12 porated via cross-attention rather than simple concatenation. Our results further13 demonstrate that noisy textual disclosures are most useful when conditioned on a14 stable, semantically structured tabular representation. In particular, using frozen15 variable-level tabular representations as cross-attention queries enables the model16 to weight MD&A chunks differently for each financial variable and observation.17 These findings suggest that cross-attention provides an efficient mechanism for18 table-conditioned aggregation of long-form narrative evidence.

0.22

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

cs.LG cs.AI ·2026-10-13 ·Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou +2 more
top venuebenchmarkcode/data similarity 0.68
Abstract · Accepted at the NeurIPS 2026 Evaluations and Datasets Track. Project page: https://windowso.github.io/MMTA/

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.

0.40

TSFM Meets LLM: Context as Covariates

cs.LG cs.AI ·2026-11-03 ·Hemanth Ram, Sai Shankar Narasimhan, Shubhankar Agarwal, Sandeep Chinchali
code/data similarity 0.67
Abstract

Textual information is ubiquitous in real-world time series datasets, appearing as news, reports, and annotations that hold key predictive signals not deducible from numerical history alone. Large Language Models (LLMs) interpret text but lack numerical precision, while Time Series Foundation Models (TSFMs) are quantitatively accurate but cannot reason about the effects of text. Moreover, the lack of large-scale paired multimodal datasets limits progress in bridging this gap. In this work, we address this challenge by providing a scalable approach for generating realistic multimodal datasets by annotating real-world time series with LLMs. Building on this, we propose Context as Covariates (CoCo), a forecasting framework in which an LLM distills its textual reasoning into numerically grounded forecast and confidence covariates that guide a TSFM backbone, thereby exploiting the strengths of both LLMs and TSFMs. We align the two models in stages, first fine-tuning the LLM via Group Relative Policy Optimization (GRPO) with a reward tied to covariate-induced improvement in forecasting performance, and then fine-tuning the TSFM on covariates generated by the aligned LLM. Our experiments show that models trained on our synthetic multimodal corpus generalize to real benchmarks spanning finance, economics, weather, traffic, and security. Notably, CoCo with Qwen3-4B-Instruct-2507 as the LLM and Chronos2 as the TSFM achieves up to 12% MAE improvement over state-of-the-art TSFMs and LLM forecasters, such as GPT-4o. Code, checkpoints, and the evaluation datasets are publicly available.

0.22

TimeWorld: Towards Action-Conditioned Multimodal World Models for Time Series Forecasting

cs.LG ·2026-10-26 ·Hang Ni, Weijia Zhang, Hao Liu
benchmark similarity 0.67
Abstract · under review

Time series forecasting supports decision-making in web applications such as online retail and network management, where planned actions can reshape future dynamics. World models provide a natural paradigm for predicting such action-dependent changes beyond extrapolating historical patterns. However, existing time series world models (TSWMs) typically rely on fixed, domain-specific covariate schemas that limit flexible intervention specification, while multimodal forecasters treat text primarily as passive predictive context. To investigate these gaps, we introduce TSWM-Bench, a cross-domain benchmark for multimodal TSWMs spanning seven real-world datasets, with textual interventions grounded in recorded action--outcome data. Our evaluation reveals that conventional multimodal forecasters underperform history-only time series foundation models (TSFMs) despite dataset-specific training, whereas simple text-prefix adaptations improve forecasting across all evaluated TSFM backbones. These findings suggest that pretrained TSFMs encode implicit world knowledge of temporal dynamics, offering a strong foundation for action-conditioned modeling beyond conventional multimodal fusion. Building on this insight, we propose TimeWorld, a plug-in framework that adapts pretrained TSFMs through explicit action-response modeling. Specifically, the Prototype-Guided Latent Modulation (PLM) module grounds textual actions in temporal dynamics by composing representative response prototypes to modulate future representations based on the action and historical patterns. The Action Contrastive Learning (ACL) component strengthens action-specific response modeling by contrasting these updates under alternative interventions sharing the same history, distinguishing action types while preserving relative intensity relationships.

0.06

Frontiers in FinTech: Multimodal Foundation Models for Financial Reporting and Decision Science

cs.CE ·2026-08-24 ·Yulu Huang, Niannian Yu, Yaxin Yang, Yong Huang
similarity 0.67
Abstract

Financial information no longer arrives in a single format. Research reports come as PDFs, financial statements live in spreadsheets, market trends are captured in images, and policy documents reach analysts as scans, each carrying part of the picture the others cannot supply. Accounting information systems built around single-modality extraction pipelines and rule-based tools therefore struggle to assemble the full picture, slowing financial statement analysis, complicating audit evidence corroboration, and limiting investment decision support. This study presents FinVision, a multimodal large language model that unites vision-language models with domain-specific financial reasoning. Instead of processing documents in isolation, FinVision reads text, tables, and images together, converts them into consistent structured data, and verifies cross-modal agreement, in the same spirit as auditors corroborating evidence from independent sources. The model is trained in two stages, pre-trained on large-scale public financial corpora and fine-tuned on institution-specific investment data, so it can apply established valuation methodologies and audit risk assessment frameworks while outperforming zero-shot and single-stage baselines. A natural-language decision pipeline lets users describe what they need and turns those descriptions into executable workflows, supporting portfolio optimization, real-time risk monitoring, and refinement through multi-turn dialogue. Across 200 listed companies, FinVision reduced valuation error by 19 percent relative to the strongest baseline; a user study with 48 accounting and investment professionals reported a 51 percent reduction in task completion time. These results carry implications for audit automation, financial reporting quality, and more inclusive access to expert-level financial analysis.

0.08

XForecast: The KDD 2026 Finance Day Challenge on News-Grounded Financial Forecasting

cs.LG ·2026-10-12 ·Jaehoon Lee, Jun Seo, Seunghan Lee, Tae Yoon Lim, Dongwan Kang +8 more
benchmark similarity 0.66
Abstract · 6 pages

Financial forecasting has traditionally relied on numerical time-series (TS) signals, while textual information such as news remains largely underutilized. We introduce XForecast, a benchmark and competition for news-grounded financial forecasting, where the task is to predict stock price movements four weeks ahead using both historical price data and paired news signals. XForecast combines stock price data with curated financial text generated using FinTexTS, a multimodal dataset generation method. It covers 100 large-cap stocks from 2019--2023, with training on 2019--2022 and evaluation on 2023. To better evaluate models that leverage textual signals, we adopt a weighted hit rate metric that places greater emphasis on instances where news information is expected to have stronger influence. The XForecast Challenge, hosted on Kaggle as part of KDD 2026 Finance Day, aims to benchmark methods for integrating financial TS and textual information.

0.09

FinVision: Multimodal AI for Visualizing Financial Narratives

cs.CV cs.LG ·2026-11-15 ·Yixuan Tang, Hui Wang, Yi Yang, Allen H. Huang
similarity 0.66
Abstract

Financial narratives contain rich information about firms' performance and business conditions. Conventional text analysis often summarizes this information into numerical representations, which may not fully preserve the dynamic structure described in the narrative. In this study, we develop FinVision, a domain-specific multimodal generative model that translates financial narratives into visual representations of contemporaneous market dynamics. Using 87,940 paired observations of Management Discussion and Analysis (MD&A) disclosures and stock price charts, FinVision learns the average relationship between a firm's narrative account of a fiscal year and its stock-price dynamics over the same period. Given an MD&A alone, the trained model generates a text-implied stock chart, extending financial text analysis beyond numerical representations toward text-to-visual generation. We validate FinVision in two ways. First, held-out tests show that the model generates stock-chart representations that closely resemble observed charts and outperform direct LLM prompting, general-purpose text-to-image models, and the same architecture without financial-domain adaptation. Second, we examine the economic validity of the generated representations by comparing text-implied and observed stock charts. The resulting discrepancy is systematically associated with weaker investor responses around subsequent 10-K filings and with worse subsequent market and operating outcomes. Overall, our findings show how domain-adapted multimodal generative AI can create a new visual representation of financial text and provide an additional modality for studying business communication.

0.10