CS Paper Radar

How People Evaluate AI-, Expert-, and Peer-Style Financial Advice

Aryan Ramchandra Kapadia, Eshwar Chandrasekharan, Koustuv Saha

cs.HC cs.AIcs.CLcs.CYcs.SI ·submitted 2026-08-10 02:19 UTC · v1

Abstract

As generative AI increasingly becomes a common source of daily decision-making, including financial choices, it is critical to understand how people evaluate AI-generated financial advice. We conducted a preregistered vignette experiment (N = 285) in which substantive financial content---including facts, numerical values, recommendation direction, and core reasoning---was held constant while communication style varied across AI Financial Assistant (AI), Certified Financial Planner (Expert), and Online Community Forum (OC) advice. Displayed source attribution was independently manipulated through correctly labeled, unlabeled, and mislabeled conditions, allowing us to separate attribution effects from source-specific communication cues. Expert advice was rated more favorably than AI advice on 9 of 10 outcomes (|d|=0.20--0.47), and this advantage remained visible without source labels, where Expert advice outperformed AI advice on 8 of 10 outcomes (up to d=0.60). Correct labels added limited differentiation, whereas mislabeling increased ratings of AI advice for situational fit and overall quality (d=0.42 for each) and attenuated the Expert advantage in situational fit (d=-0.36). Descriptive analyses further showed that AI advice was most responsive to displayed attribution and, conversely, that advice-style differences were most visible under an AI label. These findings show that financial-advice evaluations are shaped jointly by displayed attribution and message-level communication cues. We position disclosure not as a neutral transparency mechanism, but as an interpretive frame whose accuracy and interaction with message cues can shape trust and reliance.

Closest papers

Nearest neighbours by meaning across the whole index.

Trustworthy FinAInce: Unpacking How AI-Mediated Financial Advice is Judged

cs.HC cs.AIcs.CLcs.CY ·2026-09-17 ·Aryan Ramchandra Kapadia, Eshwar Chandrasekharan, Koustuv Saha
similarity 0.88
Abstract

As generative AI is increasingly used as a source of personal financial guidance, understanding how people appraise such advice is important for supporting appropriate reliance. We conducted a randomized vignette experiment with 285 U.S. adults across eight financial decisions, independently varying three advice styles---AI, expert, and online community---and displayed source labels while holding the underlying recommendation consistent. Advice style most strongly shaped message and safety appraisals, Expert labels selectively increased perceived source knowledge, and decision context primarily shaped risk and safety appraisals. These appraisals were associated with downstream judgments, with models explaining 69.2% of overall quality, 75.9% of trust, and 82.9% of intended reliance. Expert-style advice also remained most preferred when shown without source labels. Our findings have implications for understanding financial advice evaluation, distinguishing the roles of advice style and source labels, and designing financial AI that supports grounded evaluation rather than simply maximizing trust.

0.09

Annie, Are You Okay? How Style- and Context-Based Personalization Shape AI-Assisted Decision-Making

cs.HC cs.AI ·2026-09-21 ·Hasibur Rahman, Benjamin R. Cowan, Smit Desai
similarity 0.70
Abstract

As people turn to generative AI for financial advice, these systems can personalize how they communicate and what they say. Whether these forms of personalization shape decisions differently remains unclear. We conducted a preregistered 2 x 2 between-subjects factorial experiment (N=240): participants ranked three comparably viable stocks, discussed them with an AI, and reranked them. Participants perceived both forms of personalization, but only context-based personalization reliably changed ranking behavior: it increased reconsideration and moved rankings toward the AI's assigned recommendation. Participants felt more influenced without judging the AI as more correct, trustworthy, intelligent, likeable, or high-quality. Those initially farther from its recommendation moved more toward it while judging its advice less correct; exploratory analyses suggest greater susceptibility among lower-expertise participants. These findings show how personalized AI can steer decisions among defensible options with only a minimal evaluative trace, raising concerns for the design and governance of personalized decision support.

0.09

The Differential Effects of AI Abstention on Advice Acceptance

cs.HC ·2026-10-19 ·Nicholas Wolczynski, Yotam Liel, Maytal Saar-Tsechansky, Lior Zalmanson
similarity 0.68
Abstract

AI advising systems are typically designed to always provide a response. Recent work has focused on developing systems that withhold advice in some instances, such as when advice quality is questionable, to improve the overall quality of guidance when it is given; a design paradigm we refer to as AI abstention. While substantial work has focused on optimizing when models should abstain, little is known about how users respond to systems that abstain. In a preregistered experiment, participants made repeated financial decisions, assisted by one of two advising regimes: an AI that advises selectively or one that advises on every decision. We find that the same AI abstention regime produces differential effects depending on the user: it increases advice acceptance among users with high perceived expertise and tends to decrease it among users with low perceived expertise. This differential effect is accounted for by the cognitive burden users face when repeatedly reconciling AI recommendations with their own judgment; AI abstention appears to relieve this burden among users with high perceived expertise, and in doing so increases their receptivity to advice. Further, AI abstention mitigates the decline in advice acceptance observed among expert users over repeated interactions. Exploratory evidence also suggests it increases appropriate acceptance where advice would improve the decision, increasing the downstream accuracy of perceived experts. The AI abstention regime is thus a first-order design property of AI advisory systems, and one that must be calibrated to the users a system serves.

0.19

FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

cs.AI q-fin.CP ·2026-09-28 ·Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim +15 more
benchmark similarity 0.64
Abstract · preprint

Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. In long-horizon loops that follow the expert guidance, a writer agent researches every expected value and a reviewer agent verifies it, and failures escalate to a human. On three expert-authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator while stating the expert rubric's expected value for more criteria, their scores agree with human grading, and in-house analysts prefer them in a blind review. The released 100-query FinAutoRubric Benchmark, built from in-house analysts' key questions across 78 tasks and eight asset classes, shows that rubrics from an earlier model generation still leave headroom for a later one.

0.30

Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets

cs.HC cs.AIcs.LG ·2026-11-03 ·Honson Ling, Kidus Admassu, Stephanie Rosenthal
similarity 0.63
Abstract · 20 pages, 7 figures, 9 tables

Evaluating and improving domain-specific LLM responses requires high-quality reference data and rubrics, yet expert authoring is slow and expensive, and synthetic datasets may fail to capture practitioner preferences unless they are grounded in expert judgment. We present SME-C (Subject Matter Expert Critique), a human-AI workflow that captures expertise through lightweight critiques of candidate responses and complements it with LLM-driven analysis and refinement to iteratively curate response datasets and evaluation rubrics. We validate SME-C in the domain of financial business briefs, collecting 3,853 critiques from 411 finance professionals on 520 queries across two survey rounds. We evaluate two LLM-driven refinement strategies: one guided by the experts' written critiques and one that infers critique solely from their preference votes. In a blind validation survey with 100 additional professionals, we found that both strategies significantly improved quality over the experts' top-ranked baseline responses. The gains were concentrated among weaker original responses, and the two strategies excelled in different ways: critique-based refinement produced more domain-grounded improvements, whereas vote-based refinement was more effective at identifying and removing irrelevant content. The resulting rubrics and refined dataset support reproducible offline evaluation and benchmarking of LLMs, suggesting that structured expert critique is a viable alternative to expert authoring for building high-quality, domain-specific evaluation resources more quickly, at scale, and at lower cost.

0.12

People prefer AI to human advice in multi-turn conversations but neither changes incentivised behaviour

cs.HC ·2026-11-12 ·Maximilian Maier, Tsz Kwan Emmy Wong, Jiahao Eric Yang, Falk Lieder, Vanessa Cheung
similarity 0.62
Abstract

People increasingly ask AI chatbots for advice, yet large language models (LLMs) are sycophantic, overconfident, and can amplify human cognitive biases. In this article, we investigate how people perceive advice from humans versus LLMs and whether AI advice reshapes incentive-compatible donation decisions. Using Carrier, a new experimental platform, we designed an experiment where participants made a donation decision after discussing it with an AI chatbot advisor, another human, or receiving no advice. Participants judged LLM advisors to be significantly more rational, trustworthy, convincing, and sycophantic than human advisors. Analysis of chatlogs showed that LLMs were more engaged than human advisors: they shared more facts and more often stated their opinion and justified their recommendations. Despite this, donation decisions did not vary across conditions.

0.32

Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?

cs.AI ·2026-10-28 ·Huangfei Jiang, Anning Hu, Kai Sheng, Yuchen Cai, Rui Ge +5 more
benchmark similarity 0.60
Abstract

Financial agents must form forecasts and investment judgments from decision-time evidence, with predictions evaluated against subsequent outcomes. Existing benchmarks assess financial knowledge, research workflows, or forecasting, but evaluating the full research process requires a strictly frozen information set and professional evaluation criteria. We introduce Investor's Last Exam (ILE), an equity-research benchmark led by experienced buy-side investors and fund managers. Its three components are a versioned financial database spanning 3.43 million document versions and nine source categories, supplemented by structured market data, with validated timestamps governing retrieval and computation; a task taxonomy organized around professional analyst deliverables; and an expert-led framework linking scoring criteria to sources, calculations, and assumptions through financial dependency graphs. Evaluation spans 86 assignments and jointly assesses research quality, forecast accuracy, and simulated investment performance. GPT-6 Astra and Claude Opus 5.5 lead overall at 68.1 and 66.9 out of 100, yet even Astra scores only 52.5 on Forecast & Model. Our results show that even state-of-the-art models fail to translate overall research capability into predictive performance. ILE shifts the evaluation unit from isolated capabilities to the evidence--assumption--judgment chain, allowing researchers to locate where analytical advantages fail to translate into forecasts and investment judgments, and providing a traceable, diagnostic foundation for training and evaluating financial agents.

0.32

When the AI Leaves the Tailorshop: Measuring What an LLM Advisor Leaves Behind in Complex Problem Solving

cs.HC cs.AI ·2026-09-14 ·Robin Welsch
similarity 0.59
Abstract · 40 pages, 14 figures, 6 tables, including appendices

Complex problem solving depends on acting effectively and understanding how a system works. AI advice may support these outcomes unequally. Two preregistered experiments compared participants managing a simulated clothing factory with and without an LLM advisor. Across studies, AI-supported participants reported greater confidence and understanding with less effort. In the first study (N=200), assistance increased company value but produced no detectable prediction-accuracy difference. After withdrawal, previously supported participants outperformed controls when decisions were scored against repeating previous choices, but not default settings. Within the AI-supported group, more frequent recommendation alterations predicted better unaided performance. In the second study (N=198), AI-supported participants went bankrupt less often and showed a small knowledge advantage in the registered analysis, largely associated with remaining solvent. More frequent recommendation alterations predicted higher knowledge within the AI-supported group. Applied HAI evaluation should assess users' understanding and independent capability alongside the performance achieved with AI support.

0.38