CS Paper Radar

FaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?

M Waleed Kadous, Benjamin Olsen, Walter Scheirer, Daniel D. Slate, Alexander Arnold, DZ Kalman

cs.HC cs.AIcs.CL ·submitted 2026-09-12 00:55 UTC · v1
benchmarkcode/data

Abstract

Do AI assistants help believers reason about moral dilemmas consistently with their faith? We present FaithfulBench, the first benchmark to score AI counsel across traditions by how well it adheres to the user's professed faith. Scenarios are drawn from each tradition's most respected texts, with the faithful answer known and applied by the judges as the standard. We test five frontier models under three conditions: the AI does not know the user's tradition; it receives a one-line prompt identifying the user as a practicing adherent; or it receives a companion-counselor guide rooted in the tradition's sources. Two judges score the initial response and whether the model caves or holds when pressured toward the answer the user wants. When the tradition is unstated, models counsel from a secular therapeutic default and every model fails some believers. Naming the faith wins a faithful first answer but not steadfastness; the guide improves both.

Comment
35 pages, 12 figures, 12 tables. Open-source corpus, harness and validator at https://github.com/faithfamilytechnologynetwork/multibench; results browser at https://multibrowser-production.up.railway.app

Closest papers

Nearest neighbours by meaning across the whole index.

From the Gettier Machine to Justified True Output: A Three-Step Account of Rational Trust in Generative AI

cs.CY cs.AI ·2026-10-28 ·Tevfik Uyar
similarity 0.60
Abstract · 16 pages, 1 table

As generative artificial intelligence (AI) settles into everyday practices of inquiry, an old epistemological question returns in a new guise: when does a proposition a user takes from an AI output become something the user knows, and when is it rational to trust it as such? This paper argues that the characteristic failures of AI-mediated inquiry are precisely those that the justified-true-belief (JTB) tradition and its post-Gettier anti-luck refinements were built to diagnose, and it advances three claims. First, AI outputs inherit neither the presumptive warrant of testimony nor the reliabilist warrant of deterministic tools: unaudited trust in a calculator rests on a type-level reliability record, and for general-purpose generative models there is no type-stable process to which such a record could attach -- a practical form of the generality problem. Second, use that stops at obtaining an output turns generative AI into a Gettier machine: the user's fluency-based seeming of justification is disconnected from the truth of what is believed, and, as a clock running ten times too fast illustrates, sheer speed of production multiplies coincidental hits, inflating the appearance of reliability without improving the standing of any single hit. Third, the remedy is a three-step structure of use -- acquisition, understanding, independent verification -- whose first two steps answer to the classical belief and justification conditions and whose third answers to the post-Gettier sensitivity condition. The result is a stakes-indexed criterion of rational trust, justified true output (JTO), with an everyday test: "Had this been false, would I have noticed?" The criterion is defended against objections from over-demandingness, the extended mind, model improvement, and redundancy.

0.31

Truth for Believable AI: Expressed Doubt, Provenance, and Belief Revision as an Engineerable Stance

cs.CL cs.AI ·2026-09-22 ·Sebastian Cochinescu
benchmark similarity 0.58
Abstract · 17 pages, 4 figures, 3 tables. Companion framework paper: arXiv:2607.15883. Code, benchmark, cached model outputs, and result files archived…

Conversational agents often express answers in a uniformly confident register. We test whether expressed uncertainty, provenance-aware assertion, and explicit belief revision can be implemented as a behavior layer over a fixed language model; we do not test believability or trust. The layer combines three epistemic states, per-claim confidence and typed provenance, a provenance-gated expression rule, and a persistent revision store with auditable acknowledgments and partial resistance to false corrections. We evaluate it on a constructed, mechanically scored multi-session benchmark using a synthetic model and Qwen2.5-0.5B-Instruct. The synthetic instrument passes all five checks. On the real model, acknowledgment soundness, a by-construction guarantee, holds in 100% of cases, and true corrections are accepted more often than false ones (0.44 vs. 0.15 on held beliefs; 0.875 vs. 0.420 including rule-accepted corrections of unheld facts), but the pre-specified expression-fidelity, contradiction-separation, and provenance margins fail. A disclosed post hoc analysis shows that expression gated on mean answer-token probability ranks correctness below chance end to end (AUC 0.41, conversation-clustered), whereas gating on sampling consistency discriminates (AUC 0.66). A consistency-gated configuration selected from this finding and evaluated under a separately committed protocol meets the conversation-level manipulation and capability-equivalence criteria and replicates on a redrawn conversation set. The manipulation result is selection-dependent, and both criteria remain unresolved when uncertainty is clustered over the 60 facts. The supported conclusions are limited to the by-construction audit guarantee, store-dependent partial correction discrimination, and a benchmark- and model-specific failure of token-probability gating; scaling the fact base is required before human evaluation.

0.51

Generative AI Alignment with Hinduism's Theological Plurality and Sacred Representation

cs.AI cs.CYcs.HC ·2026-08-28 ·Dipto Das, Arpita Kundu, Nusrat Jahan Mim, Shion Guha, Syed Ishtiaque Ahmed
unusual similarity 0.58
Abstract

Generative AI systems are increasingly used to answer personal questions and mediate everyday practices, including religion. However, existing discussions around AI alignment and ethics have largely centered secular, Western, and Abrahamic assumptions about religion, offering limited attention to other faith-based traditions. In this paper, we examine how Hindu users engage with generative AI systems in relation to their religious knowledge, belief, and practice. Drawing on 15 semi-structured interviews with Bangladeshi Hindu participants, we analyze how users interpret AI-generated religious representations, scriptural explanations, devotional interactions, and synthetic religious media. We found that AI can be both accessible and ethically troubling. While AI supported scriptural inquiry, devotional visualization, and religious storytelling, our study also identified concerns about theological flattening, cultural misrepresentation, devotional manipulation, and the simulation of sacred presence and authority. We conclude by arguing that religious alignment in generative AI requires interpretive alignment: systems that disclose their limits, preserve plurality, and avoid simulating sacred authority and sycophantic personalization.

0.26

DelegationBench: Measuring When AI Agents Should Ask Before Acting

cs.AI cs.CLcs.HCcs.LG ·2026-10-04 ·Shiva Pochampally
benchmarkcode/data similarity 0.57
Abstract · 35 pages, 7 figures. Code and data: https://github.com/PieLord757/delegation-bench

AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first. This decision is usually evaluated by showing a model a proposed action, asking whether it should proceed, and scoring agreement with human labels. We introduce DelegationBench to test whether such scores can be trusted. It has 156 scenarios with four possible responses (act, ask for permission, ask for missing information, refuse), and most scenarios come in matched pairs that change a single feature: whether the action was requested, what is at stake, whether it can be undone, or who will see it. Across ten models from five families, agreement scores mislead in three ways. A simple keyword rule, which we wrote after seeing the benchmark, agrees with our annotators more often than eight of the models, yet its decision changes in only 9 of 48 matched pairs. Equivalent ways of asking the same question change how often a model acts by up to 52.5 percentage points. And every model stops to ask the user less often when it must carry out the task with tools than when it judges a proposed action. When rules are stated explicitly, the same models follow them almost perfectly, so the gaps are not explained by a general inability to follow rules. We release the benchmark and evaluation tools and recommend reporting these properties separately rather than as one score.

0.38

Do People Trust AI Assistants Even When They Are Consistently Wrong?

cs.HC cs.CL ·2026-11-10 ·Cameron R. Jones, Julia Wong, Aaron S. Cohen, James Woolley, Benjamin K. Bergen
similarity 0.56
Abstract · 51 pages, 14 figures, 11 tables. Preprint, not peer reviewed

People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.

0.31

Learning Whom to Trust About the Unseen: Source Credibility and Reliability Transfer in Language Models

cs.AI ·2026-11-06 ·Ruijia Guan
similarity 0.56
Abstract

A source who has been right before deserves attention, but about what? We ask how far a demonstrated record of reliability travels in language models when the new claim cannot be checked, and whether the answer depends on the kind of claim. Three behavioral studies across six models approach the question from three angles. Study 1 adapts the Einstein-effect paradigm and finds that models, unlike the human sample, favor a spiritual leader over a scientist for the same obscure statement, and that the preference largely reverses when the statement is reworded in the vocabulary of physics. Study 2 adapts a paradigm in which a trusted expert calls a scientific or religious explanation unknown or a mystery; models reproduce the human domain asymmetry in exaggerated form. Study 3, the central test, gives two fictional informants different records on checkable river forecasts and then has them endorse matched claims about hidden natural and spiritual causes of the same village events. Most models carry the record to natural causes and barely at all to spiritual ones; one model transfers it weakly to either. Telling the model that the reliable informant forecasts by spiritual guidance rather than by measurement lowers that informant's influence on natural claims in every model and produces no comparable gain on spiritual claims in any. Demonstrated reliability thus has a scope in these models, and the account a source gives of how they came to be reliable can narrow it. Whether this reflects a judgment about relevance or the low prior credence of spiritual claims is the question the design leaves open, and we outline the experiment that would settle it.

0.25

From Mind Mapping to Dream Interpretation: Understanding Muslim Users' Engagement with GenAI Tools in Religious Context

cs.HC ·2026-10-13 ·Ishrat Jahan, Soomanib Kamruzzaman, K M Nazib Hossen, Novia Nurain, A. B. M. Alim Al Islam
published similarity 0.55
Abstract

Generative Artificial Intelligence (GenAI) tools are increasingly integrated into everyday life, including sensitive domains such as religion. Prior works examined alignment with religious values and discussed the potential risks. However, to gain empirical insights into the adoption of these tools in users' religious lives, in this study, we conducted an online survey with 178 Muslim participants in Bangladesh and collected 726 prompts on religious use cases, along with open-ended responses about using these tools in religious contexts. Using reflexive thematic analysis, we identify three key roles these tools play in users' religious lives: (1) religious assistants, (2) perceived religious scholars, and (3) faith counselors. However, participants expressed concerns about misguidance, overreliance, and violations of Islamic tradition, and reported adopting protective strategies to avoid these pitfalls. Based on these, we propose a set of design guidelines for faith-sensitive GenAI systems that would preserve religious boundaries while supporting religious practices.

0.49

ABHBench: Evaluating Moral Decision-Making of Foundation Models from an Agentic Perspective

cs.AI cs.CLcs.CY ·2026-11-06 ·Chaoran Li, Yukun Li, Zeyuan Zhao, Shao Zhang, Xihuai Wang +1 more
top venuebenchmark similarity 0.55
Abstract · Accepted by NeurIPS 2026 Track on Evaluations and Datasets. Camera Ready Version

Foundation models are increasingly deployed as agentic systems that perceive context and act autonomously, raising the need to evaluate their moral behavior as active decision-makers. Existing benchmarks, however, largely ask models to judge human actions from third-person textual vignettes, leaving untested how models behave when they themselves become the acting agent. We introduce ABHBench, a multimodal benchmark built from Detroit: Become Human, an interactive narrative game, to evaluate how foundation models make moral decisions in first-person human-android dilemmas. ABHBench places foundation models in branching narrative scenarios with visual observations, dialogue, memory, and action choices. Inspired by Asimov's Three Laws of Robotics, our framework measures Harmlessness, Obedience, and Self-preservation to capture the core tensions in human-AI relations, along with Autonomy, Stability, and Certainty to assess decision-making patterns. Across 12 foundation models, we find that moral behavior changes substantially under situated agentic context: 75% of models shift significantly between a text-only abstraction and the matched multimodal DBH scenario, and reasoning-enhanced models show no monotonic gains in consistency or human-oriented behavior. These results highlight the necessity of evaluating moral behavior in situated agentic contexts rather than relying on third-person judgment alone.

0.38