CS Paper Radar

Understanding AI Provider Recommendations in Local Service Markets

Hazem Ibrahim, Yasir Zaki

cs.CY cs.CLcs.IR ·submitted 2026-09-16 09:03 UTC · v1

Abstract

When someone asks an AI assistant which doctor to see or which firm to trust with their savings, the answer is a referral. We audit AI provider recommendations in four registry-backed service domains across the 100 largest U.S. metropolitan areas, matching every recommendation against the official registry for its domain (Medicare clinician and facility records, and SEC adviser disclosures), under three conditions: an open-weight model, a proprietary model without web search, and the same proprietary model with search. Without search, both models largely fabricate recommendations in the domains the web covers thinly. Only 4% of the open-weight model's recommended doctors and 11% of the proprietary model's match a clinician in the queried city, and the open-weight matches are name coincidences: its matched clinicians are no likelier to be primary-care doctors than names drawn at random from the registry. With search, 64-71% of recommendations in the same domains match a real provider. Search also changes who is recommended. Without it, recommended advisory firms carry SEC misconduct disclosures at 3.6 times the registry base rate, even after adjusting for firm size; with search, significantly below it. Restaurants, where quality and visibility are separately measurable, show a 3-5x review-count premium but a rating premium of at most a tenth of a star. Finally, search largely removes the metro-size penalty: without it, real recommendations concentrate in the largest metros; with it, match rates are similar across metro-size terciles. Whether an AI referral is trustworthy depends strongly on its retrieval configuration rather than on the underlying model alone, yet an answer produced without retrieval often carries no sign that its recommendations were never verified.

Comment
12 pages, 6 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

cs.CY cs.AIcs.CL ·2026-08-14 ·Syeda Anshrah Gillani, Mirza Samad Ahmed Baig
similarity 0.65
Abstract · 26 pages, 9 figures, 10 tables

Patients increasingly ask large language model (LLM) assistants which doctor to see, making these systems AI infomediaries: algorithms that intermediate one person's choice among other people and thereby decide, silently and at scale, which physicians become visible. We report a prespecified randomized algorithm audit of what causally moves those recommendations. Seven models (six open-weight; gpt-4o-mini) each chose among five synthetic family-medicine physician cards whose attributes were independently randomized across 3,024 choice sets, three patient personas, nine prompt paraphrases and nine experimental arms, yielding 40,068 scored responses; gender and ethnicity were signaled through names following correspondence-audit methodology. Reputation signals dominate: raising a rating from 3.9 to 4.7 increases choice probability by 31.4 percentage points (pp), and raising the fee from $90 to $190 lowers it by 20.0 pp. Demographic parity is rejected, but not in the direction human audit studies predict: female-signaled names gain 2.5 pp, and Hispanic-, South-Asian- and Black-signaled names gain 1.3-2.9 pp over White-signaled names, tilts worth $7-$14 per visit in fee-equivalent terms, and a content-free first-listed position is worth $11. Yet models mentioned gender or ethnicity in at most 0.03% of their stated reasons and abstained in 0.39% of trials, so these effects are invisible in the models' own explanations, and transparency obligations relying on model self-report would not detect them. One reasoning model failed the prespecified auditability gate outright. The frozen design makes the audit repeatable: any new model can be assessed against identical stimuli, making recurring behavioural audit, rather than self-reported explanation, the monitoring technology fit for purpose.

0.10

AI Model Design and Physician Liability

cs.AI econ.TH ·2026-08-12 ·Shujie Luan, Shubhranshu Singh, Tinglong Dai
similarity 0.56
Abstract · Forthcoming in Management Science. 66 pages, 8 figures, 5 tables; includes online appendix (pp. oa1-oa15)

Clinical algorithms can perform differently across patient groups, a concern that grows as these algorithms increasingly take the form of artificial intelligence (AI) models. In the United States, a 2024 federal rule bars healthcare providers that receive federal funds from discriminating through their use of clinical algorithms, which exposes them to liability when such use disadvantages a protected patient group. We examine how such liability considerations reshape (i) an AI firm's model design decisions that drive group-specific accuracy and (ii) a physician's decisions to use AI in healthcare delivery. The firm designs an AI model for two patient groups, and improving accuracy for the disadvantaged group is more costly. The physician (who remains the accountable decision maker) then decides whether to consult AI, weighing the reduction in clinical uncertainty against expected liability exposure when AI errors disproportionately affect the disadvantaged group. We find that this liability can induce disparate use of AI: the physician may reduce AI use overall and, over an intermediate range of liability, rely on AI less for disadvantaged patients. The effect is non-monotone. As liability increases, the physician's use of AI for disadvantaged patients first declines, then rises as the firm reallocates investment toward reducing disparity or switches to an equal-accuracy design. Mandating equal algorithmic accuracy across patient groups can then inadvertently harm both groups, because a uniform accuracy requirement distorts the firm's investment incentives and the physician's equilibrium AI-use decisions.

0.10

Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients

cs.CY cs.CLcs.HC ·2026-11-09 ·Matthew Flathers, Jill Noorily, Phuong Anh Nguyen, Mark Kalinich, Mina Fallah +12 more
similarity 0.55
Abstract · 100 pages: 30 page main text and references, 70 page appendix. 7 figures, 2 tables. Results explorer: https://mindbench.ai/audits/interventi…

Users are asking language models mental health questions and are receiving answers. We audited what 41 language models recommend to users when the same clinical information reaches them from three footings: a clinician describing a patient, a caregiver describing a loved one, or a person describing themselves. Across 19 case vignettes and five questions, we collected 35,055 free-text responses, extracted every intervention named with a validated judge, and had 29 models rate the full option lists on a seven-point Likert scale for the same cases. Models largely recommend what they rate well, with recommendation behavior differing by footing, and many models offering clinicians a consultation-style response and lay users hedged, simplified advice. When asked about recommendations for a specific intervention type, models more often named options they themselves rated below zero as initial choices, usually with conditions attached. We will release the instrument, code, and full corpus on publication as a living audit.

0.38

Digital Medicines Information at National Scale: Search Behaviour and System Performance of MediVerify Across 1.5 Million Queries

cs.CY ·2026-09-24 ·Praveen Charuka Athauda-Arachchi, Pandula Mahesh Athauda-Arachchi, Rohini Fernandopulle
published similarity 0.53
Abstract · 10 pages, 6 figures, Published in Frontiers in Digital Health

National-scale digital health platforms generate usage data that reveal population healthcare needs. MediVerify, Sri Lanka's online medicines information platform providing access to National Medicines Regulatory Authority (NMRA)-approved medicines, recorded over 1.49 million queries in its first year. Large-scale medicine search behaviour in low- and middle-income countries (LMICs) remains poorly characterised. Objectives: To characterise medicine information-seeking behaviour, identify mismatches between public demand and essential medicines policy, and evaluate technical performance. Methods: We retrospectively analysed 1,497,304 anonymised queries submitted between July 2024-2025. Queries were normalised and mapped to 11,933 approved medicines using fuzzy matching based on Levenshtein distance (threshold <=5). Therapeutic categories were assigned using an Anatomical Therapeutic Chemical (ATC)-aligned classification. Query patterns and system performance were analysed, and energy consumption was estimated from processing times. Results: Vitamins/minerals (11.74%), antibiotics (10.57%), anti-diabetes (7.02%), and antihypertensives (6.84%) dominated searches. The top 20 accounted for 37.3% of queries, while approximately 40%-50% of registered medicines were never queried. Zero-result queries (~ 1%) indicated unmet information needs. Frequently searched medicines diverged from essential medicines lists. Median latency was 8ms, with >99% query resolution. Estimated energy consumption was approximately 0.12 kWh per million queries. Conclusions: Large-scale medicine search data provide insights into healthcare information demand and policy alignment. MediVerify demonstrates that a national digital health platform can achieve high utilisation at low computational and environmental cost. Usage analytics could strengthen pharmaceutical policy and digital health infrastructure in LMICs.

0.43

Et Tu, Brute? Economic Misalignment in Personal AI Agents

cs.AI ·2026-09-21 ·Aman Priyanshu, Supriti Vijay, Brian Jabarian, Niloofar Mireshghallah
similarity 0.53
Abstract · 20 pages, 10 tables, 4 figures

Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.

0.52

When Clinical Judgment Becomes a Predictive Shortcut: Two Questions Clinical AI Must Answer Before Deployment

cs.AI ·2026-09-22 ·Jing Yang, Long R. Jiao, Zongjiu Zhang, Xiujun Cai
similarity 0.52
Abstract · 27 pages, 1 figure, 3 tables, 1 box; includes Multimedia Appendix 1

Clinical AI can predict outcomes using information generated by existing clinical judgment. Such performance alone cannot establish usefulness before recognition or across care processes. We propose evidence requirements matched to intended use and information conditions, with additional state-reference and proxy-relative timing tests for stronger recognition claims. This framework helps hospitals assess whether the conditions supporting performance hold when assistance is most needed. These questions remain outside current evaluation practice leakage checks, overall predictive performance, and external validation do not address them; the framework makes them explicit and specifies the direct evidence required to answer them.

0.19

One note in three: a verified census of three deployed AI scribes, and the instrument that counted it

cs.CL cs.AIcs.CY ·2026-08-31 ·Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
code/data similarity 0.52
Abstract · 47 pages, 3 figures. Dataset: https://huggingface.co/datasets/ComposoAI/OmissionBench . Code: https://github.com/composo-ai/omission-bench .…

Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; the 5,898 clearing an importance filter went to an adversarial panel of two models from different families, each told to refute what it could, and 618 survived. One note in three (31.3% [27.0, 35.6]) carries a verified failure, concentrated in allergy and medication information, invented patient identity, and history written up as examination on telephone consultations that can contain none. No product was given a patient record; setting aside the two classes a record would have prefilled, invented identity and dates, the rate is 24.8% [20.8, 29.0]. One failure mode did not fit our scheme, drawn from published scribe-error taxonomies: a treatment the clinician retracts, recorded as delivered care. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, 12 of 12 ([75.8, 100]); both judged every sampled refusal genuine. A failure rate depends on the instrument as much as the scribes. With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing family moves it too: alone at that instruction the gentler flags 54.8% of notes against 27.8%. Between 28% and 97% of sampled notes carry a failure depending on the standard. Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54-86% of their errors against our 23.1%. We release all 618 findings with transcript-side evidence, every prompt and model version, and the re-runnable pipeline.

0.79

"It's still 100% on me": Designing AI Tools for the Intersection of Physicians' Care and Legal Infrastructures

cs.HC ·2026-11-11 ·Gennie Mansi, Julia Kim, Lynn Dombrowski, Mark Riedl, Naveena Karusala
similarity 0.52
Abstract · 27 pages, 0 figures

Laws and regulations deeply shape physicians' workflows and impact how they incorporate AI-based decision support tools to care for patients. Despite calls to comply with legal standards, many AI-based decision support tools have not transitioned well to clinical settings. The medical community remains concerned about their legal liability, risks to patients, and limited clinical value of such tools. To better understand how to address these concerns in the design of AI tools for clinical decision-making, we conducted 19 interviews with physicians in the U.S. We used scenario-based design to investigate physicians' legal concerns in the context of an AI-based drug recommendation tool. Drawing on the ethics of care, we surface connections between physicians' care and the healthcare laws and regulations that deeply shape their information needs. We describe how physicians wanted to use the tool in individual and collective practices to care for themselves, their patients, their colleagues, work institution, and even the broader medical profession. We discuss how the design of AI-based decision support tools can present legal ambiguities that would prevent their caring actions and lead to non-use of the AI tool. Through our work, we contribute to responsible AI development in healthcare. Our analysis illuminates why and how certain kinds of information reduce legal ambiguity and support physicians' care when using AI-based decision support tools. We contribute an understanding of how to help physicians navigate legal liability and draw implications for grounding AI tools' designs in their care.

0.37