CS Paper Radar

LABEL: Literature Asymmetry between Efficacy and Limits in Drug Knowledge and Language Models

S M Rayeed, Charles V. Stewart, Yao Ma

cs.AI cs.IR ·submitted 2026-11-14 05:14 UTC · v1
code/datahot direction

Abstract

Every medication comes with two types of medical facts: indications, which describe what conditions the drug treats (benefits), and contraindications, which describe who must avoid it to prevent serious harm (safety limits). Although contraindications are critical for patient safety and FDA regulations require listing them as rigorously as indications, we find that contraindications are far less documented than indications in general prose. In this paper, we measure how often these texts contain both facts across 574 drugs from PrimeKG by comparing a drug's indications against its own contraindications. Across Wikipedia articles and PubMed abstracts, we find that reference prose names a drug's benefits about twice as often as its safety limits (within-drug odds ratio 2.19). We call this documentation gap LABEL (Literature Asymmetry between Efficacy and Limits). Language models mirror this gap. As we scale Qwen2.5 from 0.5B to 14B, accuracy on indications rises by 18.3 points, whereas accuracy on contraindications remains flat near random guessing. However, when we provide models with a retrieved passage that states the answer, they use safety facts just as effectively as benefits. Furthermore, routing safety questions to official FDA labels boosts accuracy by 15.8 points whenever the label states the fact. Overall, our findings show that models fail on medication safety not because they cannot reason about risk, but because free-text corpora lack safety facts. Clinical AI systems should therefore route medication-safety questions to official regulatory labels and structured databases rather than relying on general public records. Code and data are available at https://github.com/SMRayeed/LABEL.

Comment
27 pages (9 main + supplementary material), 8 figures, 7 tables

Closest papers

Nearest neighbours by meaning across the whole index.

PlainPairs: A Corpus Pairing the Prescriber and Patient Texts of U.S. Prescription Medicine Labels

cs.CL cs.CY ·2026-11-16 ·Obada Kraishan
similarity 0.63
Abstract · 19 pages, 9 figures, 11 tables

A prescription medicine in the United States comes with labeling written for prescribers, and many medicines also carry a text written for patients: a Medication Guide or a patient package insert. The two texts describe the same medicine for two readers, yet public resources rarely hold them side by side. I present PlainPairs, an open corpus built from the openFDA drug label files. Version 0.1.0 holds 35,177 prescription labels that carry a patient text, 21,416 distinct patient texts, 250,829 patient-text segments matched to prescriber sections through their regulatory headings, reading-level measures on both sides, and rule-based counts of plain-language wording. A second source, the National Cancer Institute's cancer information summaries, adds 756 summaries in health-professional and patient versions, in English and Spanish. Measured on the structure of the label, the median Medication Guide reads at grade 8.7 by Flesch-Kincaid and at grade 11.1 by SMOG. The same guides measured on the flattened text that the bulk files serve read about two grades higher, a difference that comes from lost list and heading boundaries. Patient texts address the reader in 41 of every 100 sentences, against 0.2 in prescriber sections. A blind hand check of 100 pairs supports the heading-based alignment for seven of ten heading types. The corpus, its code and its documents are public, and the release keeps texts that show a company copyright mark out of the text tables.

0.28

Large language models show systematic failures in incorporating evolving medical knowledge

cs.AI ·2026-10-22 ·Min Zeng, Zaifu Zhan, Kai Yu, Rui Zhang
similarity 0.62
Abstract

Medical knowledge evolves, yet whether large language models (LLMs) faithfully incorporate revisions into clinical recommendations remains unclear. We introduce ChronoEvo, comprising 1,709 recommendation-level revisions across 12 forms of medical knowledge change and 275 guideline lineages, to distinguish recognition of revised guidance from its full incorporation. With updated guideline evidence explicitly provided, revision recognition exceeded full incorporation across 14 models, with a 20.1-percentage-point gap in the strongest model. For Qwen3-4B, 90.8% of responses recognized revisions, 80.2% were clinically correct, and 58.5% fully incorporated them, yielding a 32.3-percentage-point gap. Two clinicians' adjudicated assessment of 300 responses yielded a 40.3-percentage-point gap in this enriched validation sample; 61 of 144 responses with incomplete incorporation were clinically incorrect. Prompting Qwen3-4B to prioritize revised evidence reduced outdated-content retention and improved clinical correctness without significantly improving full incorporation. These findings identify faithful incorporation of revised guidance as a distinct challenge for clinical LLM reliability.

0.17

Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?

cs.CL ·2026-10-10 ·Duaa Nakshbandi, Zhiyu An, Yiping Li, Sachin Giri, Wan Du
dataset similarity 0.62
Abstract

Large Language Models (LLMs) are increasingly being used to summarize and verify claims in critical sources that inform consequential decisions. Such claims are full of plural definites. If three of five patients improved after taking drug A, the claim the patients improved after drug A is not true, since two did not, yet its negation, the patients didn't improve after drug A, is not true either, since three did. In formal semantics, this truth-value gap is known as the Homogeneity Gap: both the plural definite and its negation are neither true nor false when the property holds for only a fraction of the plural referent. A model that cannot represent this gap turns mixed evidence into a clean verdict, an error that can be costly in high-stakes use. We introduce HomogeneityNLI, a diagnostic dataset of 1,000 minimal pair items crossing determiner, polarity, and group composition across ten logical conditions. Evaluating seven open-weight models, we uncover a pervasive Bivalence Bias: models assign plural definites their homogeneous truth conditions. In the zero-shot setting, no model recognizes the gap for both polarities in more than 15% of scenarios, and negated definites are judged False in 84% of cases. Explicit linguistic rules, chain-of-thought, and enumeration prompts produce unstable, model-specific effects that often entrench the bias. Our findings indicate that current open-weight LLMs acquire the truth conditions of plural predication but not its trivalent logic, and cannot yet be trusted to summarize or verify claims whose evidence is mixed.

0.24

When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine

cs.AI ·2026-10-19 ·Yusuke Watanabe, Yohei Kobashi, Taisei Tosaki, Takeshi Kojima, Yusuke Iwasawa +2 more
benchmark similarity 0.62
Abstract · 17 pages, 8 figures, 8 tables

Clinical decision-making often begins with underspecified information and requires not only gathering missing information but also knowing when to stop. This calls for judging determinability (whether the available information is sufficient for a decision) by considering whether missing information could change the conclusion, not simply the amount of available decision-relevant information. Reliance on this amount as a cue for determinability, which we term \textit{factor coverage bias}, could undermine the trustworthiness of large language models (LLMs) through premature conclusions or excessive information gathering, such as unnecessary invasive tests. Yet whether LLMs exhibit this bias remains insufficiently studied. To address this gap, we construct a controlled pairwise benchmark in which LLMs identify the determinable case within each determinable--undeterminable pair. We add a decision-relevant factor to either side while keeping each case's determinability unchanged, and similarly add irrelevant text to distinguish this bias from context-length effects. Adding a relevant factor made LLMs more likely to select the augmented case as determinable even when it remained undeterminable, an effect that persisted with increased reasoning effort, while adding irrelevant text made them less likely to select the augmented case, an effect that reasoning substantially reduced. These findings identify factor coverage bias as a failure mode distinct from context-length effects and a potential risk to LLM trustworthiness. Our work provides an evaluation framework for LLM trustworthiness under underspecification.

0.29

Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?

cs.CY cs.AIcs.CL ·2026-08-10 ·Lorenzo Molfetta, Alessio Cocchieri, Luca Ragazzi, Ilaria Bartolini, Marco Patella +1 more
top venuebenchmarkhighly cited similarity 0.62
Abstract · Please cite the definitive, peer-reviewed version of this article published in the Proceedings of the 64th Annual Meeting of the Association…

In medicine, claims remain valid when supported by empirical evidence grounded in stable biological reality. In law, by contrast, truth is contingent, defined by jurisdiction, temporal validity, and the hierarchy of authoritative sources. The recent success of large language models (LLMs) on medical licensing examinations has encouraged an expectation of comparable legal competence. This analogy, however, obscures a critical distinction between domains. Unlike in medicine, legal performance often depends less on inference than on determining when external authority is applicable, valid, and non-contradictory. We introduce a comparative diagnostic framework evaluating legal reasoning against medical baselines along four axes (knowledge recall, grounding, confidence, and robustness), uncovering a sharp domain asymmetry when applied to a new benchmark that encodes temporal validity and normative relationships. While medical LLMs reliably benefit from verified sources, legal LLMs struggle to assess when retrieved citations are useful or misleading, exhibiting overconfidence in perturbed contexts and sensitivity to superficial formatting cues. Increased model scale amplifies this tendency, revealing that stronger instruction following can coincide with weaker resistance to authoritative perturbations. These findings show that LLMs treat law as unstructured text rather than binding precedent, while revealing a tendency to over-trust authoritative but false information when external references conflict with a model's internal knowledge.

0.90

Right Answer, Wrong Reason: Accuracy, Consistency, and Consensus Are Misleading Indicators of LLM Faithfulness in Clinical Decision Support

cs.AI ·2026-09-26 ·Bharath Kumar Bolla, Chaitali Deb Purkayastha, Vishnu Surya Reddy Nandi
published similarity 0.61
Abstract · Accepted and received best paper award from ICETCI 2026 Conference, https://ietcint.com/

Clinical Large Language Models (LLMs) achieve strong medical-exam accuracy; however, a correct answer does not guarantee that the explanation names the concepts that actually drove the decision. We introduce three lightweight, directly interpretable metrics for this faithfulness gap: the Explanation Stability Index (ESI), which measures reasoning consistency across repeated queries; the Causal Faithfulness Score (CFS), which tests whether cited concepts drive predictions via concept ablation; and the Perturbation Stability Score (PSS), which measures robustness to semantic-preserving paraphrases. By evaluating six LLMs on 150 MedQA-USMLE questions (900 model-question observations), we found that only 23.3% of the cited clinical concepts were causally necessary. Correct answers had lower CFS than incorrect answers (0.212 vs. 0.398), answer consistency negatively predicted CFS (Spearman r = -0.466), and model pairs could agree on answers while sharing only 8.8% of cited reasoning concepts. These results show that accuracy, consistency, and consensus are incomplete safety signals for clinical decision-making support. The evidence is behavioral rather than mechanistic: concept ablation tests counterfactual sensitivity of outputs, not internal circuits.

0.28

An evidence-traceable benchmark of clinical reasoning in large language models

cs.CL ·2026-11-10 ·Zhiqun Zhao, Xiaodong Yang, Bruce Gu, Ming Li, Yi Fu +11 more
benchmark similarity 0.61
Abstract · 25 pages, 8 figures, 4 tables. Zhiqun Zhao, Xiaodong Yang and Bruce Gu contributed equally. Corresponding authors: Zheng Wang, Haiquan Lu an…

Benchmark scores for medical large language models (LLMs) are difficult to interpret when scoring criteria cannot be traced to clinical evidence. Here we developed Med Evidence Bench (MEB; https://www.medevidencebench.org.cn/), comprising 1,536 physician-constructed items across three levels: guideline knowledge, standardized clinical scenarios and real-world laboratory scenarios. Each criterion stating a medical claim is linked to a versioned source-text span. Across ten LLMs, the Evidence Trace Score (ETS) was 0.975-0.997 for guideline knowledge, 0.292-0.658 for standardized scenarios and 0.182-0.447 for real-world scenarios, and rankings varied across levels. For real-world scenarios, scores for identifying clinical information (mean, 0.770) exceeded those for matching evidence (0.165) or formulating recommendations (0.160). On 1,296 criterion-level judgments, automated grading agreed with the majority vote of three physicians (macro-F1, 0.930; 95% confidence interval, 0.915-0.945), although agreement was lower for evidence application. These findings from Chinese-language materials indicate that guideline recall does not ensure patient-specific evidence use and do not establish clinical effectiveness.

0.25

From Mutation to Medication: Characterizing Failures of Clinical LLMs to Respect Genomic Resistance

cs.AI cs.LGq-bio.GN ·2026-11-14 ·Negin Ashrafi, Roxana Daneshjou
top venue similarity 0.60
Abstract · Accepted for publication in the Proceedings of the Machine Learning for Health (ML4H) 2026 Symposium

Language models are increasingly proposed to support molecular tumour boards, but may recommend cancer therapies despite genomic evidence of resistance. We evaluated ten models on 1,464 resistance cases from five curated oncology knowledge bases, collecting 58,910 judgments. Models endorsed resisted therapies in 21.8-81.4% of cases. However, raw rates can reflect a tendency to agree rather than an ability to distinguish resistance from drug sensitivity: one open-weight model endorsed 81.4% of resistance cases and 82.3% of sensitivity controls. We therefore compared endorsement within matched gene and evidence-level groups. This analysis is exploratory because it was specified after the pre-registered analysis, which yielded null-to-weak results. Measured failures depended strongly on the reference knowledge base and evidence strength. When a second source independently confirmed resistance, four models across three vendors endorsed at most 2.4% of cases; at weaker evidence tiers, discrimination was not measurable. Models sometimes acknowledged resistance yet recommended the therapy. Repeating a model's own resistance conclusion reduced endorsement to 0.0-3.5% on cases where it had stated resistance. Falsely attributing the opposite conclusion increased endorsement above baseline for all seven hosted models, reaching 95.2% for GPT-4o; an instruction without an attributed conclusion changed endorsement by only 0.9 percentage points. Providing the correct curated evidence reduced unqualified strong recommendations to 0.3-3.5%, testing evidence use under an oracle condition rather than retrieval.

0.50