CS Paper Radar

Not What You Meant: Can LLMs Follow a Specified Negation Semantics?

Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock, Kostas Stathis

cs.AI ·submitted 2026-09-23 08:14 UTC · v1
code/data

Abstract

Negation does not carry a uniform interpretation across domains. In legal, regulatory, and medical reasoning, the intended interpretation depends on the reading in force -- open- versus closed-world, two- versus three-valued, and credulous versus skeptical. We study which reading of negation large language models adopt by default and whether they can override that preference when a different reading is explicitly specified. To this end, we introduce NAFBench, a procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics (WFS), and credulous and skeptical reasoning under stable-model semantics. The generator emits ground normal logic programs with controlled depth, width, and cycle structure. Each program is solved under all four viewpoints using SWI-Prolog, a well-founded semantics solver, and clingo, yielding up to four divergent labels. The programs are then verbalized into natural language under multiple framings and rule orderings that leave the answer invariant. The results expose a consistent gap. Across open-source models, following a specified negation semantics remains unsolved: the strongest models score 59--74% across the four semantic viewpoints, while the weakest score 31--67%. All models are order-sensitive on more than half of logically identical rule shufflings, while the two weaker models frequently overcommit on well-founded "undefined." Two frontier models reach 100% on the main fixed-complexity evaluation set, and a third, o4-mini, is near-perfect, falling only to 81% on well-founded "undefined." Delegating reasoning to a solver, fine-tuning on certified traces, or forcing an explicit three-valued verdict each partly closes the gap.

Closest papers

Nearest neighbours by meaning across the whole index.

Prove What You Can: Hybrid Neurosymbolic Reasoning for Logical Natural Language Inference

cs.CL ·2026-11-06 ·Ada Canoğlu, Sadiq Qara
theory similarity 0.63
Abstract · 32 pages, 6 figures, 13 tables; under review at ARR (October 2026 cycle)

Logical reasoning over natural language requires both semantic interpretation and deductive search. Large language models (LLMs) can propose plausible intermediate derivations, but they often fail to compose them into a complete proof and overpredict uncertainty. Neurosymbolic methods improve search by translating an entire problem into first-order logic and handing it to a prover, but a single bad clause can make the whole translation nonexecutable. We present Hybrid Reasoning, a three-stage pipeline that decouples proposing a proof, validating its steps, and deciding a label. An LLM proposes a structured proof tree with a raw verdict; a theorem prover validates each derivation locally, keeping only the subset it can prove; and an uncertainty-gated repair attempts to re-derive the label from the premises plus the validated derivations, when the raw verdict is UNCERTAIN. Across four benchmarks (FOLIO, ProofWriter, ProverQA, PrOntoQA) and five contemporary LLMs, Hybrid Reasoning is the top-performing method on the two hardest datasets, improving on the strongest baseline by 5.3 points on FOLIO and 4.9 on ProverQA, and reaches ceiling-level accuracy on the easiest. The repair stage operates at 78-98% precision, producing 326 correct relabellings against 19 regressions.

0.30

Verifiable Logic in Medical LLMs: A Critical Review of Agentic Clinical Reasoning Optimization and Neuro-Symbolic Grounding

cs.CL cs.AIcs.CYcs.LG ·2026-11-19 ·Youssef ElNahas, Adel Ammar, Soumaya Chaffar, Wadii Boulila
position similarity 0.63
Abstract

Clinical large language models are usually judged by whether the final answer is right. Yet a model can reach a fluent, plausible output through invalid reasoning that static benchmarks do not detect, and confident hallucination, causally invalid inference, and spurious demographic shortcuts can pass conventional evaluation. This critical review covers four paradigms in turn: how clinical hallucination arises and why static metrics miss it; the agentic hallucination cascade, in which multi-agent and tool-augmented systems propagate and amplify early reasoning errors; process reward modeling as step-level supervision; and neuro-symbolic grounding, in which formal logic engines, ontology reasoners, and knowledge graphs place verifiable constraints on neural output. Within their encoded domains, systems with symbolic veto layers offer guarantees that hold relative to their encoded knowledge, and they report large gains on the safety measures they use. These results come from single studies, mostly in simulated or small-sample settings, and are not directly comparable with neural baselines. Process reward models remain bounded by the statistics of their training data. We propose a conceptual framework that maps each agentic failure mode to a symbolic intervention point, and we argue that clinical AI safety depends substantially on system architecture. Because current regulation emphasizes performance and does not require verification of reasoning steps, we argue that responsible certification should also audit reasoning trajectories alongside accuracy on static benchmarks.

0.12

Walking on the DARKSIDE

cs.AI cs.LO ·2026-08-24 ·Aldo Gangemi, Emanuele Bottazzi
similarity 0.62
Abstract · 23 pages, 4 figures, 5 tables

Large Language Models (LLMs) do not natively track the path of exclusions that a coherent discourse demands. When an input rests on a fabricated authority, a misapplied mechanism, or a surreptitious analogy, an unsteered LLM tends to engage with it as if it were well-posed, and this affects its generation. POLANYI++, an LLM-steering method that uses heuristics, ontologies and problem-solving methods for tacit-knowledge extraction, produces an Extended Knowledge Graph (XKG) in OWL2, but when a sophisticated nonsensical input is reified into the graph alongside the legitimate triples, it gets hardly detectable by automated reasoners, since the XKG is generated jointly with the wrong assumptions. We introduce DARKSIDE, a coherence-auditing method on top of POLANYI++. DARKSIDE formalises an explicit data structure of accumulated exclusions over discourse time, complemented by a warrant axis that classifies each named referent as Warranted, Unattested, Misattributed or Fabricated. The method is anchored in nine theoretical fragments unified under a shared deep frame of path integrity. The resulting DARKPOLANYI is evaluated as a steering layer over Gemini 3 on BSBench, a 100-item adversarial corpus of sophisticated-sounding nonsense across multiple domains, with Claude Sonnet 4.6 as an independent judge. DARKPOLANYI scores 1.89/2 mean versus 0.95/2 for the unsteered Gemini 3 Pro baseline; on the 97 cases with valid judgments in both arms, paired McNemar gives a paired bootstrap mean-diff = +0.92 (95% CI [+0.75, +1.08], p = 0.0001). The evidence supports an architectural claim: when an LLM forward pass is wrapped in an ontology-mediated auditing, structurally inevitable hallucination can be partially recovered. The XKG functions as the missing memory that LLMs lack, and the warrant axis as an epistemic firewall.

0.49

Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?

cs.CL ·2026-10-10 ·Duaa Nakshbandi, Zhiyu An, Yiping Li, Sachin Giri, Wan Du
dataset similarity 0.61
Abstract

Large Language Models (LLMs) are increasingly being used to summarize and verify claims in critical sources that inform consequential decisions. Such claims are full of plural definites. If three of five patients improved after taking drug A, the claim the patients improved after drug A is not true, since two did not, yet its negation, the patients didn't improve after drug A, is not true either, since three did. In formal semantics, this truth-value gap is known as the Homogeneity Gap: both the plural definite and its negation are neither true nor false when the property holds for only a fraction of the plural referent. A model that cannot represent this gap turns mixed evidence into a clean verdict, an error that can be costly in high-stakes use. We introduce HomogeneityNLI, a diagnostic dataset of 1,000 minimal pair items crossing determiner, polarity, and group composition across ten logical conditions. Evaluating seven open-weight models, we uncover a pervasive Bivalence Bias: models assign plural definites their homogeneous truth conditions. In the zero-shot setting, no model recognizes the gap for both polarities in more than 15% of scenarios, and negated definites are judged False in 84% of cases. Explicit linguistic rules, chain-of-thought, and enumeration prompts produce unstable, model-specific effects that often entrench the bias. Our findings indicate that current open-weight LLMs acquire the truth conditions of plural predication but not its trivalent logic, and cannot yet be trusted to summarize or verify claims whose evidence is mixed.

0.24

Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs

cs.CL cs.AI ·2026-08-31 ·Ramya Keerthy Thatikonda, Wray Buntine, Ehsan Shareghi
top venue similarity 0.61
Abstract · Accepted to Findings of EMNLP 2026

Logical reasoning with large language models (LLMs) is a critical capability, as it reflects a system's ability to correctly deduce hypotheses from a given context using faithful deductive processes. However, LLM reasoning has often been shown to be sensitive to small surface-level variations in problem formulation, raising questions about whether models truly follow the underlying logical structure. Studying this behavior is challenging because the symbolic components of logical problems, such as operators and predicates, are difficult to systematically manipulate in natural language. We introduce a tool-driven framework for generating controlled, label-preserving edits to logical reasoning problems. Our method operates on symbolic representations of first-order logic and constraint satisfaction problem tasks, enabling targeted modifications to logical operators and other structural components before translating them back into natural language. Using this framework, we evaluate various LLMs under cumulative and individual operator edits and analyze their behavior in response to these changes. Our quantitative and qualitative analyses show that LLM reasoning behavior under controlled operator edits is inconsistent, regardless of model size or family: models sometimes adapt correctly to structural changes but often fail to track their logical consequences. The results from this automated stress test enable an evaluation of language models across different dimensions and help measure the reliability of their reasoning.

0.43

ALBI: LLM-Guided Inductive Logic Programming via Iterative Generate-Test-Constrain Refineme

cs.AI cs.SC ·2026-11-03 ·Aboubaker Seddiq Benterki, Jüri Vain, Gert Kanter, Leelo Aino Jürgenstein
similarity 0.61
Abstract

We introduce ALBI (Agent for Learning By Induction), a neuro-symbolic ILP framework that replaces the ASP-based hypothesis generator in the Learning from Failures (LFF) paradigm with a Large Language Model (LLM), while retaining SWI-Prolog as a formal verifier. ALBI implements an iterative generate-test-constrain loop: the LLM proposes candidate hypotheses from structured prompts; Prolog tests each candidate; and diagnostic feedback drives refinement. We further introduce semantic anonymization, a metamorphic testing methodology, to distinguish structural symbolic induction from surface-level pattern matching on predicate names. Experiments on 30 standard ILP benchmarks show that ALBI achieves 100% success across three frontier LLMs (Gemini 3.1 Pro, GPT 5.4 Pro, Claude Opus 4.6), matching Popper while requiring only 1.4-1.7 iterations on average. Under anonymization, all models maintain 100% success with a moderate iteration increase (1.36-1.67 times), providing evidence of robust symbolic reasoning.

0.16

Neuro-symbolic PRM: Enhancing Scientific Reasoning via Structured Traces and Symbolic Verification

cs.CL ·2026-08-26 ·Yuxin Zi, Cong Xu, Suparna Bhattacharya, Martin Foltin, Amit Sheth
similarity 0.60
Abstract

While tool-augmented Large Language Models have significantly improved multi-step reasoning in quantitative STEM tasks, a critical residual failure mode remains: intermediate reasoning steps that are syntactically well-formed, mathematically executable, and unit-consistent, yet contextually ungrounded. Current approaches either rely on formal verifiers that cannot assess semantic intent, or burden Process Reward Models (PRMs) with the dual task of checking both arithmetic and logic. In this paper, we propose a neuro-symbolic framework that cleanly decouples reasoning into two formal dimensions: Symbolic Validity ($V$) and Semantic Groundedness ($G$). We guarantee $V$ by construction using a deterministic symbolic verifier acting as a hard filter. To assess $G$, we train a PRM conditionally on the verifier-accepted manifold. To train this PRM efficiently, we introduce Counterfactual Symbolic Perturbation (CSP), a novel data synthesis strategy that algorithmically generates constraint-preserving hard negatives (steps that perfectly pass the verifier but are logically flawed). At inference, we deploy a verifier-first constrained search that guarantees execution consistency for verifier-covered operations while relying on the PRM solely to rank semantic grounding. By targeting the exact residual error class of strong tool-using LLMs, our method significantly improves reasoning reliability without the sprawling heuristics of prior frameworks.

0.29

Repairing LLM-Generated Specifications with Deterministic Diagnosis

cs.AI cs.CL ·2026-10-15 ·Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi +4 more
similarity 0.60
Abstract

Large language models (LLMs) are increasingly used as formalizers: given a reasoning problem expressed in natural language, they produce a formal specification that a symbolic solver can then solve, with guarantees relative to that specification. Many specifications are runnable yet semantically wrong, encoding incorrect constraints, domains, objectives, predicates, or goals, and solver-error feedback dries up once execution succeeds. Purely LLM-based diagnosis, lacking a formal ground truth, often fails to spot these errors, and its ungrounded critiques cause regressions in 11.8% of initially correct specifications in our experiments. We propose DRAGO, a repair framework that replaces this ungrounded critique with deterministic, formalism-specific diagnosis, and retains a rewrite only when the diagnosed error weight strictly decreases under a reference-free acceptance predicate. Across five benchmarks spanning logical reasoning (SMT-LIB2), constraint programming (MiniZinc), and planning (PDDL), DRAGO cuts regressions by nearly an order of magnitude, to 1.3%, while improving answer accuracy by 12.9 percentage points over zero-shot formalization and 8.7 percentage points over LLM-diagnostic repair, averaged across benchmarks. We also uncover two rendering defects in Planetarium that cause 50% (3,086 instances) of its floor-tile split to be inconsistent with the reference formalizations, and release a corrected split.

0.17