CS Paper Radar

Evidence-Grounded LLM-Assisted Android Mobile Forensic Reporting

Madhab Chandra Das, Sajid Ahmed, Tasnia Tabassum, Qingzhong Liu

cs.CR cs.CY ·submitted 2026-10-18 06:22 UTC · v1
published

Abstract

Large language models can assist forensic report drafting, but fluent output does not ensure evidentiary reliability. This study presents an evidence-grounded framework for evaluating LLM-assisted Android forensic reports at the atomic-claim level. Gemini, Llama, and Qwen each generated reports for two controlled cases: a public Android 13 dataset and a physical LG device with a zero-reportable-fact boundary. The six accepted reports were decomposed into 612 claims and evaluated against frozen, human-reviewed authorities through content labeling, citation validation, fact-recall analysis, independent rechecking, and dispute resolution. The final corpus contained 424 supported, 163 unsupported, four contradicted, four overstated or incomplete, and 17 excluded claims. In the public case, all three models recovered all 27 authorized report-level facts. Citation validity nevertheless differed substantially: Gemini produced 168 valid citations from 189 required citations, Qwen produced 145 from 189, and Llama produced none from 135. In the physical device case, the models generated 65 claims despite the zero reportable-fact boundary, including 38 unsupported and one contradicted claim. These results show that evidence restriction and complete fact recall do not guarantee claim support or valid evidence attribution. The framework provides an auditable method for detecting unsupported content, preserving claim-to evidence lineage, and retaining investigator authority over final forensic reports.

Comment
10 pages, 3 figures, 7 tables. Accepted at the 8th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (IEEE TPS 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

A Symbolic-Neural Framework for Context-Aware Evidence Extraction from Mobile SQLite Databases

cs.CR ·2026-11-02 ·Jeel Piyushkumar Khatiwala, Divyangkumar Patel, Dianxiang Xu, Weifeng Xu
published similarity 0.63
Abstract · 10 pages, 1 figure, 5 tables. Accepted at the 60th Hawaii International Conference on System Sciences (HICSS-60), January 2027

Reliable forensic evidence extraction from mobile SQLite databases is challenging because application schemas are undocumented, evolve across releases, and differ across operating systems, making schema-specific methods brittle and difficult to maintain. We present SNEEF, a symbolic-neural framework for context-aware evidence extraction that encodes each SQLite row as context-value pairs with standardized provenance metadata and column labels, uses a language model to recognize nine forensic entity types, and binds each accepted entity to its source table and row through the PA-EEP provenance protocol. We evaluate SNEEF against examiner-verified ground truth from five corpora built from public Android and iOS device images, spanning multiple application versions. SNEEF achieves an F1 score of 0.967 for evidence entities extracted from SQLite row content, excluding application names, compared with 0.728 for a context-free configuration, with consistent gains across all corpora. We also introduce an Evidence Failure Mode taxonomy to characterize why extraction errors occur.

0.32

Correct Answers, Unsupported Findings: Evidence Binding in Forensic Reconstruction of LLM Agent Logs

cs.CR cs.AI ·2026-10-07 ·Taehyeon Yun, Dongho Kim, Geonwoo Kim, Juyoung Seo, Minseok Hur +1 more
similarity 0.63
Abstract · 11 pages, 2 figures

Forensic reconstruction of LLM-agent actions requires not only recovering the correct value, but establishing which preserved record supports that finding. Tool logs, generated explanations, and local citation identifiers capture different parts of this evidence, yet a citation identifier does not establish a source unless its binding to a record is preserved. We audit this distinction using 64 mechanically checkable cases from saved AgentDojo Banking executions. Two LLM readers reconstruct source relationships under controlled variations in visible evidence and identifier-to-record bindings. We separately evaluate complete-record agreement, evidence-grounded findings, justified abstention, and unsupported assertions. With original identifiers and no binding table, Sonnet recovered every literal source location but made unsupported citation-source assertions in 26 of 28 cases requiring the missing relation; 22 nevertheless matched the complete reference. Adding explicit bindings improved grounded reconstruction for both readers, whereas identifier renaming alone provided no consistent remedy. A deterministic same-packet comparator correctly resolved the bounded task or abstained throughout. These results show that factual agreement alone is insufficient for evaluating forensic reconstruction of agent logs and motivate preserving explicit record bindings to distinguish supported findings from correct guesses.

0.19

When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks

cs.AI cs.PFstat.ME ·2026-10-29 ·Carole Lunny, Ian Goodman, Stefano Berrone, Connor TA Brenna, Emma K Reid +10 more
similarity 0.58
Abstract · 76 pages, 2 figures, 2 tables, 1 box, 6 supplementary files

LLM read-and-evaluate tasks include study screening, data extraction, risk-of-bias assessment, and critical appraisal in the evidence synthesis field. These task differ in the amount of LLM interpretation they require. Some tasks involve retrieving relatively simple information, such as an author name, sample size, or publication year. Others require the LLM to read source material, interpret it against predefined criteria, and make a judgement. Researchers should assess whether LLM judgements are accurate, supported by appropriate source evidence, and reproducible under specified conditions. A correct judgement does not necessarily indicate that the underlying evaluation process is reliable. We developed a taxonomy for identifying mechanisms that may compromise these properties and for informing the selection and reporting of appropriate evaluation methods. The taxonomy outlines potential ways that LLM judgments from read-and-evaluate tasks can fail, called TRAILS (Taxonomy of Reliability And Instability in LLM-assisted evidence Synthesis). The TRAILS taxonomy addresses two complementary questions. First, what mechanisms can cause the same read-and-evaluate task to produce different judgements across repeated LLM runs? Second, what mechanisms can systematically affect the judgement or required output in a way that is not apparent from the judgement itself? This study presents 38 mechanisms and identifies the deployment settings in which each can operate, approaches for testing them, and mitigation strategies. We also show why repeatability or accuracy does not rule out a hidden judgement failure and why human evaluation should assess whether a judgement is grounded in the supplied evidence, independent of inappropriate external influence, and produced through a process that can be examined and verified.

0.23

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

cs.CL cs.AI ·2026-10-07 ·M. Mikail Demir, M. Abdullah Canbaz
publishedbenchmarkcode/data similarity 0.57
Abstract · 9 pages, 2 figures, 7 tables. Published in the 21st International Conference on Artificial Intelligence and Law (ICAIL 2026), Singapore. Dat…

Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether a legal claim is supported by the underlying authority. Using recent New York State Court of Appeals decisions, we build a dataset of 3,396 parenthetical-style claims labeled as Supported, Refuted, or Not Found. We cast this task as a three-way natural language inference problem and evaluate several state-of-the-art LLMs in a zero-shot setting. Although the strongest models reach up to 0.97 accuracy, the results also show an important weakness: models still incorrectly mark unsupported claims as supported, even when the full opinion text is provided. Across models, missing support is harder to detect than direct contradiction, and fabricated but plausible citations cause the largest drop in performance. Overall, PARCEL provides a practical benchmark for testing claim-level groundedness in legal RAG systems.

0.55

LLM-Generated Investment Research Memos Need Validation

cs.CE ·2026-11-03 ·Xukang Gong, Chenri Luo, Binyan Yu, Xiao-Yang Liu
publishedhot direction similarity 0.56
Abstract · 24 pages, 4 figures. Accepted at the TAE (Trust-AI-Eval) Workshop, NeurIPS 2026

LLMs are increasingly adopted to draft long-form investment research memos, but their reliability is judged claim by claim, while readers read each memo as a whole. In this paper, we show that a low claim-level error rate does not necessarily mean that a memo is reliable: even with a refutation rate of just 1-2%, roughly four in ten memos contain at least one claim contradicted by official filings. First, we generate 1,440 memos with four recent LLM-powered agents for 30 companies over 12 quarters, decompose them into 77,272 financial claims, and verify each against the same official source bundle by retrieving evidence, matching it to the claim, and verifying the outcome. Second, we find that only 1.17-1.77% of claims are refuted, yet 36.94-41.11% of memos contain at least one refuted claim, and agents differ in severity: Codex has the lowest memo-level rate; a qualitative review further groups errors into four patterns (time, unit, context, concept).

0.70

Structural Inference in Undocumented Mobile Databases: A Reproducible Benchmark for Evaluating Agentic Reasoning in Digital Forensics

cs.CR ·2026-08-21 ·Jeel Piyushkumar Khatiwala, Divyangkumar Patel, Weifeng Xu
publishedbenchmark similarity 0.56
Abstract · 10 pages, 2 figures. Published in Proc. 50th IEEE Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, July 202…

Agentic large language models are increasingly used in digital forensic analysis, yet their ability to infer relational structure inside undocumented mobile application databases remains poorly understood. In forensic contexts, structurally incorrect inferences can yield results that appear plausible while remaining evidentially unsound. This work evaluates agentic structural inference as an isolated capability, treating execution success and structural correctness as distinct evaluation axes. It examines how an agent reconstructs table relationships, linking attributes, and executable join paths when given only a raw database and a natural-language investigative prompt. We apply a fixed, deterministic evaluation pipeline to two contrasting SQLite repositories: Android's SMS database with stable identifier propagation, and Snapchat's database with irregular schemas, ephemeral identifiers, and polymorphic relationships. Using expert-verified SQL ground truth, we evaluate (i) structural correctness of inferred relational links, (ii) execution coherence under multi-table reasoning, and (iii) robustness and failure modes of inferred structure when execution succeeds but relational interpretation diverges from expert ground truth. Evaluation is performed independently of semantic interpretation, with full queries and execution traces provided in the Appendix. Results show that structural inference remains reliable in regular schemas but degrades sharply as schema ambiguity increases, frequently producing structurally plausible yet incorrect joins that execute successfully. These findings clarify where schema-agnostic agentic reasoning can support forensic analysis, how its robustness degrades under realistic schema irregularities, and why additional verification remains essential before inferred relationships can be treated as reliable evidence.

0.70

Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes

cs.AI cs.CLcs.CY ·2026-08-23 ·Heather Renze
code/data similarity 0.56
Abstract · 20 pages, 4 figures, 6 tables. Code and derived data available at https://github.com/heathriel/synthetic-memoir-audit

When a large language model (LLM) is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED (scene positively corroborated): 354 of 366 days fail, 96.7% (Wilson 95% CI 94.4-98.1%). Only 12 days contain a corroborated scene; 19 days (5.2%) assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline (no evidence the original rate was inflated) while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure (83.3%). We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.

0.31

A2A-ForensicTrace: Offline Verification of Tamper-Evident A2A Runtime Evidence

cs.CR ·2026-09-27 ·Adil Alshammari, Sareh Assiri, Hayretdin Bahsi
published similarity 0.56
Abstract · 9 pages, 4 figures. Accepted at IEEE IEMCON 2026

Security-relevant Agent2Agent (A2A) executions can cross organizational boundaries, leaving investigators without live access to all participating systems. Offline investigation involves checking preserved records and their cross-record relationships for consistency. This paper presents A2A-ForensicTrace, an offline verification layer that converts runtime observations into typed records, derives protocol-relevant relationships, and commits both under an incident-trace root. An Ed25519-signed receipt binds the root and capture digest to the incident context. Evaluation comprised 240 executions through the official A2A software development kit (SDK), with actions selected by a large language model (LLM). These included 120 condition runs and 120 matched controls. All condition runs returned the expected bounded findings. No control produced an indication, and all roots and receipts verified. Median in-memory latency of the full offline verifier was 1.61 ms for the scenario traces. In the separate scaling experiment, it was 30.85 ms at 1,000 committed leaves. Future work will broaden A2A lifecycle coverage.

0.43