CS Paper Radar

norregel: a preregistered benchmark of dated numeric values in Norwegian law

Eirik Botten Nicolaysen

cs.CL ·submitted 2026-10-21 04:55 UTC · v2
benchmarkunusual

Abstract

norregel asks language models for numeric values in Norwegian statutes, regulations and parliamentary decisions (NOK amounts, percentages, multiples of the National Insurance basic amount (G) and age limits), each at a stated date. The 916 items were extracted mechanically from Lovdata's open data, and each carries the dated history of its provision, so an answer can be classed as correct, outdated, wrong or uninterpretable. We report a first and a second preregistered run on the same items. In the first run (five models, listed publicly only after local analysis), H1, that outdated answers are more frequent for values that took effect after a model's cutoff, could not be tested in substance, and H2 and H3 were inconclusive; Opus 5.5 and Sonnet 4.6 answered 46.7% and 40.2% of the 880 primary items correctly. The second run was preregistered and published before any of its calls, and eight models were analysed. Its pooled primary test supported H1 under the preregistered decision rule (d = 17.8 percentage points, 95% interval [7.7, 28.9], p = 0.0001). In the preregistered sensitivity analysis S8, which counts as outdated only the most recent value that differs from the key, d was 3.9 [-3.6, 12.7] (p = 0.1754), an interval that contains 0. H1 was supported in every other preregistered sensitivity analysis, but S8 is the only one that gives both groups one target value, so we do not regard the support for H1 as robust. The share of outdated answers equal to that most recent value (H2) was 56.1% [46.6, 65.0], an interval that contains 0.5, and the pooled test of a lower share correct in Nynorsk than in Bokmål (H3) was inconclusive. Several models, among them Haiku 4.5, gave mostly uninterpretable answers. Items and keys are withheld; item IDs, per-answer classes, code and results are on Zenodo.

Comment
24 pages. Preregistrations, code and results: Zenodo concept DOI 10.5281/zenodo.23422302 (versions v1 10.5281/zenodo.23422303, v1.1 10.5281/zenodo.23432124, v1.2 10.5281/zenodo.23434782, 2.0 10.5281/zenodo.23437444, 2.1 10.5281/zenodo.23502580). Items are withheld. Added NLOD attribution for Lovdata data

Closest papers

Nearest neighbours by meaning across the whole index.

Outdated numeric values from Norwegian statutes and regulations in a further-trained model and its base model: a preregistered paired comparison of Borealis 2 and Gemma 4

cs.CL ·2026-11-02 ·Eirik Botten Nicolaysen
benchmarkcode/data similarity 0.73
Abstract · Preregistered: doi:10.5281/zenodo.23858965. Data and code: doi:10.5281/zenodo.23870450, https://github.com/avalyset/norregel. v2: title corr…

We compare how often two released language models give outdated numeric values from Norwegian statutes and regulations: NbAiLab/borealis2-26b-a4b-preview (Borealis 2), which the National Library of Norway further trained from google/gemma-4-26B-A4B-it (Gemma 4), and Gemma 4 itself. The study was preregistered (doi:10.5281/zenodo.23858965) and uses the closed norregel benchmark, scored deterministically against a value history. On the 411 of 503 primary items where both models gave an interpretable answer, Borealis 2 alone was outdated on 10 items and Gemma 4 alone on 11 (d = -0.2 percentage points, 95% cluster bootstrap interval [-2.7, +2.5], exact one-sided McNemar p = 0.668). By the preregistered rule this is no effect of interest: the whole interval lies within the smallest effect of interest of +/-10 percentage points. Among the secondary measures, only the share of answers giving a value that took effect after the question date differs after Holm correction; Gemma 4 gave such answers more often. An exploratory comparison of two smaller Borealis models had no discordant pairs and is not informative. The design compares released models and does not isolate the effect of any part of their training.

0.27

NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains

cs.CL ·2026-11-02 ·Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot
benchmarkcode/data similarity 0.56
Abstract · Under review

Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce NitiBench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 26 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it twenty of the 26 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.

0.32

Annotated Surrogate Retrieval for Polish Statutory Law

cs.CL ·2026-08-31 ·Orkun Yiğit Cengiz
benchmarkcode/data similarity 0.53
Abstract · 16 pages, 5 figures, 5 tables. Code and data: https://github.com/OryCore/Research

We present a family of retrieval methods for Polish statutory law built on document surrogates: language-model annotations attached to statutory articles at index time. Three designs occupy different points on the cost-quality frontier. ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both language-model stages with three lexical and dense retrievers, weighted reciprocal rank fusion, and a deterministic re-scoring prior, using no model call before generation. We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations (264 with their reference article in the corpus), over 82,508 articles from 1,133 acts. On paired McNemar tests, ASCR-H places the reference provision at rank one significantly more often than every other non-oracle configuration except one of its own ablations (eighteen of twenty comparisons significant in its favour at p < 0.005), reaching 72.3% against 61.7% for BM25 and 52.3% for dense retrieval. The advantage is concentrated at the head and does not survive depth: it is significant at cutoffs of one and five, disappears by ten, and by twenty DTF leads on point estimate (86.0% versus 84.5%) at one ninth the latency and less than half the cost. Ablation attributes 27.6 points of rank-one accuracy to the reranking stage alone. We further report that the ranking advantage does not extend to citation accuracy, where DTF matches the oracle ceiling, and three negative results on lemmatisation, pseudo-relevance feedback and query rewriting. Surrogate annotation covers 27.0% of the corpus but every reference provision in the benchmark, an asymmetry we disclose and discuss. Benchmark, per-question outputs and paired significance tests are publicly available.

0.24

Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

cs.CL ·2026-09-23 ·Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng +9 more
highly cited similarity 0.51
Abstract

Aggregate accuracy summarizes how often a model is correct, but not which judgments remain correct when requests change. We evaluate Jev and nine language models on ContractNLI, with repeated requests that preserve the contract and target while varying visible hypotheses, requested answers, and output order. A four-model follow-up adds an equal-call repetition control and coverage of the remaining test contracts. Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions; consistent answers can also preserve errors. Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the observed error. Gold-class diagnostics further reveal different error profiles behind aggregate model rankings. Resource comparisons add distinct trade-offs: Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy. Together, these findings connect decision-level evaluation to whether additional calls improve correctness and what the resulting decisions cost.

0.59

Empty Intersection: Provenance Coverage Rose to 98% and Neither Verification Decision Moved

cs.SE ·2026-09-23 ·Dong Hyeon Jeon
similarity 0.51
Abstract · 12 pages

Two structural defenses for provenance, a grade on every row, so that a verification routine cannot mistake the system's own output for an observation, and a single write ingress, so that the grade is enforced rather than merely conventional, were measured against the production deployment that motivated them, over a frozen snapshot of 194,620 rows and the two verification decisions the snapshot supports. Neither reaches either decision. Both were prescribed by a companion paper, which diagnosed that deployment: its verification routines decided outcomes using values the system itself had written. Neither prescription is new: both are established practice in fields that do not cite one another, and no prior work measuring whether either changes a verdict was found, so what is offered here is the measurement and not the prescriptions. Filtering the verification queries by grade turns both decisions from pass to undetermined; widening the grade vocabulary raises classified coverage from 36.1% to 98.4%; a single ingress requiring a grade refuses 3,070 writes. None of the three gives either decision admissible input. The prescriptions do not fail at what they specify. Each is stated over the population and makes no reference to any decision, so neither says which rows a decision will read, and the rows each intervention repairs and the 32 rows the decisions read do not intersect. The intervention that changed the most rows shows the reach most plainly: all 121,296 rows it moved from unnameable to named fall outside both query windows. This paper reports the conditions, measured rather than designed, under which the decisions would have admissible input at all, and notes that the two decisions are blocked for different reasons.

0.37

Anatomy of a Spreadsheet Failure, Analysing the EuSpRIG Horror Story Corpus

cs.CY cs.HC ·2026-09-25 ·Simon Thorne, Angela Collins
published similarity 0.50
Abstract

Spreadsheet disasters have been documented for more than thirty years, but they have not gone away. This paper examines the EuSpRIG horror stories archive as evidence of how ordinary spreadsheets fail and the consequences that follow. After cleaning and deduplication, 121 incidents from 1995 to 2026 were classified into twelve failure categories and analysed by mechanism, context and consequence. Formula errors, data-entry slips and data-handling mistakes account for 71 cases and recur across the whole period, from Fidelity's missing minus sign in 1995 to a mistyped date that cost Norway's sovereign wealth fund about $92 million in 2024. A second pattern concerns information present in a workbook but not visible to the recipient: hidden rows, hidden sheets, pivot-table caches, embedded objects and retained source layers can cause harm even when no calculation is wrong. The clearest rise in reported cases is in this hidden or embedded data category, whose most serious example is the 2022 Ministry of Defence spreadsheet that exposed details of roughly 18,700 Afghan applicants through hidden rows. The paper argues that spreadsheet risk is no longer only a problem of wrong numbers. It also involves visibility, disclosure, governance and trust. Existing taxonomies remain useful for formula, entry and handling errors, but do not fully capture the public harm caused by spreadsheets used as uncontrolled operational infrastructure. The same mistakes have persisted for three decades, what has changed is the authority organisations give to the spreadsheets that contain them.

0.43

Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA

cs.LG ·2026-10-19 ·Fred Hu, Yeqi Wang, Xing Zhou
code/data similarity 0.50
Abstract · 26 pages. Companion artifacts are prepared but not yet publicly available

Similarity-gated coverage does not establish that retrieved evidence contains the facts needed to answer a question. We audit Mexican-Spanish financial-regulatory QA and find that 35 of 135 covered items lack some or all load-bearing facts. On the post-hoc full-evidence slice (n=98), eleven prompt-matched models score 95-98/98 and none differs significantly from the anchor; this does not establish equivalence. Removing evidence-defective items changes the nominal significance status of 14 of 55 pairwise comparisons. On 56 zero-evidence items, accuracy spans 33.9%-89.3%. Removing evidence from the audited items under an abstention-permitting prompt reduces accuracy by 63-81 percentage points in all ten tested legs, with no recovered items. Building on sufficient-context research, we contribute MexRegQA v1.0: 192 items with provenance and evidence-status labels, frozen retrieval evidence for 191 evaluated items, predictions, judge outputs and offline checks. The audit was outcome-aware, the later model/panel sweep reused the same questions, and all reported scores and labels rely on machine evaluation without independent human validation. We propose releasing evidence-status labels alongside grounded QA benchmarks so that model comparisons can be examined separately from evidence availability.

0.35

Precedent Labels and Facts Shift LLM Approval Decisions in Opposite Directions

cs.CL cs.AI ·2026-10-31 ·Junwei Yao, Zike Yang, Yongxi Zhou, Yanxian Chen, Jianan Liu +2 more
similarity 0.49
Abstract

When a language model decides refund or claim requests under written policies, the system can show it similar past cases. Each past case has two parts: its facts and the decision recorded for it. We separate the two. A model sees a policy, a request and three past cases, and we change either the recorded decisions or the facts, keeping the request fixed. We select requests that are close calls: deciding without past cases, DeepSeek V4.1 Flash approved each in some of four tries and denied it in others. When the past cases are near-copies of the request (its facts with a few details changed), switching their recorded decisions from DENY to APPROVE raises approval by about 30 to 55 percentage points in DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5 on policies written for the study, and DeepSeek and Luna show the same on nine public policies paraphrased from real ones. The facts push the other way: shown without their recorded decisions, past cases whose facts support approval make these two models approve less often than past cases whose facts support denial. One reading is that the request looks weaker next to them. This push is not just a correction toward the right answer: in an additional DeepSeek analysis, it was of similar estimated size on requests that human annotators judged approvable and on those they judged deniable. In a separate DeepSeek run on the public policies, when the past cases are near-copies of a different request, switching their decisions moves approval by 9 points instead of 43, while facts that support approval still lower it by 22 points against facts that support denial. Systems that retrieve past decisions should test the two parts separately.

0.29