CS Paper Radar

OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes

Wuraola Oyewusi, Eliana Vasquez Osorio, Goran Nenadic, Gareth Price

cs.CL cs.AIcs.LGq-bio.QM ·submitted 2026-10-02 11:38 UTC · v1
published

Abstract

Real-world outpatient oncology notes contain specialised terminology, tumour staging expressions, treatment names, toxicity descriptions, and institution-specific de-identification markers that may not be represented efficiently by general biomedical or adjacent clinical language models. We developed and evaluated oncology-specific BERT-style encoders using a governed UK outpatient oncology corpus comprising 290,026 notes from 21,564 patients treated for lung and head-and-neck cancer. We compared RadBERT and PathologyBERT with two local strategies: OncoNote-RadBERT, produced by continued masked language model pretraining, and OncoNoteBERT, trained from scratch with an oncology WordPiece tokenizer. Models were evaluated using masked language modelling loss and perplexity on the validation set, tokenizer fragmentation metrics, clinical term tokenisation, masked-token probes, and exploratory representation analysis. Both external encoders fit the oncology corpus poorly in zero-shot evaluation (perplexity 113.04 for RadBERT; 2035.03 for PathologyBERT), while continued pretraining produced the strongest fit (2.10 for OncoNote-RadBERT). OncoNoteBERT achieved perplexity 2.83 but produced the most efficient tokenisation, with lower subword fertility and shorter normalised sequence length. It also returned a clinically acceptable prediction for 12 of 13 masked-token probes, compared with 7 of 13 for OncoNote-RadBERT. This divergence between corpus-level fit and masked-token performance was partly attributable to tokenizer fragmentation rather than learned semantics alone. Both locally developed models represented the institutional placeholder as a single learnable token. These findings show that continued adaptation and bespoke tokenisation provide complementary benefits, and that representation-layer design matters before adjacent-domain encoders are applied to oncology NLP.

Comment
Accepted at the AI at Scale for Clinical Impact (ASCI): Cancer Pathology Foundation Models Workshop at NeurIPS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Fine-Grained Japanese Clinical Named Entity Recognition: A Controlled Comparison of BERT and Open-Weight Large Language Models

cs.AI cs.CL ·2026-11-17 ·Feng Han, Ayako Yagahara, Kenji Hirata, Atsushi Yoshida, Kohsuke Kudo
benchmark similarity 0.67
Abstract · 33 pages, 3 figures, 5 tables; includes appendices with supplementary tables and one supplementary figure

Background: Generative large language models (LLMs) offer an alternative to encoder-based clinical named entity recognition (NER), but their performance on fine-grained Japanese extraction tasks and sensitivity to generation settings remain uncertain. Methods: We compared two fine-tuned BERT-CRF baselines with nine open-weight LLMs on iCorpus, which defines 113 entity categories. The evaluation used 433 sentences from 36 test documents; 1,672 sentences were used for training. LLMs were evaluated using zero-shot prompting with and without JSON Schema constraints and three-seed QLoRA fine-tuning. Fine-tuned models were assessed under different output-token limits, including 900 and larger model-specific limits of 2,600 or 4,096 tokens. Performance was measured using sentence-level label-surface matching with relaxed and exact-surface F1. Results: The highest zero-shot relaxed F1 was 0.337, compared with 0.836-0.841 for the BERT baselines. At the larger output limits, all eight fine-tuned LLMs with at least 13B total parameters exceeded both BERT baselines (relaxed F1, 0.882-0.914; exact-surface F1, 0.840-0.880). Across three-seed means, increasing the output limit improved relaxed F1 by 0.052-0.085, alongside higher recall and fewer length-limited outputs. JSON Schema constraints changed relaxed F1 by -0.014 to +0.008 among seven models outside the two reasoning-oriented configurations, while medical-general model pairs showed no consistent advantage for medical specialization. Conclusions: On this historical Japanese case-report benchmark, task-specific fine-tuning combined with adequate generation capacity was associated with stronger LLM extraction performance. Output-token limits materially affected comparisons with BERT, whereas enforcing output structure alone did not overcome low zero-shot accuracy.

0.10

Large language models as oncology world models

cs.LG cs.AIcs.CL ·2026-11-13 ·Julien Vibert, Jakob Nikolas Kather
code/data similarity 0.66
Abstract · 63 pages (main text 35, Supplementary Information 28), 6 figures, 9 Extended Data figures. Code: https://github.com/julienvibert/llm-oncolog…

Large language models (LLMs) can synthesize cancer patient cases, but their internal representation of oncology, or world model, has not been explored systematically. By only specifying tumor type, we show that LLMs tend to generate similar patients per cancer, with on average 81.0% to 93.2% of a cancer's patients sharing the most frequent value of each baseline feature. By systematically varying patient features in 12 clinical scenarios, we show that they follow biomarker and treatment logic in 29 of 36 model-scenario assessments, against 1 of 12 when previous treatment or frailty matters. Continuing real patient trajectories, an LLM predicted 89.9% 12-month survival, and 79.2% with similar patients as examples, against 72.2% observed. To represent these trajectories, we developed the Clinical State Space, a dynamic representation framework that we also applied to real patients. Despite the limits we show, improved world models may support digital twins, which our framework can evaluate.

0.42

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

cs.LG ·2026-09-24 ·Rakib Abdullah, Md. Maruful Islam Maruf
publishedbenchmark similarity 0.66
Abstract · 6 pages, 2 figures. Accepted at the 2026 IEEE International Conference on Biomedical Engineering, Computer and Information Technology for He…

Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical benchmark evaluating three fine-tuned transformer encoders-BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa-against GPT-4o mini under zero-shot and few-shot prompting configurations for Bangla medical NER. In contrast to prior studies that evaluated large language models on limited subsets of only 50 samples, we conduct a large-scale evaluation across the full test set of 3,179 samples, providing statistically robust and reproducible baselines. Our fine-tuned XLM-RoBERTa model achieves an F1- score of 0.5959, establishing a new state-of-the-art and surpassing the previously reported best result of 0.5848. Crucially, we demonstrate that the language-specific BanglaBERT model consistently underperforms its multilingual counterparts with an F1-score of 0.4937, indicating that pretraining domain diversity can outweigh language specificity in highly specialized clinical settings. Furthermore, we present a detailed per-entity-type analysis for this task, revealing that Medicine and Specialist categories are recognized with high reliability, achieving F1- scores above 0.83, while the Symptom category remains the most challenging with an F1-score of 0.4367 despite being the most frequent training class. Finally, fine-tuned transformer models outperform the optimal prompting configuration by a factor of 3.76, confirming that prompt-only pipelines remain inadequate for structured clinical entity extraction in low-resource language environments.

0.19

An end-to-end-trained vision-language model for native-language prostate pathology report generation

cs.CV ·2026-08-24 ·Christian Grashei, Fabian Gülhan, Maximilian Legnar, Fabian Stögbauer, Cleo-Aron Weis +2 more
similarity 0.66
Abstract

Prostate cancer is among the most frequently diagnosed malignancies worldwide, and structured reporting of each biopsy core burdens pathologists. Existing tools frame this as classification, leaving pathologists to assemble coherent reports, while many slide-level vision-language models rely on English-centric encoders that transfer poorly to other clinical languages. We present a slide-level framework generating prostate biopsy reports that is language-independent by construction: tokenizer and model are trained from scratch, demonstrated here in German. To address paired-data scarcity, an automated pipeline uses a locally deployed large language model to split composite reports into core-specific image-text pairs, yielding 17,344 pairs from 2,402 historical cases without manual annotation. Evaluated for clinical attributes rather than linguistic similarity, the model achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, competitive with an FDA-cleared classifier. Grading is further validated on three external cohorts with latent-space augmentation. Institutions can thus train native-language reporting models on their own archives.

0.14

Making Clinical Language Models Auditable: Concept-Guided Fine-Tuning for Robust Prediction

cs.CL cs.AI ·2026-08-27 ·Jin Mu, Guanhua Chen
top venue similarity 0.66
Abstract

Clinical language models can achieve strong in-hospital accuracy yet fail under deployment shifts because they exploit note-specific artifacts (e.g., templates, separators, boilerplate) that do not reflect patient state. We propose CAST (Concept-guided Artifact Suppression Tuning), an SAE-based framework for auditable clinical text classification. CAST uses Sparse Autoencoders to expose sparse, human-auditable features from intermediate Transformer activations, labels SAE latents with an LLM-assisted interpretation pipeline and ICD-10 retrieval constraints, suppresses verified artifact latents via residual subtraction during fine-tuning, and provides post-hoc per-concept attributions for auditing model decisions. On MIMIC-IV discharge-note mortality prediction, CAST improves over its corresponding fine-tuned encoder baselines and remains competitive with strong LLM baselines, while producing a feature-level audit trail of the clinical concepts that support each prediction and the artifact concepts suppressed during training.

0.44

MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition

cs.CL ·2026-09-04 ·Justin Behling
code/data similarity 0.66
Abstract · 10 pages, 3 tables. Code, term list, and evaluation scripts: https://github.com/Nordis-Tech/medwer. Package: pip install medwer. Source arch…

Overall word error rate hides clinically critical errors: a transcript can be 95% correct and still swap one drug for another. The usual fix weights errors on medical entities, and almost always depends on an evaluation-time named-entity recognition (NER) model or cloud API, which makes the metric's denominator a versioned black box. We present MedWER, an evaluation protocol and open-source tool for medical ASR whose denominator is a fixed, license-clean term list: 19,373 drug, diagnosis, symptom, and injury-mechanism entries projected from public sources. The protocol couples a pinned text normalizer with a phrase-aware term-restricted WER, the MedWER, so the only versioned component is a normalizer dependency held at an exact release and checked against committed golden fixtures. Coverage is validated against an independent provincial drug-benefit file the list was not built from; the matching heuristic is calibrated against ground-truth entity spans. Baselines for Moonshine~base, Whisper~base.en, and MedASR on two open benchmarks are scored with the released tool and reported with 95% confidence intervals from resampled per-utterance scores.

0.33

EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

cs.AI ·2026-09-27 ·Andre R Goncalves, Vincent Liu, Priyadip Ray
similarity 0.64
Abstract

Electronic health records (EHRs) encode clinical histories as (time, modality, code) tuples, whereas pretrained language models expect text tokens. Serializing tuples as text inflates sequence length and repeatedly encodes the same structure. We introduce EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space. Modality and time gaps enter through a learned embedding and attention biases, and event codes receive dedicated vectors. Learning them is challenging because clinical vocabularies are long-tailed, leaving rare events with few observations. EHRAdapt therefore represents each event vector as the sum of a semantic prior and an evidence residual. The prior is a frozen, ontology-trained biomedical embedding of the event's clinical description, mapped into the input space by a shared learned projection, so it supplies clinical meaning even when observations are scarce. The residual is a learned low-rank, event-specific correction that refines it as evidence accumulates. We perform continued pretraining on records from about 4 million patients with three frozen LLM backbones, OLMo2 1B, Llama3.2 1B, and OLMo2 7B, training only the adapter (0.1--0.6% of all parameters). The full adapter outperforms all ablations in held-out next-event prediction on every backbone. Removing the semantic pathway hurts rare events most, over ten times more than the most frequent ones, whereas removing the residual hurts overall prediction but improves it for the rarest events. On reportable infectious-disease and syndromic classification, EHRAdapt outperforms text-based LLM and count-based baselines and the same LLMs adapted to the same records through next-event prediction, and both pathways improve rare-disease discrimination. The two pathways therefore play complementary roles, visible only when results are broken down by event frequency.

0.17

CRUX: Clinical-Text Anchoring for Cross-Institutional CT-Pathology Fusion in Non-Small Cell Lung Cancer Survival Prediction

eess.IV ·2026-10-15 ·Kyungwon Kim, Dosik Hwang
similarity 0.64
Abstract · 15 pages, 5 figures, 6 tables. Preprint. Extended version of a MICCAI 2024 paper (Kim et al., 2024)

Accurate prognosis in non-small cell lung cancer (NSCLC) benefits from integrating computed tomography (CT) and histopathological whole-slide images (WSIs), which capture complementary macroscopic and microscopic disease characteristics. Fusing these modalities is difficult: their features come from different pretrained foundation-model encoders and lack spatial correspondence, and both are sensitive to institution-specific acquisition and staining, causing multimodal models to overfit site-specific artifacts and generalize poorly. We present CRUX (CRoss-modal Unification via teXt anchoring), a fusion framework that uses clinical text---encoded by a biomedical language model---as an institution-invariant semantic anchor. Because clinical variables follow standardized classification systems, their embeddings occupy a representation space that is consistent across institutions, providing a stable reference frame for aligning heterogeneous imaging features. CRUX introduces a text-mediated cross-modal bridge in which CT- and pathology-derived text-query summaries exchange information within a compact text semantic space through a learnable hub token, avoiding direct image-space cross-attention that would propagate domain-specific artifacts. We evaluate on 2,157 NSCLC patients from five hospitals, holding out one hospital (N=446) entirely for external validation. CRUX attains an external concordance index (C-index) of 0.761, the highest among all unimodal, bimodal, and trimodal configurations. Notably, direct CT-WSI fusion degrades under distribution shift, whereas text anchoring restores and improves cross-institutional generalization. These results indicate that language-model-encoded clinical text is an effective structured prior for robust, generalizable multimodal cancer prognosis.

0.12