CS Paper Radar

CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

Ilan Theodoro, Jordão Bragantini, Loïc A. Royer

cs.CV ·submitted 2026-10-22 17:09 UTC · v1
benchmarkcode/data

Abstract

Large language models have shown outstanding performance on text-first tasks such as coding, math, and writing. Recent efforts are shifting models' training to include scientific problems that often live in highly specialized domains beyond text, such as protein structure and biological images. However, in these scenarios, their general-purpose accuracy compared with specialized algorithms remains in question. One such domain is cell tracking, which requires spatiotemporal awareness of cell dynamics. To evaluate this, we created a new benchmark, CellTrack-Bench, which assesses Vision-Language Models' (VLMs) cell-tracking-related performance against specialized trackers. The full benchmark draws from 15 standard cell-tracking datasets across four imaging modalities and comprises 1,537 problems corresponding to 9,041 cell-linking decisions. We selected a hard subset requiring greater reasoning effort by identifying failures shared across distinct algorithmic trackers. With this proposed dataset, we: (i) evaluated different environment harnesses for VLMs; (ii) benchmarked 9 frontier closed-source models and 13 open-weight models; and (iii) provided a detailed analysis of their failure modes. Code, the stored model outputs, and a pipeline that regenerates the benchmark from the source datasets are available at https://github.com/royerlab/celltrack-bench.

Closest papers

Nearest neighbours by meaning across the whole index.

Learning to Track Cells from Sparse Annotations in 3D Time-Lapse Microscopy

cs.CV ·2026-11-04 ·Yusuke Uchida
similarity 0.63
Abstract

Reconstructing cell lineages from 3D time-lapse microscopy requires distinguishing cell motion from division and inferring temporally consistent connections. When only a subset of cells is annotated, however, training procedures that treat unannotated regions as background or incorrect correspondences can be problematic. We propose a cell tracking pipeline that integrates detection from sparse annotations, calibration of candidate connections using image features, and joint selection of lineage graph events. For detection, we combine the outputs of 2.5D and 3D models trained to suppress erroneous background supervision at unannotated cells. We then calibrate ordinary link and division probabilities using motion features from dense flow, correspondence features from a learned cell-matching model, and features from models that identify cells immediately before division, dividing parents, and daughters immediately after division. Ordinary links, divisions represented as daughter pairs, and node inclusion are selected jointly under a surrogate objective derived from the evaluation metric, followed by gap filling and temporal coordinate refinement. The pipeline achieves an overall score of 0.9762 in 5-fold cross-validation on 199 videos from 2 zebrafish embryos. With candidates and calibrated values held fixed, joint selection outperforms a sequential method that allows replacement of existing divisions and reoptimization of ordinary links by 0.0105. The improvement arises primarily from division recovery.

0.32

Benchmarking Vision-Language Models on Synapse and Neuron Proofreading in Connectomics

cs.CV cs.AIcs.LG ·2026-09-29 ·Yicong Li, Junjie Wang, Leander Lauenburg, Ella Hugie, Alexandra Irger +3 more
similarity 0.62
Abstract

We benchmarked vision-language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse proofreading (presence and polarity) and neuron proofreading (split errors and merge errors). For synapse proofreading, we evaluated 19 open and 2 closed models across various architectures and sizes under zero-shot, four-shot in-context learning and LoRA settings, against specialist models, on datasets constructed by us using public resources. For neuron proofreading, we evaluated 3 open and 2 closed models on the ConnectomeBench2 dataset, with cross-species transfer from fly and mouse to human and zebrafish. Most models were at chance zero-shot; a few examples helped mainly the closed and largest open ones. LoRA on a few thousand labels brought open models level with specialist models. When evaluated on unseen species, the best adapted VLMs outperformed specialist models in identifying merge errors. The project will be publicly available upon acceptance.

0.35

See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology

cs.CV cs.AI ·2026-09-28 ·Chengyang Zhang, Wenchuan Zhang, Bo Li, Mengran Li, Xinyu Liu +6 more
benchmark similarity 0.61
Abstract

Pathological assessment relies on recognizing fine-grained visual details in histological images. Vision-language models (VLMs) increasingly support pathology interpretation, yet their ability to perceive these details remains inadequate. This weakness leads to inaccurate cellular observations that can persist even when final answers are correct. In this paper, we propose ASPECT to improve visually grounded reasoning through explicit supervision of cellular appearance and abundance. ASPECT trains intermediate visual tokens through pathology feature reconstruction, cell feature alignment, and count supervision. Three-stage supervised fine-tuning teaches the model to perceive, generate visual tokens, and reason, followed by reinforcement learning that rewards answer correctness and consistency with reported measurements. We also introduce PathoVernier, a benchmark of 759 expert-reviewed questions from five pathology datasets covering four cellular composition tasks. It evaluates both final answers and intermediate measurements to expose errors hidden by answer accuracy. On PathoVernier, ASPECT achieves relative accuracy gains of approximately 19.2% over the strongest baseline, Gemini-3.1-Pro, and 99.3% over its Qwen3-VL-8B backbone, while reducing RAWR, which measures counting errors within correct responses, by 28.1% and 42.7%, respectively. ASPECT also improves over its backbone on three external pathology benchmarks covering classification and question answering beyond cellular composition tasks.

0.26

MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

cs.CV ·2026-10-13 ·Yanran Ruan, Zhixiang Wei, Shiyao Hong, Xiaoxiao Ma, Yuxuan Gu +3 more
dataset similarity 0.61
Abstract

Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision-language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure-cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.

0.19

How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

cs.CV cs.AI ·2026-11-03 ·S M Rayeed, Qian Ma, Charles V. Stewart, Yao Ma
publishedbenchmark similarity 0.61
Abstract · Accepted to WACV 2027, 8 pages main paper; 4 figures, 4 tables

In biodiversity monitoring, a single image raises several questions: which taxon, how many, which trait, and whether the individual has been seen before. Vision-language models (VLMs) could answer all of these from the image, but their ability is largely untested in biology, as existing benchmarks focus mostly on species identification within a few taxonomic groups. We present Bio-VQA, the largest biological visual question answering (VQA) benchmark and the only one to span four aspects of biological visual understanding: hierarchical taxonomic reasoning, fine-grained attribute grounding, quantitative spatial reasoning, and individual re-identification. Bio-VQA contains 1.86M questions over 217K images from 11 public datasets, and extends right-or-wrong scoring to graded scoring, so a near-miss counts for more than a distant one. Evaluating 18 open-weight models from 3.75B to 38.4B parameters, we find that general visual capabilities do not transfer to biological organisms: the best model counts animals exactly on 41.7% of images with 2 to 10 individuals, less than half its accuracy on everyday objects at the same counts. Taxonomic accuracy falls to one sixth, from 0.72 at phylum to 0.12 at scientific name, and scaling helps mostly at coarse ranks. The gap to a domain specialist widens accordingly, from 0.731 against BioCLIP-2's 0.874 at class (1.2x) to 0.269 against 0.544 at scientific name (2x). On trait size measurement, 15 of the 18 models do no better than an image-blind baseline that predicts one fixed number for every specimen, and on re-identification, 17 of the 18 models fall below chance under adversarial conditions. These failures persist across a tenfold range of model size. We hope Bio-VQA becomes a standard test of how well VLM capabilities transfer to the biological domain.

0.46

BIABench: Evaluating AI agents on real-world bioimage analysis tasks

cs.AI cs.CL ·2026-09-28 ·Zixuan Pan, Davide Panzeri, Lukas Johanns, Marilin Moor, Yu Zhou +3 more
benchmark similarity 0.61
Abstract · 41 pages, 6 figures, 11 tables

Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end. Such analyses are hard for agents because 2D images, 3D volumes and time-lapse sequences are often too large to read as context, so an agent must choose and run an analysis through code, specialized software and rendered views. Published studies make this capability testable, because each pairs raw images with a peer-reviewed result. We introduce BIABench, a benchmark of 16 tasks reconstructed from published biological studies that retain their scientific questions, imaging data and ground truth. The tasks span eleven analysis subtasks and modalities from H&E histology to single-molecule localization microscopy. Each submission receives an outcome score, which compares the output files with the ground truth using field-standard metrics, and a process score, in which a vision-language model judges method choice and quality control against an expert-written rubric. We evaluated general-purpose and biology-specific agents across several language models, with repeated runs of every task. Routine two-dimensional tasks were solved well, but on some tasks that added a third dimension or a time axis no agent scored above 0.19. Neither biological specialization, stronger models nor detailed expert instructions closed this gap. The agents were also unreliable, with scores varying more between repeated runs of one agent than between different agents, and without ground truth a correct run could not be told from a wrong one by its process score or by the time spent. Released openly with its data and code, BIABench provides a verifiable framework for evaluating, and eventually training, agents for reliable long-horizon bioimage analysis.

0.17

SWIFT-Cell: Segmentation Without Iterative Flow Tracking

cs.CV ·2026-10-21 ·Devansh Lalwani, Swapnil Bhat, Muralikrishna H H, Anand Reddy, Maulik Shah
unusual similarity 0.60
Abstract

Generalist segmenters such as Cellpose-SAM (CPSAM) outline cells accurately across stains but slowly; watershed detectors such as QuPath's are fast but less accurate. In time-critical use, such as interactive retrieval of cells from a user-drawn region, CPSAM at the reference settings takes 58-303 s per measured 8192 x 8192 region on an RTX 4500 Ada GPU, beyond a seconds-scale interactive target. We distil CPSAM into SWIFT-Cell (SWIFT), a 0.63M-parameter U-Net trained only on CPSAM masks of 1,202 H&E and IHC tiles. On three 8192 x 8192 development regions SWIFT takes 2.0-5.2 s on 4 CPU cores in int8, 2.5-5.1x faster than QuPath's detector on the same cores, and 0.36-0.49 s on a GPU. Against human annotations on eight H&E, IHC and ISH sets it keeps 95% of CPSAM's panoptic quality (PQ) on PanNuke (0.593 vs. 0.627) and is level on NuCLS (0.546 vs. 0.540). On Ki-67 IHC masks scored to the outline centre it is 0.205 PQ above QuPath's better mode, 0.079 below CPSAM and 0.027 below InstanSeg (CI includes 0; at the outer edge 0.008 below CPSAM); on 212 pathologist-outlined nuclei in Ki-67 fields of 12 untouched patients its recall is 0.925 (CPSAM 0.920, InstanSeg 0.901; paired difference CIs include 0); one method-blinded pathologist accepted its outlines in 96% of 100 fields (60 from two development tissue patients, 40 hematology; CPSAM 83%, QuPath 53%; tissue 100% for SWIFT and CPSAM); on ER/PR points from 12 patients its detection F1 is 0.882 (CPSAM 0.896, InstanSeg 0.857), and on 4-reader PD-L1 points 0.83 (CPSAM 0.85, readers 0.87). SWIFT is close to CPSAM in accuracy and faster than watershed detection.

0.28

AdaVLA: Adaptive Step Flow Matching for Training-free Acceleration of Vision-Language-Action Models

cs.RO cs.LG ·2026-08-29 ·Sunghwan Han, Youngtae Han, Youngmin Yi
publishedbenchmark similarity 0.60
Abstract · Accepted to the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026)

Vision-Language-Action (VLA) models, built upon Vision-Language Models (VLMs), have significantly enhanced robotic capabilities by leveraging internet-scale knowledge and multimodal reasoning. However, the intensive computational overhead of VLAs constrains on-device deployment, hindering real-time responses to environmental changes. While various acceleration techniques have been proposed, they often rely on fine-tuning or access to training datasets, which are frequently unavailable due to privacy and proprietary concerns. Moreover, although flow-matching-based VLAs have emerged as efficient alternatives to standard diffusion models, current acceleration efforts largely target VLM inference costs, failing to address the iterative ODE solving process inherent in flow matching inference. To address these limitations, we propose AdaVLA, an online, training-free adaptive framework for fast yet accurate flow-matching-based Vision-Language-Action models. We introduce a novel metric derived from the flow matching trajectory curvature to quantify action generation confidence during inference. This metric enables the dynamic reduction of inference steps and the adaptive adjustment of MLP pruning ratios through an efficiently computed importance evaluation, requiring no access to training data. Experimental results on the LIBERO benchmark using a Jetson AGX Orin device demonstrate that our method achieves $1.87\times$ and $2.24\times$ speedups for $π_{0.5}$ and X-VLA, respectively, with negligible degradation in success rates. Furthermore, we validate the robustness of our approach on real-world robotic tasks using SmolVLA.

0.52