CS Paper Radar

Automatic Field-of-View Adjustment for a View-Expansive Microscope via LSTM-Based Gaze and Pipette Motion Interpretation

Kenta Yokoe, Takuya Hara, Tadayoshi Aoyama

cs.HC cs.LGcs.ROeess.IV ·submitted 2026-08-11 02:49 UTC · v1
published

Abstract

Intracytoplasmic sperm injection (ICSI) operators frequently adjust the field-of-view (FOV) during procedures, which interrupts workflow and increases procedure time. Conventional microscopes require manual objective lens switching and illumination adjustments to achieve different FOV sizes. We propose an AI-based automatic FOV adjustment method integrated with a view-expansive microscope. This microscope enables the simultaneous acquisition of a large FOV and high-resolution images using a single objective lens through multiview imaging with galvanometer mirrors and high-speed vision, thereby eliminating the need for physical lens exchanges. Our method utilizes a long short-term memory (LSTM) model to predict the appropriate FOV size based on real-time analysis of the pipette's position and velocity, combined with the operator's gaze position. The AI model is trained using ICSI procedure data from an expert with over five years of micromanipulation experience. Experimental evaluation with novice operators reveals that the proposed automatic FOV adjustment system significantly improves the ICSI procedure speed, reducing the average task completion time from 60.5 to 48.0 s (p < 0.001). The experiments also demonstrate that this improvement enables novice operators to achieve ICSI working speeds equivalent to those of expert operators.

Comment
This is the accepted version of an article published in IEEE Access 13, 182915-182923 (2025). DOI: 10.1109/ACCESS.2025.3624246. Open Access under CC BY 4.0
Journal ref
IEEE Access, vol. 13, pp. 182915-182923, 2025
DOI
10.1109/ACCESS.2025.3624246

Closest papers

Nearest neighbours by meaning across the whole index.

MicroSplat: Towards Automating 3D Small-Scale Reconstruction

cs.RO ·2026-09-28 ·Roman Mineyev, Robert Azarcon, Suvrat Jain, Chuizheng Kong, Shreyas Kousik
similarity 0.61
Abstract

Reconstructing sub-millimeter detail of small objects underpins tasks such as circuit-board defect detection, specimen digitization, and archiving. 3D Gaussian Splatting (3DGS) makes photorealistic reconstruction possible from posed images alone, but macro optics violate two conditions it depends on. The field of view (FOV) and the depth of field (DoF) shrink to millimeters, so viewpoint error can take a sample out of frame, and only a thin focal plane is sharp in frame. Existing automated macro-capture rigs handle the shallow DoF with per-view focus stacking, but typically rely on classical photogrammetry and offline, human-in-the-loop processing. We present MicroSplat, which generates a 3DGS using a low-cost consumer microscope on a manipulator arm. After a single up-front object-localization step, the system runs unattended. It performs calibration, plans collision-free viewpoints, captures and fuses a focal stack at each viewpoint, segments the object across all views, and optimizes a 3DGS model warm-started with the arm's own poses. We further compare against blur-aware 3DGS methods, which infer sharp scenes from defocused images, and find that measuring sharpness with a controlled focal sweep outperforms inferring it. We provide real and synthetic datasets (including robot and camera parameters) and the fitted 3DGS reconstructions.

0.28

MicroHookACT: Monocular Microscopic Vision Guided Visuomotor Policy for Flexible Microelectrode Hooking

cs.RO ·2026-09-18 ·Yitong Chen, Fangbo Qin, Yang Wang, Ruihua Hu, Kui Zhang +1 more
similarity 0.59
Abstract

Automated needle-loop hooking is a critical step in flexible microelectrode (FME) implantation. This paper presents MicroHookACT, an imitation learning-based visuomotor policy for automated 3D hooking under monocular microscopic vision. First, a unidirectional hooking strategy exploits defocus cues and optical-axis guidance to enable palpation-free precise alignment and contact-rich threading. Second, an action-supervised object attention module built on a frozen ViT backbone learns to focus on the micro-needle tip and micro-loop directly from human demonstrations, without requiring manual visual annotations for training. Third, attention-centered global coarse and local fine features are dynamically weighted according to predicted action progress, enabling a single ACT policy to adapt to changing defocus blur and visual requirements throughout the operation. In the experiments, visuomotor policies were trained on 60 human demonstrations and evaluated under five setups with varying difficulties. Our MicroHookACT framework achieved the highest overall success rate of 96.7\% with an average execution time of 11.5 s. These results demonstrate the potential of visuomotor policy learning for micron-level control under varying operating conditions.

0.32

Assessing the Marginal Utility of Exocentric Head Views in UMI-Style Large-FoV Wrist-Camera Manipulation

cs.RO ·2026-10-08 ·Yuhui Duan, Shi Jin, Yuntian Wang, Jin Wang, Siao Liu +3 more
similarity 0.56
Abstract

Multi-view perception is often treated as a safe design choice for robot manipulation: if one camera is useful, more cameras should be better. We revisit that assumption in tabletop bimanual manipulation by comparing policies trained with two large-field-of-view (FoV) wrist cameras against policies that additionally receive a head view, using controlled-variable experiments on a real bimanual platform. Across three tabletop tasks, adding a head view does not consistently improve task completion or execution quality: when the wrist cameras already observe the objects, grippers, and goal regions, wrist-only policies form a strong baseline. Restricting the wrist-camera FoV increases the potential value of a head view, but performance improves most reliably when the added view supplies concentrated, task-relevant information: region-of-interest (ROI) cropping yields more consistent head-view gains than the uncropped image, which is dominated by background and task-irrelevant regions. The data-scaling results further suggest an interaction between FoV and demonstration count: wide observations improve coverage but may require more data to stabilize, and they may reach a higher performance ceiling once enough demonstrations are available. These results suggest that camera design should be framed as a task-relevance and information-density problem rather than as a simple count of input views.

0.12

CRISP: Corneal Confocal Microscopy Real-Time Image Stitching Pipeline

cs.CV ·2026-09-07 ·Qincheng Qiao, Puli Zhang, Jian Zhou, Xinguo Hou
code/data similarity 0.55
Abstract · 7 pages, 2 figures. Code: https://github.com/SummerColdWind/CRISP

Morphology of the sub-basal nerve plexus (SNP) reflects peripheral nerve health, and corneal confocal microscopy (CCM) provides an important means for in vivo, real-time, non-invasive observation of the SNP. However, mainstream CCM devices offer a limited field of view per frame, whereas the SNP is spatially non-uniform; discrete image sampling is therefore sensitive to sampling location and frame selection, which limits the reproducibility and clinical adoption of CCM as a quantitative assessment tool. Wide-field stitching can reconstruct larger SNP mosaics by integrating sequentially acquired CCM images, but existing methods largely rely on offline post-processing, additional hardware, or specific acquisition protocols, and lack open-source real-time solutions for conventional CCM video streams. This paper presents CRISP (Corneal confocal microscopy Real-time Image Stitching Pipeline), an open-source real-time SNP wide-field stitching framework for conventional CCM examination video streams. CRISP excludes defocused and discontinuous segments via focus-aware gating, propagates poses through local pairwise registration, and maintains non-redundant spatial coverage with a sparse anchor map; when local temporal continuity is interrupted, the system completes relocalization and subgraph merging through global appearance retrieval followed by geometric verification. The framework prioritizes low-latency coverage feedback during examination while outputting accepted frames, poses, and anchor information to initialize offline fine stitching. To our knowledge, CRISP is the first open-source real-time SNP wide-field stitching framework released for conventional CCM video streams. By lowering the barrier to adoption and reproduction of wide-field stitching, CRISP may help move SNP wide-field imaging from a research tool into routine clinical examination workflows.

0.43

MACAW: Reliable And Efficient Surgical Debridement Using Monocular Adaptive Compact Attention Windows

cs.RO ·2026-09-02 ·Ziyang Chen, Shutong Jin, Preethi Satish, Sareena Mann, Cael Magner +4 more
similarity 0.54
Abstract

Augmenting the dexterity of human surgeons has the potential to free them from tedious subtasks. We consider debridement (removal of diseased or dead tissue fragments), which is challenging due to imprecision in spatial perception and cable actuation. We develop an augmented dexterity system for surgical debridement that uses visual servoing to align the cable-driven gripper with the target position in the image plane, and then introduces a novel approach to depth control, MACAW: Monocular Adaptive Compact Attention Windows. Across 100 physical trials using the da Vinci Research Kit (dVRK) robot, camera-frame servoing reduced average gripper position offset from 37 to fewer than 5 pixels within 4 optimization steps, taking an average of only 0.39s. MACAW significantly outperforms procedural and learned VLA baselines, achieving a 93% success rate at 11 seconds per fragment, yielding a throughput of 304 fragments per hour. Extending MACAW to a bimanual debridement setup maintains a 92% success rate at an average of 7 seconds per fragment, increasing the throughput to 473 fragments per hour.

0.48

Vision-Language Agents for Active Perception in Optics Laboratories

cs.AI physics.opticscs.CVcs.RO ·2026-09-26 ·Ryan Lopez, Sachin Vaidya, Seou Choi, Serena Landers, Marin Soljačić
similarity 0.54
Abstract

Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.

0.56

Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

cs.RO eess.SY ·2026-10-25 ·Enduo Zhao, Fangzhou Li, Yujie Zhu, Sicheng Lu, Fang Chen +2 more
benchmark similarity 0.54
Abstract · 8 pages, 5 figures. Submitted to IEEE Robotics and Automation Letters (RA-L) for possible publication

In minimally invasive surgery (MIS), maintaining an appropriate field of view (FoV) is essential for surgical safety and efficiency. However, conventional autonomous laparoscope control methods often treat visual tracking and surgical phase recognition independently, limiting robustness under occlusion, entity overlap, and camera motion. This paper proposes a closed-loop autonomous laparoscope control framework that couples real-time surgical phase recognition with camera-motion-compensated visual perception. By exploiting phase-dependent relationships among surgical entities, the framework improves target state estimation under challenging visual conditions, supporting task-aware autonomous FoV control. Experiments on benchmark datasets evaluate the proposed framework in surgical phase recognition and target tracking, including scenarios involving occlusion and entity overlap. Phantom experiments further demonstrate the feasibility of autonomous FoV control, showing improved target visibility and task execution performance compared with fixed-FoV control. These results highlight the potential of incorporating task-level semantics into visual state estimation for autonomous laparoscope guidance.

0.18

Visual Cue Interactions in AR-Guided Needle Insertion: A Prostate Biopsy-Inspired Phantom Study

cs.GR ·2026-08-27 ·Xinrui Zou, Mingxu Liu, Thomas T. Jones, Braden Millan, Sandeep Gurram +3 more
published similarity 0.53
Abstract · Accepted at IEEE ISMAR 2026. The definitive version will be available in IEEE Xplore

Despite the apparent simplicity of the motor action involved during percutaneous needle procedures, manipulating the tool's direction becomes challenging when clinicians cannot directly visualize internal anatomy and must rely on ultrasound images, which increase cognitive demand. Augmented reality (AR) offers the promise to assist with these tasks by providing pertinent visual information in the clinician's field of view. However, simply providing visual information that ignores meaningful visual cues can complicate depth perception and spatial understanding. In this work, we introduce and evaluate three visualization techniques for needle alignment developed during design sessions with medical experts: a localized focus-and-context window, a color-based proximity encoding, and an explicit trajectory overlay. These techniques were evaluated in a user study (n=26) including clinical experts (n=7) using a prostate-biopsy-inspired phantom. Results from this study suggest that cue effects depended on the surrounding cue configuration and user expertise. For novices, explicit trajectory overlay improved targeting accuracy and reduced retreat behavior, but its effect on completion time varied across cue configurations, with slower performance when the overlay was presented alone. For experts, the focus-and-context window reduced completion time and retreat events, while color-based proximity overlay improved completion time. Subjectively, color cues were often perceived as helpful even when their effects on accuracy were not consistent. These results suggest that AR guidance strategies for percutaneous interventions should consider user expertise and visual context.

0.41