CS Paper Radar

Moving Beyond the Flat Screen: Opportunities and Challenges of Extended Reality for Studying Human Brain and Behavior

Tom Bullock, Radha Kumaran, You-Jin Kim, Emily Machniak, Tobias Höllerer, Barry Giesbrecht

cs.HC ·submitted 2026-10-13 21:08 UTC · v1
published

Abstract

Much of what we know about human perception and cognition comes from austere experimental settings: static stimuli presented to sedentary observers on flat monitors, complex responses reduced to simple button presses, and brain activity recorded from immobilized heads using EEG or fMRI. These highly controlled paradigms have provided remarkable insights, but capture a small and often unusual slice of behavior, raising questions about how well the findings generalize to the real world. Extended reality (XR), spanning both immersive virtual reality (VR) and augmented reality (AR) that overlays digital content on the real world, offers a way beyond these constraints, supporting controlled, reproducible experiments in which participants actively engage with three-dimensional tasks. Drawing on a growing body of VR and AR studies of attention, memory, and language -- our own and others', some involving EEG -- we outline what XR makes newly possible for cognitive neuroscience and reflect on the practical and conceptual obstacles that remain. The opportunities are substantial, but realizing them requires solving practical problems of hardware, timing, and measurement, and being realistic about what a virtual or augmented world can and cannot stand in for.

Comment
4 pages, 1 figure. Workshop paper presented at the NeuroXR 2026 Workshop at IEEE ISMAR 2026 (ISMAR-Adjunct)

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond the 1D Reality-Virtuality Continuum: A 2D Framework for Seven Extended Reality Paradigms

cs.HC ·2026-10-27 ·Rajashekhar Vachiravelu Saminathan
published similarity 0.69
Abstract · Accepted for IndiaHCI 2026 Conference in the Student Research Consortium held at Indian Institute of Technology - Hyderabad, India; 7 pages,…

The Reality-Virtuality (RV) continuum helps in describing the relationship between physical and computer-generated environments. However, the increasing number of Extended Reality (XR) paradigms introduces new technologies. Their characteristics cannot be completely represented by a single real-virtual dimension. It can be noted that XR systems not only vary in their proportion of real and virtual content but also adds, suppresses, or transforms information available to the user. This work proposes a framework for characterizing seven XR realities, namely Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Augmented Virtuality (AV), Phenomenal Reality (PR), Annexing Reality (AnR), and Diminished Reality (DR). It uses a two-dimensional extension of the conventional 1D RV continuum. The first dimension helps in identification of the position of an XR technology between real and virtual environments. The second dimension captures information addition and subtraction. The proposed approach gives a qualitative basis for comparing existing and emerging XR paradigms within a unified XR framework.

0.25

Wait for It: A Component-Level Analysis of Latency Feedback Mechanisms for LLM Agents in VR

cs.HC ·2026-10-16 ·Pratham Arora, Abhinav Lodha, Siddharth Siddharth
similarity 0.62
Abstract

Delays in AI responses pose a significant challenge to immersive Virtual Reality experiences. While previous research demonstrates that feedback can reduce the perception of waiting, we lack understanding of which types work best and why. We conducted a within-subjects study with 18 participants to systematically compare three isolated feedback approaches during VR agent interactions: verbal fillers (e.g., um..., hmm...), visual indicators (loading spinners), and embodied gestures (thinking animations). Our results revealed that embodied gestures significantly outperformed visual loading indicators across multiple dimensions, making agents feel more lifelike and socially present. Interestingly, explicit visual feedback like loading spinners actually degraded perceived humanness and broke immersion by forcing users to shift from social to technical mental models. These findings suggest that natural, behavior-based feedback maintains the illusion of conversing with a thinking entity, while technical indicators expose the computational substrate. We present empirically-grounded design guidelines for creating more engaging AI-powered VR agents.

0.08

Exploring Object Recall in an Object-Rich AR Office Scene

cs.HC ·2026-10-13 ·Sangita Kunapuli, Shane Dirksen, Radha Kumaran, Yanxiu Jin, You-Jin Kim +2 more
published similarity 0.62
Abstract · 11 pages, 9 figures. Accepted to IEEE ISMAR 2026 and published in IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 32, …

Augmented reality (AR) changes how users perceive and attend to their surroundings. Prior work suggests AR can reduce attention to the physical world, but it remains unclear whether this holds in environments dense with both virtual and physical objects, and which factors most affect object recall. This study examines recall in a mixed-reality office scene after an object-selection task, testing object-level factors (virtuality, size), context-level factors (semantic congruency, saliency), and user-level factors (cueing, dual tasking). Cueing informed participants of a later recall test, while dual tasking added cognitive load through a secondary audio task. Virtual objects were recalled more accurately than physical ones. Cueing produced a modest improvement in recall of present objects. Using playback software to reconstruct each participant's view throughout the trials, we computed object-level saliency and found a weak positive correlation with recall, slightly stronger than the effects of virtuality, but not reaching statistical significance. These findings have implications for AR interface design and for building mixed-reality environments that better support cognition during real-world tasks.

0.30

C-LISTEN: Cognitive Load Impacts of Sensory-Triggered Environmental Navigation in Virtual Reality

cs.HC ·2026-09-29 ·Md. Monowar Hossain, M. Rasel Mahmud
top venue similarity 0.61
Abstract · 6 pages. Published in AIVR 2026, The Third International Conference on Artificial Intelligence and Immersive Virtual Reality, Lisbon, Portug…

Nowadays, Virtual Reality (VR) systems are more helpful for making decisions, training, and rehabilitation. Multimodal settings in these systems impose a significant balance and cognitive difficulties. Additionally, decreased performance, higher cognitive load, user overwhelm, and limited accessibility of VR technology can be caused by excessive auditory, visual, and sensory stimulation. In our pilot study, we examine the strategic design of directional auditory cues to improve cognitive load and enhance user experience in virtual environments. This research also enhances fundamental knowledge that auditory feedback reduces cognitive load in VR, with statistical analysis confirming this improvement (p = .0011). This study investigates how postural balance and pupil diameter correlate with cognitive load while the users navigate. Moreover, this study provides realistic overview design guidelines for accessibility, which will make VR experiences safer and more useful for everyone, including people with cognitive impairments.

0.49

Unpacking the process of watching a 360-degree video: How physiological responses and head rotation relate to affective and cognitive outcomes

cs.HC ·2026-10-30 ·Portia Wang, Yujie Tao, Monique Santoso, Jeremy Bailenson
similarity 0.61
Abstract

A defining feature of 360-degree video is that the perceptual experience is actively constructed. As viewers turn their heads, shift their gaze, and encounter changing visuals, their perceptual experience is not predetermined, but rather emerges through ongoing exploration. Yet, it is unclear how the process of watching immersive videos unfolds, and how it is related to psychological and physiological outcomes. We present a large-scale study (N=200) in which participants experienced one of 10 pieces of 360-degree videos in virtual reality. Results showed that greater head rotation was associated with greater valence, higher attention focus, longer length of subjective review, and lower simulator sickness; greater pupil dilation was associated with lower valence, lower review ratings, and greater use of negative tone in written reviews. A mediation analysis revealed that participants who rotated their heads more experienced greater frame-to-frame pixel-level perceptual changes in their field of view, which in turn resulted in greater valence and lower simulator sickness. Moreover, participants tended to rotate their heads toward field of view regions with greater perceptual movement, while directing their eye gaze toward less perceptually dynamic areas within that view. We outline theoretical and practical contributions for designing 360-degree content and predicting experiential outcomes.

0.14

Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

cs.RO cs.HC ·2026-09-25 ·Carl Tornberg, Alicia Torck, Lotfi El Hafi, Tadahiro Taniguchi
code/data similarity 0.61
Abstract · Submitted to Advanced Robotics, Special Issue on "Next Generation Cognitive Robotics: Nurturing Embodied Intelligence for a Symbiotic Future…

As populations in developed countries age and labor shortages intensify, Cybernetic Avatars (CAs) are proposed to extend human capabilities through robotic embodiments, requiring effective Human-Robot Interaction (HRI) frameworks. Extended Reality (XR), an umbrella term for Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR), offers such interfaces, but prior research typically fixes the XR modality without evaluating its effect on task outcomes. This study examines whether the XR modality impacts HRI performance and whether an adaptive interface adjusting the level of virtuality along the Reality-Virtuality Continuum (RVC) at runtime improves it. A custom XR application interfaced with a mobile manipulator supports immersive control and runtime modality switching. In a within-participant multi-room pick-and-place experiment comparing fixed AR, AV, and VR with dynamic RVC through task metrics, the NASA-TLX, and the System Usability Scale (SUS), this study demonstrates that 1) the fixed reality modality affects HRI results, and 2) dynamically changing the modality along the RVC improves them. AR yielded significantly lower mental demand, effort, and frustration than AV and VR, while the dynamic RVC condition achieved the highest throughput and lowest workload, highlighting the value of adaptive XR interfaces for human-robot symbiosis. The implementation is available at https://github.com/CarlTornberg/XR-HRI.

0.38

Gaze Under Strain: How Oculomotor Behavior Tracks Cybersickness Progression in VR

cs.HC ·2026-10-23 ·Nevzat Umut Demirseren, Mehedi Hasan Raju, Oleg Komogortsev, Isayas Berhe Adhanom
published similarity 0.59
Abstract · Accepted for publication at ACM VRST 2026. 10 pages, 4 figures

Cybersickness remains a major barrier to comfortable, accessible virtual reality (VR). Arising from mismatches between visual and vestibular signals, it places sustained demands on gaze stabilization and visual processing. Because gaze is the mechanism through which users actively sample the virtual environment, cybersickness-induced disruptions may alter both what users perceive and how discomfort itself emerges. Yet, how eye-movement behavior evolves alongside discomfort during prolonged VR exposure remains poorly understood. We present a behavioral analysis of eye-tracking data from 19 participants navigating a realistic virtual environment over a 20-minute session, with discomfort rated repeatedly throughout the experiment. Linear mixed-effects models examined gaze constructs spanning fixation stability, visual exploration, and gaze allocation as functions of discomfort, exposure duration, and their interaction. Our results show that fixation stability degraded progressively with increasing discomfort, with the effect strengthening over time. Visual exploration showed no marginal discomfort effect but exhibited significant discomfort-by-time interactions, indicating that saccadic behavior adapts dynamically rather than responding uniformly to discomfort. Gaze allocation was not associated with discomfort but shifted systematically with time. These findings show that cybersickness does not uniformly disrupt gaze. Instead, different aspects of eye-movement behavior respond differently and change in ways that only become visible when discomfort is tracked repeatedly over time. Understanding how gaze evolves during cybersickness progression provides the behavioral grounding needed to better interpret how discomfort shapes the user's visual experience of immersive environments and to design more sensitive detection approaches.

0.31

Slide to Report: Continuous Reporting Symptom-Specific Cybersickness in VR

cs.HC ·2026-11-05 ·Purnata Saha, Rifatul Islam, Qiliang He, John Quarles
similarity 0.59
Abstract

Cybersickness is often assessed during VR exposure through discrete overall sickness, which cannot capture specific symptoms or their changes over time. Post-exposure questionnaires such as the Simulator Sickness Questionnaire provide symptom-specific measures but cannot be collected during immersion. We investigate whether separating cybersickness into three symptom-specific dimensions -- nausea, oculomotor discomfort, and disorientation -- during immersion provides more informative continuous measurements. We conducted a between-subjects study with 52 participants in a VR-rollercoaster, comparing a three-slider interface with a single-slider interface. Concurrently, participants also completed an auditory Simon task to measure attention. We additionally collected multimodal set of physiological signals, including GSR, ECG, PPG, eye and head tracking. Results show that each slider correlated most strongly with its corresponding SSQ subscale, whereas the single slider correlated with all SSQ subscales. Using three sliders did not significantly affect usability, workload, or attention. We publicly release the multimodal dataset to support future cybersickness research.

0.20