CS Paper Radar

Toward Neuroadaptive Communication Interfaces: A Framework for Personalized Speech and Language Support

Carlos Bruno, Luis Ricardo Santos

cs.HC cs.CL ·submitted 2026-10-25 17:23 UTC · v1

Abstract

Speech and language technologies have expanded access to communication, but the support they provide is often fixed. The same linguistic or visual aid may help one user and hinder another, or become less useful as task demands change. This project proposes a framework for adapting language presentation to individual abilities and ongoing interaction. It separates access to a message from the estimation of user needs and from the selection of support, including sentence segmentation, pictograms, and optional spoken output. Behavioral feedback provides the primary evidence for adaptation; electroencephalography is considered an additional, uncertainty-aware source of information rather than a direct measure of comprehension. We specify a staged evaluation comparing fixed, behavior-adaptive, and neuroadaptive interfaces. The central question is whether neural information improves comprehension and communication efficiency beyond a strong behavioral baseline.

Closest papers

Nearest neighbours by meaning across the whole index.

From Neurons to Conversation: Speech Brain-Computer Interfaces

cs.HC cs.CLq-bio.NC ·2026-09-29 ·Moein Khajehnejad, Forough Habibollahi, Tommaso Boccato, Margarida Sousa, Michal Olak +2 more
position similarity 0.70
Abstract · Review article, 28 pages, 4 figures, 2 boxes, 2 tables

Speech brain-computer interfaces (BCIs) aim to restore communication by transforming neural activity related to speech, language, or communicative intent into external outputs such as text, synthesized voice, or avatar control. Recent advances in intracortical and electrocorticographic recording, deep sequence models, and language-model-assisted decoding have enabled rapid progress, including high-performance attempted-speech decoding and increasingly naturalistic speech synthesis. Yet these achievements also reveal that speech BCIs are not simply neural-to-text decoders. They are adaptive clinical systems in which neural representations, recording hardware, decoding architectures, language priors, feedback, and user learning interact over time. Here, we synthesize speech BCI research from a system-level perspective. We first examine the neural substrates of speech and language, emphasizing their hierarchical, distributed, temporally structured, and non-stationary organization. We then examine recording and decoding choices, closed-loop adaptation, evaluation, clinical translation, and ethics. Across these domains, we highlight recurring trade-offs between signal resolution and invasiveness, low-level motor and high-level semantic targets, decoder accuracy and user agency, and language-model fluency and faithful neural evidence. We argue the next generation of speech BCIs should be evaluated not only by offline accuracy, but also by robustness across sessions, calibration burden, latency, uncertainty, usability, and safeguards against unintended decoding. By reframing speech BCIs as adaptive, user-centred systems, we outline the interdisciplinary priorities spanning speech neuroscience, neural engineering, machine learning, clinical practice, and neuroethics needed to move from proof-of-concept decoding toward reliable, expressive, and controllable communication neuroprostheses.

0.20

Personalized Visual Support After Automatic Speech Recognition

cs.HC ·2026-11-03 ·Beatriz Calero, Duarte Neves, Carlos Bruno, Luis Ricardo Santos, Mario Egea
similarity 0.68
Abstract

Many spoken-language interfaces stop at producing a transcript. This project examines how a second layer of optional text and pictorial support can be adapted to individual needs. We specify four presentation levels and a behavior-based selection policy driven by replay requests, response latency and recent comprehension. After expert review of the mappings, we will compare fixed and adaptive presentation using counterbalanced spoken instructions. The study is designed to measure comprehension, usability and which modality listeners rely on, not to claim that transcript embellishment uniformly benefits listeners.

0.08

A Common Measure of Communication for Speech Brain-Computer Interfaces

cs.LG q-bio.NC ·2026-09-02 ·Dulhan Jayalath, Benjamin Ballyk, Oiwi Parker Jones
code/data similarity 0.68
Abstract · Code and OVMI Explorer available from the project page at https://neural-processing-lab.github.io/OVMI/

Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.

0.68

Brain-to-Language Decoding: Paradigms, Signals, Methods, Evaluation, Practical Use and Beyond

cs.CL cs.HCcs.NE ·2026-09-23 ·Yiqian Yang, Yiqun Duan, Chenyu Liu, Yiqi Wang, Xinliang Zhou +2 more
survey similarity 0.67
Abstract

Brain-to-language decoding translates neural activity associated with language production, internal speech and perception into linguistic or expressive outputs. It offers a route to restoring communication after speech loss and a means of studying how the brain represents language. Advances in neural recording and representation learning have expanded the field from constrained recognition and acoustic reconstruction to text generation, streaming personalised speech and facial animation. This survey synthesises these developments across invasive and non-invasive measurements. We connect Articulated, Inner and Perceived paradigms to the neural populations they engage, the representations available to decoders and the outputs those representations can support. We examine model development, public resources and the evolution of evaluation, and compare published performance and communication costs within their reported protocols. The synthesis identifies complementary routes to progress: phonetic, acoustic and semantic targets preserve different aspects of a message; shared representations support reuse across recording conditions and tasks; and online communication increasingly depends on calibration, feedback and user control alongside decoding accuracy. Shared benchmarks enable algorithmic comparisons, while longitudinal studies reveal the demands of sustained use. We discuss these developments and their remaining limitations, then outline a prospective five-level trajectory from commands and language to meaning, scenarios and bidirectional cognitive exchange.

0.53

Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality

cs.HC ·2026-10-17 ·Sina Elahimanesh, Joao Marcelo Evangelista Belo, Isidora Jeknic, Alexander Koller, Anna Maria Feit
similarity 0.63
Abstract

Mixed reality interfaces must continuously adapt to a user's changing task and environment, yet adaptation policies are fixed in advance, either preset by the system or configured by the user. Natural language offers a way to state changing needs mid-task, but whether users can express interface behavior this way, and what it gains over direct manipulation, is unknown. We first elicited how users verbalize adaptation needs (N=19). We then built ConverseMR, which translates a spoken request into adaptation objectives and constraints and satisfies them continuously as the user's context changes. In a within-subjects study (N=24), conversational and direct control proved complementary: direct manipulation for precision, ConverseMR for reach and expressive breadth, with their combination rated highest. Spoken commands were rarely final and users frequently refined and replaced them upon experiencing the result, characteristic of, what we call, user-led adaptation, where users communicate adaption needs and the system identifies a solution.

0.11

FETTUCCINE: Fast and efficient brain-to-text decoding on mobile devices

cs.LG q-bio.NC ·2026-11-13 ·Jonathan D. McCart, Pranav I. Deevi, Mehdi Azabou, Nanda H. Krishna, Caleb A. McKinney +3 more
published similarity 0.62
Abstract · NeurIPS 2026

Recent advances in speech neuroprostheses have enabled new communication avenues for people who have lost the ability to speak due to neurological illness or injury. These systems can decode neural activity during attempted speech into text by relying on language priors to achieve high decoding accuracy. This performance currently comes at the cost of using a compute-intensive language modeling stack, presenting a major bottleneck for real-world use. Further, current language models incur highly variable and disruptive inference latencies, limiting communication throughput and preventing users from reliably participating in natural conversation. We introduce FETTUCCINE, an end-to-end brain-to-text framework that leverages the speed and portability of automatic speech recognition (ASR) models to address the aforementioned practical challenges, while achieving usable (<10%) word error rates (WER), efficient adaptation to neural non-stationarities, and reliable performance on mobile devices. When evaluated using publicly available data from an intracortical speech neuroprosthesis user, FETTUCCINE outperforms previous end-to-end methods, achieving as low as 4.39% WER. Most notably, our models successfully run on a commercially-available mobile device with throughput reaching up to 370x real-time, providing the first demonstration of brain-to-text decoding on a mobile device. We also show that our models can be successfully finetuned to extend usable performance to future days. Beyond performance, our approach enables localization of the neural features most relevant for decoding via gradient-based saliency maps, which we show align with well-established physiological priors across different ASR model families. Taken together, our results show that FETTUCCINE overcomes core infrastructural and computational barriers, yielding a new class of portable brain-to-text communication.

0.21

RecalibrateGPT: AI Fatigue Resilient Conversational Interfaces

cs.HC cs.AI ·2026-09-01 ·Nikhil Wani
top venue similarity 0.61
Abstract · 5 pages, 3 figures. Accepted at UIST Adjunct 2026

Large language models are powerful, but their interfaces often devolve into a type $\rightarrow$ read $\rightarrow$ retype loop, creating conversational AI fatigue, cognitive load, and eventual task abandonment. To mitigate this, we present RecalibrateGPT, a system introducing five cross-turn operators (Anchor, Replay, Delta, Scope, and Steer) that each target a distinct fatigue type, recalibrating LLM responses through a structured panel by acting on the full conversation history with a single click. Users invoke these operators through the AssistiveButton in one of three operator palette layouts: Vertical, Arc, or Tablet. We conducted two pilot studies with the same 12 advanced LLM users. An initial formative qualitative study identifies a taxonomy of four fatigue types (retyping, scanning, decision paralysis, and context drift) and derives two design objectives for RecalibrateGPT. A follow-up quantitative evaluation finds it reduces perceived cognitive workload by half (NASA-TLX = 2.7) at high perceived usability (SUS = 86.5), suggesting AI fatigue is not just a model-quality issue but an interaction-flow cost that interfaces can remove.

0.49

Decoding silent reading from non-invasive EEG

cs.LG q-bio.NC ·2026-08-20 ·Ingo Marquardt, Anthilia Alchanat, Priyanka Jain
similarity 0.61
Abstract · 45 pages (including 12 pages of Supplementary Material), 11 figures (including 2 in Supplementary Material)

Non-invasive decoding of inner speech faces a fundamental data problem: a corpus pairing brain activity with a person's spontaneous inner monologue cannot be collected, and the available proxy paradigms (cued repetitive and retrospectively reported generative inner speech) are slow to acquire, poorly time-locked, and subject compliance is unverifiable. We therefore treat silent reading as a scalable proxy task and ask how much lexical and semantic information a contrastive decoder can extract from it. We report an open-vocabulary analysis of approximately 240,000 word presentations recorded from a single densely-sampled participant across 393 runs (ca. 49 h) of 19-channel dry-electrode EEG. Words from continuous narrative text were presented in rapid serial visual presentation, with typography randomised on every trial to partially decorrelate word identity from low-level visual form. A convolutional EEG encoder, optionally followed by a causal transformer, was trained with a CLIP-style contrastive objective to align short EEG windows with hidden-state embeddings of the presented word taken from a large language model. Decoding, evaluated as word-grouped top-10 retrieval against permutation baselines, was reliably above chance, extended to mid-frequency and rare words, and scaled log-linearly with training-data volume with no sign of saturation. Removing occipital and posterior-temporal electrodes reduced the word-level gain by roughly one third but left context tracking unchanged. Control analyses separate word-level decoding from narrative context tracking and from a non-neural positional prior introduced by the transformer's positional embedding. These results establish that open-vocabulary word-level information is recoverable from EEG during silent reading, and that decoding is data-limited rather than saturated.

0.48