CS Paper Radar

AIfred: Augmented Learning through Functional Robotic Embodiment at the Desk

Gregorio Orlando, Milan Groshev, Eduardo Castelló Ferrer

cs.RO cs.HC ·submitted 2026-09-30 01:20 UTC · v1

Abstract

Desk-based learning and creative activities benefit from handwritten engagement. However, current generative AI tools deliver guidance through a separate screen, creating a gap between where users think and where assistance appears. To address this, in this work we design AIfred, a desk-based robotic arm with a projector mounted at the end-effector that places AI-generated guidance alongside handwritten work. AIfred combines workspace perception, context-aware content generation, and robot-mediated projection to support math assignments, image generation, and drawing tasks. In a user study (n = 36), we compared AIfred against ChatGPT (GPT-5.6 Luna) running on a laptop. Both tools performed comparably while assistance was available during the math assignment (6.7 vs. 7.3/10, p = .41), but AIfred improved short-term learning transfer by 60% once assistance was withdrawn (7.0 vs. 4.4/10, p = .003). In addition, independent art and design professors ranked drawings produced with AIfred better in 33 of 36 cases. Our findings indicate that spatially co-located AI assistance benefits tasks whose guidance shares a spatial frame with the work.

Comment
7 pages, 9 figures. Submitted to ICRA 2027. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Closest papers

Nearest neighbours by meaning across the whole index.

Exploring Visual Access and Proactive AI Tutoring for Handwritten Mathematics

cs.CY cs.HC ·2026-10-19 ·Nachiket Kotalwar, Conrad Borchers, Shubham Gandhi, Partha Sarathi Purkayastha, David Yaron +1 more
similarity 0.64
Abstract · 24 pages, 7 figures, 13 tables

Handwritten work captures mathematical reasoning that learners may not express in typed or voice-dictated messages. We built a multimodal AI tutoring platform in which a phone camera gives the tutor a live view of learners' work on paper. In a randomized online study, 117 adults solved calculus problems with Text-Reactive, Visual-Reactive, or Visual-Proactive assistance. Analysis of 2,140 tutor messages showed that visual access shifted assistance toward checking ongoing work, while proactivity enhanced error identification and directive hints. Learners valued feedback grounded in their writing; those who received proactive help rated it as supportive, and about a third wanted it more often. These findings inform when tutors should intervene and what they should address. We release the platform, prompts, and instruments, together with 255,984 timestamped images of handwritten work linked to tutoring conversations, intervention decisions, and learning outcomes, to support multimodal tutoring systems that preserve the affordances of handwritten work while providing adaptive feedback.

0.18

Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks

cs.HC cs.AI ·2026-09-21 ·Muzhe Wu, Zuchen Li, Xu Wang, Anhong Guo
similarity 0.63
Abstract

Visual instructions for physical tasks are typically authored in one context and followed in another, requiring users to translate demonstrated tools, materials, and spatial relationships into their own environment. We introduce Generative Tutorial, a conceptual framework for live visual instruction that depicts intended outcomes and actions within the user's environment and task flow. A formative evaluation of state-of-the-art image and video generation identifies failures and potential benefits across 15 physical tasks. Drawing on these findings, we build an augmented-reality prototype system that proactively generates goal images and demonstration videos using observed workspace context and predicted visual outcomes of preceding actions. A 24-participant lab study found higher task performance quality, greater perceived workspace correspondence, and shorter step-confirmation intervals with the system than with pre-authored guidance. Qualitative findings highlighted how contextual resemblance shapes trust, how generation errors affect interpretation, and how guidance delivery should adapt to users' needs, informing future designs.

0.41

PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

cs.AI cs.CLcs.HC ·2026-09-28 ·Shane K. A. Dalumura Hettige, Jonas Oppenlaender
benchmark similarity 0.61
Abstract · 25 pages, 7 figures, 11 tables

Figural divergent thinking is the ability to develop a given shape fragment into an original drawing. In humans, this ability is assessed with incomplete-drawing tasks. We introduce PainterBench, a benchmark that ports the incomplete-drawing task to the agentic setting. The agent draws on a canvas through tool calls and observes the result after every turn. The canvas includes a starting shape which cannot be erased, and the agent's goal is to incorporate this shape into the most original drawing it can produce. The task is open-ended, and the agent itself decides when the drawing is finished. The benchmark tests incremental visual planning over a short horizon and the transfer of creative ability from pretraining to multi-turn tool use. We evaluate 14 multimodal language models from small to frontier scale. Across the primary study and six sensitivity analyses, we collect 2,700 drawings and crowdsource creativity and recognizability ratings for every drawing and for 300 human reference drawings. We also present ViDrA-adapted, an automated scorer that predicts human creativity ratings of agent drawings (r = 0.85 on random held-out test split). Figural divergent thinking varies widely across the 14 models, and GPT-6 Astra produces the most creative drawings. Relative to the human drawings, the agent drawings score higher in creativity but lower in recognizability. We release the final drawings, per-round canvas snapshots, tool call traces, stimulus bank, benchmark harness, crowdsourced ratings (N = 72,000), and ViDrA checkpoint.

0.25

Artly: Exploring Digital Artists' Perceptions of AI-Generated Feedback

cs.HC ·2026-08-17 ·Ulvi Rajabli, Alexander Wiethoff, Zelun Tony Zhang
published similarity 0.61
Abstract

Recent developments in generative AI have lowered barriers to image generation, but existing tools mostly optimize for efficiency, producing generic results and offering little support for artistic growth. We present Artly, an AI system that combines personalizable AI feedback with human-authored learning resources. In a between-subjects study with artists, we compared a mode without image generation features against one that allowed to generate variations of users' illustrations. Artly was perceived as helpful for learning and self-improvement, with the exception of the most proficient participants. Participants who used the image generation feature interacted slightly less with the AI feedback. They reported feeling more creative after using Artly than participants using the restricted mode, while reporting slightly lower scores on new ideas for their work. Overall, our findings underline the potential of our feedback approach for supporting artistic growth in a manner that is well received by artists.

0.19

Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum

cs.RO ·2026-10-20 ·Da Huang, Jichao Wang, Junhui Wu, Tong Ran, Linting Guo +6 more
highly cited similarity 0.59
Abstract

Recent studies have explored the potential of multimodal large language models for robot control, yet the models' embodied capabilities remain insufficiently characterized across models, tasks, and execution settings. We conduct a multidimensional comparative evaluation on RoboColiseum across four tracks: Instruction Following, Spatial Reasoning, General Manipulation, and Robustness. We implement Agent2Act, a control harness that integrates GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, and Gemini 4 Argon into the evaluation environment through shared observation, action execution, and feedback interfaces. Under common task settings and scoring criteria, all four models exhibit a consistent capability profile: high instruction-following scores, with three approaching perfect performance; strong spatial reasoning; and general manipulation scores below those of the other three tracks. Within the evaluated configurations, Astra leads instruction following and robustness, while Opus leads spatial reasoning, general manipulation, and overall performance. These results reveal complementary strengths across models and a persistent gap between task understanding and reliable physical execution. Component comparisons on selected tasks further show that harness design affects task success, planning cost, and execution time. Additionally, case studies with Astra examine whole-body coordination, manipulation across robot embodiments, precision manipulation, dexterous-hand manipulation, and real-robot tasks, providing further insight into the model's capabilities and limitations across embodied tasks.

0.38

Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

cs.RO cs.AIeess.SY ·2026-10-01 ·Yen-Jen Wang, Haozhe Jiang, Shuying Deng, Haoru Xue, Weirui Ye +5 more
highly cited similarity 0.59
Abstract · 17 pages, 6 figures, 10 tables

Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks. Project Website: https://rpg-robot.github.io/

0.57

TouchTwin: Human-AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch

cs.HC ·2026-10-12 ·Wanli Qian, Xinlei Yu, Heather Culbertson
published similarity 0.59
Abstract · submission CHI2027

Creating haptic experiences from real-world environments remains difficult: visual scans capture appearance but not material feel, while manual haptic authoring requires specialized expertise. We present TouchTwin, a human-AI approach that turns phone scans of tabletop scenes into editable haptic scenes. TouchTwin uses AI to propose material regions that users can experience and revise through language and spatial edits. A model trained on measured tool-surface interactions generates the haptic responses, allowing users to work with material descriptions rather than low-level signal parameters. In a study with N=12 participants and physical reference objects present, automatic drafts scored 4.0/7 for realism and 5.4/7 as useful starting points. After a median of 6.6 minutes of editing, realism increased to 5.6/7, while scoring the saved material layers against ground truth showed material-family accuracy improving from 52% to 92%. Participants also used the same workflow to author material feels without a physical reference.

0.39

Human-AI Co-Ideation Across Socio-Technical Configurations: An Interactional Analysis of Planar and Immersive Workspaces

cs.HC cs.AI ·2026-11-16 ·Orit Shaer, Mary Calabro, Zoe Mitchell, Angelora Cooper, Raffaella Sadun +2 more
similarity 0.59
Abstract

Remote collaborative platforms increasingly integrate generative AI to support group ideation. These platforms are sociotechnical configurations in which spatial mediation, interaction modality, AI output format, and the mechanisms that make contributions visible to a group co-vary as properties of a deployed system. This paper examines how such configurations shape participants' collaboration and their engagement with AI during creative work. We report a comparative study of two commercially available platforms that instantiate contrasting configurations: a planar shared canvas with text-based AI integration, and an immersive virtual reality workspace with concise AI output embedded in the shared space. Fourteen all-women triads completed a structured ideation task consisting of human-only idea generation, AI-supported idea expansion, and group convergence. We analyzed collaboration patterns, selection of AI-generated ideas, perceived idea quality, workload, presence, and their lexical characteristics. The immersive configuration yielded greater spatial presence and introduced a participation paradox, in which spatial freedom enabled exploration and fragmented group coherence; it was also associated with more balanced role distribution during convergence, at the cost of higher physical demand. In both configurations, AI multiplied the idea pool, and ideas groups authored with AI available were rated above participants' own. AI output format: about 21 words per idea on the planar canvas and 3 in the immersive workspace. Tracked how participants evaluated AI-generated ideas and how often they selected them unaltered as finalists. Neither configuration made AI contributions visible or attributable to the group: participants invoked AI individually and asynchronously in both. We derive design implications for AI output format, group-visible AI invocation, and spatial support for convergence

0.07