CS Paper Radar

"ChatGPT, what am I missing?": Designing AI Workflows around Professional Task Structure to Shape Analytic AI Use

Zilin Ma, Suzi Jazmati, Marco Chimenton, Yiyang Mei, Jacqueline Lane, Krzysztof Z. Gajos, Finale Doshi-Velez

cs.HC ·submitted 2026-09-15 01:13 UTC · v1

Abstract

General-purpose AI lets users choose what support to request, but leaves them to structure the support a professional task requires. We examine how interactive workflows can embed professional task structure without prescribing how users engage with AI. We designed two scaffolded interfaces around the same negotiation scaffold: one presented a completed AI analysis, while the other supported user-directed, incremental development. A four-condition randomized experiment with 800 participants compared these interfaces with no-AI and an AI chat interface. AI-supported conditions improved preparation coverage over unaided work; the scaffolded workflows further improved coverage over chat. Although the scaffolded workflows produced similar coverage, the user-directed workflow elicited a broader repertoire of analytic requests and lower subjective effort. Professional scaffolding therefore depends not only on displayed structure but on how workflows organize users' engagement with it. Effective professional AI must structure how users and AI build analysis together.

Closest papers

Nearest neighbours by meaning across the whole index.

A Tutorial on Prompt Engineering: From Messy Thoughts to AI Workflows

cs.CL cs.AIcs.HC ·2026-09-05 ·Erfan Loweimi, Hadi Daneshvar, Samira Loveymi, Samir Ouelha, Zhengjun Yue +2 more
similarity 0.69
Abstract

This paper treats prompt engineering as a discipline for turning informal human intent into structured AI work specifications. It develops the practice as a sequence of reusable design moves: define the work, construct only the context the answer depends on, choose a role, or a moderated panel of roles, as an attention lens, and state affirmative quality targets, reserving prohibitions for hard boundaries. To keep prompts lean, it adapts two classical principles, Occam's razor and Chekhov's gun, so that every instruction earns its place. For consequential tasks, it adds structured critique through steelmanning and premortems, followed by verification and, where tools or multi-step actions are involved, agentic operating loops with explicit boundaries and escalation. Aimed at a general readership, this tutorial is not a benchmarking study; it offers a practical, technically grounded path from casual prompting to disciplined AI workflow design, illustrated by a worked example that carries one task from a weak prompt to a strong specification. The framework is presented as principles and checklists that remain useful as models and tools change. The strongest prompt is rarely the longest prompt; it is the one that makes desired behaviour, required sources and checks, and success criteria unmistakable.

0.15

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

cs.AI cs.CLcs.HCcs.LG ·2026-10-15 ·Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang, Zhouyi Li, Khanh Chi Le +2 more
benchmarkdataset similarity 0.69
Abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.

0.48

Look Before You Prompt, and After: Scaffolding Human-AI Collaboration in Software Tutorial Creation

cs.SE cs.HC ·2026-09-03 ·Avinash Bhat, Vy Bui, Jin L. C. Guo
similarity 0.65
Abstract · 27 pages, 8 figures, 5 tables. Under Review at TOSEM Special Issue on Human AI Collaboration in Software Engineering

With LLMs, creating software tutorials now involves steering the model's output and shaping it into a coherent, accurate learning resource, yet existing LLM tools offer writers little support for this work. By analyzing interviews with technical writers ($N=17$), we identify three requirements for how they assemble and structure multiple LLM responses, curate the context the model uses, and verify the generated content. We designed a tool called dBlocks with the following features: blocks to scope content, a context manager to edit context, and inline execution to verify code. Following a human-centered design method, we iteratively refined the design through a user study ($N=5$). In a within-subjects lab study ($N=16$) comparing dBlocks with participants' preferred workflows for LLM-assisted authoring, participants reported significantly higher confidence in the tutorials they produced with dBlocks. In addition, the tool reduced friction in verification, with writers verifying code as they drafted rather than deferring or skipping it, and helped them avoid searching long chat histories by scoping their work into blocks that kept each tutorial section and its LLM conversation together. More broadly, our work offers implications for tools that scaffold human-AI collaboration in SE workflows and shows how human-centered design can guide the development of LLM-integrated tools.

0.37

From Review to Reuse: How Post-Task Workflow Can Support Human-AI Agent Interaction

cs.HC ·2026-09-11 ·Zekun Wu, Xinru Wang, Rock Yuren Pang, Chenglong Wang, Anna Maria Feit
similarity 0.64
Abstract · v2: supplementary appendix (survey questions, study materials) added as ancillary file

AI agents can automate tasks by turning a single natural-language request into a multi-step process spanning tools, files, and applications. Users are often left to judge that process from fragmented execution information and the final output. To make the completed process easier to understand, validate, and reuse, we investigate post-task workflows: editable, graph-based representations of an agent's completed execution. We first analyzed 10,803 public workflow templates from n8n to characterize real-world automation practice, then developed Trace2Flow, a research probe that translates agent execution traces into interactive post-task workflows. In a study, participants (N = 20) reviewed agent executions with prompt or agent errors. We found that post-task workflows improved their understanding and error detection over a prompt-only condition, and that validation succeeded mainly when users cross-checked across multiple evidence sources. For follow-up tasks, adapting the workflow matched adapting the prior prompt in success, time, and difficulty, and was often preferred.

0.14

MOONWALK: Mediating Operations with Intent-Evidence-Action Alignment Across Junior-Supervisor Review Workflows in Animation/VFX Pre-Production

cs.HC cs.AIcs.GRcs.MA ·2026-09-09 ·Shih-Yu Lai, Wen-Fan Wang, Sai Ling, Shaune Jan, Bing-Yu Chen +1 more
code/data similarity 0.64
Abstract

Animation and VFX pre-production review requires teams to translate loosely specified creative intent--briefs, evolving specifications, heterogeneous references, and verbal decisions--into revisions that junior artists can execute without repeated clarification. In practice, criteria drift across iterations, review judgments lose their evidential basis, and the reasoning behind a request rarely survives the senior-junior handoff. We contribute a design framework for intent-evidence-action alignment: intent is articulated into a shared project record, judgments are anchored to grounded evidence, and authorized decisions are converted into clear revision tasks tied directly to reference notes. We instantiate this framework in MOONWALK, a professional pre-production review system comprising a shared intent record, reference/specification anchoring, structured work-in-progress comparison, and supervisor-authorized action planning. In this workflow, AI handles administrative coordination--flagging missing context and organizing notes--while artists retain full creative direction. An in-studio study with professional practitioners compares MOONWALK with a chat-only (chatbot) interface using matched production materials, while participants' existing workflows provide a retrospective ecological baseline. Results indicate stronger intent alignment, decision traceability, and checklist executability, while also showing that aesthetic authority and final prioritization must remain with practitioners. The evaluation establishes the value of the integrated structured workflow over unstructured conversational AI chatbot. Code: https://github.com/Akinesia112/Moonwalk/tree/english-version

0.36

Human-AI Collaboration at Scale: Task Criticality, Agency, and Friction across 250,000 Conversations

cs.HC ·2026-08-17 ·Yijia Shao, Dora Zhao, Vishakh Padmakumar, Jennifer Wang, Diyi Yang
similarity 0.64
Abstract · 17 pages, 6 figures

AI is increasingly embedded in professional work, yet how people actually collaborate with it at scale remains poorly understood. Drawing on a corpus of 249,834 real-world conversations, we conduct privacy-preserving analysis by asking three questions: what work people bring to AI, what roles humans retain in completing that work, and where human-AI collaboration breaks down. We find that users bring to AI not only trivial tasks but also consequential, hard-to-reverse work, with high criticality delegation concentrating in advisory and professional domains. As stakes rise, users engage more actively with AI output, and conversation length nearly doubles from ephemeral to high-stake tasks. We then examine human agency, the degree of control and responsibility users retain in completing the task. Human-led collaboration dominates overall, but users construct their level of agency through how they prompt and iterate. Across different levels of agency lies a common tension: does working with AI build human capability, or substitute for human effort? We examine this tension through AI-supported learning and upskilling, finding that active teaching is common (67% of conversations) but its benefits are unevenly realized. We further show that friction arises in roughly half of all conversations but it is often productive, and users often employ active strategies to recover from friction when it does occur. Together, these results map how humans and AI work together at scale and when that collaboration makes people more capable, and when it doesn't.

0.50

Probing How Users Interact with Turn-Level Design Frictions for AI Chatbots

cs.HC ·2026-08-23 ·Helen Weixu Chen, Katy Ilonka Gero
similarity 0.64
Abstract · 32 pages, 8 figures, currently under review at ACM IUI 2027

AI chatbots can help people write faster, but they can also encourage overreliance by making it easy to turn minimal input into usable text. We study turn-level design friction: intentional constraints added to each chatbot exchange that slow, limit, or redirect how users request, access, or use model responses. We designed six friction probes, organized around three mechanisms: eliciting user contribution, restricting access to generated content, and reshaping system output. In a within-subject study with 24 participants, all six probes increased workload, task duration, and perceived ownership relative to a conventional AI chatbot, while their effects on recall and recognition were more selective. We further found that participants adapted to friction in different ways, and that the same constraint could support or obstruct involvement depending on users' goals and workflows.

0.48

The Future of Visualization Dashboards in the Age of Generative AI

cs.HC ·2026-09-28 ·Vaishali Dhanoa, Duosi Dai, Gabriela Molina Léon, Eduard Gröller, Niklas Elmqvist
similarity 0.63
Abstract

Generative AI promises easier dashboard creation, raising questions about the future of dashboards and the people who create and use them. We interviewed 16 experts based in 14 countries about their practices and expectations. Almost all expected dashboards to persist for recurring questions, monitoring, and reporting. They anticipated adaptive views and combinations of language, graphical controls, and gestures, while emphasizing interaction as part of human exploration and understanding. Participants expected authors' responsibilities to shift toward specifying requirements, curating generated work, and evaluating outputs, with design knowledge and communication remaining important. Easier creation also raised concerns about validation effort, users' understanding, maintenance, and personalization weakening shared understanding. We discuss seven opportunities for research and practice concerning validation, end-user education, dashboard proliferation and rot, organizational guidance, adaptation, novel visualizations, and accountability for AI-generated content. Our findings connect dashboard evolution with the human and organizational work needed to sustain their use.

0.08