CS Paper Radar

Evaluating Beyond the Screen: Collective Assessment of AI-Generated Business Plans with Resource-Constrained Entrepreneurs

Qi Zhao, Marjory Pineda, Ketul Chhaya, Aakash Gautam, Yasmine Kotturi

cs.HC ·submitted 2026-08-17 17:59 UTC · v2
published

Abstract

Entrepreneurs increasingly use end-user generative AI technologies such as ChatGPT for high-stakes documents like loan applications and business plans, where AI-generated errors---a wrong price, a fabricated product---can affect loan or funding outcomes. Current approaches to supporting evaluation of AI-generated text assume a single user assessing output alone, on screen. This can be especially demanding for resource-constrained entrepreneurs, whose digital and AI skills vary widely. In this early-stage work, we explore how evaluation might instead be organized in a group setting and completed as a collective activity. We extended BizChat, an AI-powered business-planning tool, with an evaluation module that links each generated claim to the entrepreneur's original input. We partner with community organizations in Maryland---embedding BizChat within various entrepreneurship programs---where workshop attendees (N=14) evaluated their plans through think-pair-share discussion. Early findings suggest interface scaffolds like claim-to-input links primed attendees with concrete, personal evaluations, which the group setting then extended beyond the screen: attendees requested printed copies, used rubrics to compare across plans, and drew on peers' knowledge to verify what they could not easily judge alone.

Comment
Accepted to ACM Collective Intelligence 2026

Closest papers

Nearest neighbours by meaning across the whole index.

We Ran a Research Sprint With AI. The AI Worked. The Research Didn't

cs.AI cs.LG ·2026-11-02 ·Anush Sankaran, Arijit Banerjee
position similarity 0.62
Abstract

Generative AI tools for research are widely deployed, yet individual productivity gains do not guarantee better team research. We argue that how AI use is structured matters alongside access. Personal AI tools may amplify two failures under deadline pressure: a consumption bottleneck, in which generation outpaces team-visible evaluation, and AI-induced research silos, in which private human-AI loops displace shared decision-making. We motivate this position with a mechanism-generating pilot: a 100-minute research sprint with twelve researchers in three independently operating teams, within a seventeen-person session. Both patterns recurred across teams and evidence sources. Of 33 logged AI interactions, one was accepted outright and 27 lacked a recorded disposition; missing records alone do not establish absent evaluation. A parallel autonomous pipeline produced a complete draft without in-session human evaluation, illustrating production capacity rather than comparative quality. We propose Human-AI-Human collaboration: an interaction topology in which AI connects researchers' contexts and supports inspectable collective evaluation and decision-making. The pilot does not isolate AI-specific effects from time pressure or workflow practices. We close with seven modular, falsifiable design hypotheses, prioritizing shared context and team-visible evaluation checkpoints for controlled follow-up.

0.23

Improving Entrepreneurial Skills by Practicing with AI

cs.CY ·2026-11-01 ·Amir Globerson, Anisha Choudhury, Anna Iurchenko, Avinatan Hassidim, Cathy Cheung +19 more
similarity 0.61
Abstract

Durable skills like critical thinking, creativity, and collaboration are essential for innovation, yet teaching them remains difficult. In this work, we focus on the entrepreneurial domain, where these skills and others are required, and ask whether AI can be used as a tool for improving entrepreneurial proficiency. As a first step towards this challenge, we present an extension of Vantage, a platform developed at Google Research, adapted to serve as an interactive practice environment for entrepreneurship. Vantage pairs students with simulated AI peers engineered to navigate realistic business scenarios and challenge learners to exercise key skills. In a randomized controlled trial (RCT) embedded in summer entrepreneurship programs run by The Network for Teaching Entrepreneurship (NFTE), students using Vantage scored significantly higher on their Capstone Business Plans than control students with access to standard AI tools. These preliminary findings suggest that orchestrated AI peers can meaningfully support short-term performance on tasks requiring complex, durable skills.

0.30

From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost

cs.CL cs.AIcs.HC ·2026-09-17 ·Saki Imai, Mert İnan, Malihe Alikhani
top venue similarity 0.61
Abstract · EMNLP 2026

AI productivity is often measured by task completion time, economic value, or improvements in outcome quality. However, these measures usually treat collaboration as a black box where they capture what output was produced, but not the interaction cost required to produce it. Motivated by economics literature, we introduce a productivity-oriented framework for evaluating human-AI collaboration as outcome quality relative to interaction cost. Across two datasets spanning four tasks, we show that: (1) sessions with identical quality ratings can differ by up to 70 times in interaction cost; (2) quality-cost relationships vary by task, with some tasks rewarding extended interaction and others favoring fast convergence; (3) subjective user ratings are not reliable substitutes for productivity; and (4) productive sessions are characterized by agents probing earlier and users spending less effort repairing the interaction. By distinguishing productive success from costly success, our framework makes interactional cost visible and shows how dialogue analysis can inform the evaluation and design of AI systems.

0.73

DocuTeam: Mixed-Initiative Multi-Agent Discussions around Evolving Documents

cs.HC cs.AI ·2026-09-24 ·Heechan Lee, Juhyeon Choi, Tae Soo Kim, Juho Kim, Joseph Seering
similarity 0.61
Abstract

In open-ended problem solving, collaborators often rely on discussion to surface concerns, challenge perspectives, and refine shared work as it evolves. While AI agents are increasingly used as discussion partners, existing multi-agent systems place a heavy burden on users to initiate and carefully orchestrate the discussions. We present DocuTeam, a mixed-initiative multi-agent discussion system in which both users and agents can initiate and steer conversations. Agents monitor document changes to proactively start and redirect discussions as the work evolves, while users can flexibly shape the conversation or adopt agent ideas. In a within-subjects study (N=20), participants using DocuTeam produced outcomes rated significantly more novel, relevant, and specific than with a baseline without any increase in cognitive load. Rather than using agents for one-off idea sourcing, participants engaged in an iterative refinement loop in which document changes prompted agent reactions, which led users to revisit and further develop their work.

0.07

"ChatGPT, what am I missing?": Designing AI Workflows around Professional Task Structure to Shape Analytic AI Use

cs.HC ·2026-09-15 ·Zilin Ma, Suzi Jazmati, Marco Chimenton, Yiyang Mei, Jacqueline Lane +2 more
similarity 0.60
Abstract

General-purpose AI lets users choose what support to request, but leaves them to structure the support a professional task requires. We examine how interactive workflows can embed professional task structure without prescribing how users engage with AI. We designed two scaffolded interfaces around the same negotiation scaffold: one presented a completed AI analysis, while the other supported user-directed, incremental development. A four-condition randomized experiment with 800 participants compared these interfaces with no-AI and an AI chat interface. AI-supported conditions improved preparation coverage over unaided work; the scaffolded workflows further improved coverage over chat. Although the scaffolded workflows produced similar coverage, the user-directed workflow elicited a broader repertoire of analytic requests and lower subjective effort. Professional scaffolding therefore depends not only on displayed structure but on how workflows organize users' engagement with it. Effective professional AI must structure how users and AI build analysis together.

0.19

Linking Event-sequence Data Claims to Analytic Provenance in Chatbots: Technique and Evaluation

cs.HC ·2026-11-08 ·Po-Ming Law, Jianheng Ouyang, Nguyen-Truong Thinh, Weizhi Li, Arpit Narechania
similarity 0.60
Abstract

Chatbots are increasingly used to analyze temporal event sequences of which website clickstreams are among the most common. Explanations are essential for understanding claims about event-sequence data. However, existing systems often require programming expertise to interpret these explanations and fail to explain the data processes underlying the data claims. We introduce a technique for linking a data claim to its analytic provenance (i.e., the analytic steps and results that produced it). It has two components: a reference table that records analytic steps and results in addressable cells, and a citation format that binds each claim to one or more cells. To operationalize this technique for clickstream analysis, we curated a set of analytic operations for clickstream analysis, organized them through a pivot-table abstraction, and developed tools that an AI agent can invoke to create a reference table and assemble a natural language response with cited data claims. To evaluate how citations affect users' assessment of data claims, we conducted two crowdsourced experiments with 90 participants each. The experiments showed that participants perceived correct data claims to be more accurate with citations when compared with no explanations, and that they perceived incorrect data claims to be less accurate with citations when compared with code and no explanations.

0.10

Executable Human Study Programs with Silicon Participants

cs.HC ·2026-11-09 ·Xuan Liu, Haoyang Shang, Yuanjun Feng, Shao-Yu Chu, Qiyu Li +2 more
similarity 0.59
Abstract · 15 pages, 12 figures, 8 tables

We introduce HumanStudyProgram, a framework that represents a documented human study as an executable program with configurable LLM-based silicon participants. Its computational abstraction separates study logic from participant implementation through standardized APIs for presenting information and collecting responses. We implement a researcher-facing authoring tool that reconstructs source-linked study descriptions from papers and supplementary materials, supports researcher review, and generates executable HumanStudyProgram artifacts. Researchers can then configure silicon participants, run the programs, and analyze results from aggregate findings to individual responses. We demonstrate the approach through study cases and evaluate the tool through a hands-on expert review with 12 social science researchers. All completed the workflow and identified potential uses in understanding published protocols, prototyping studies, and examining AI behavior. We aim to improve the reproducibility of human studies by reconstructing their documented procedures as programs that others can reuse and build upon.

0.39

Treadstone: A Social-Media-Inspired Platform for Multi-Agent Collaborative Data Analysis

cs.HC ·2026-09-17 ·Hyunwook Lee, Sungbeom Cho, William Benjamin, Changhee Lee, Hyotaek Jeon +4 more
published similarity 0.59
Abstract · 11 pages, 10 figures, 2 tables, 2 page-length appendix. Accepted to VIS 2026 and will be published on IEEE Transactions on Visualization and…

Coordinating human analysts with autonomous AI agents faces the same challenges as human-to-human collaboration: sharing intermediate results, avoiding conflicts, and maintaining group awareness. Current tools rely on unstructured messaging or single-threaded chatbot interaction, which lack the structure to track evolving hypotheses or link claims to evidence. We propose agentic social data analysis, a collaboration paradigm extending social data analysis with a shared coordination feed modeled on the content timeline in social media services. We instantiate this concept in TREADSTONE, a platform where human and AI agents asynchronously post, link, and contest analytical claims via threaded messages within a shared feed. By allowing agents to proactively broadcast hypotheses and enabling users to steer the analysis through lightweight curation, Treadstone seeks to balance machine autonomy with human analytical control. A qualitative user study shows that Treadstone fosters collaboration while preserving human analytical agency, in contrast to the solitary experience of conventional chatbot interaction.

0.56