CS Paper Radar

Sophistication in GenAI Use: Field Evidence from a Large Firm

Nicholas J. Hallman, Zachary T. Kowaleski, Anu Puvvada, Jaime J. Schmidt

cs.AI econ.GN ·submitted 2026-08-27 17:03 UTC · v1

Abstract

We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm. Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025. We document three main findings. First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities. Second, sophistication varies considerably across functions and is highest in Strategy, Digital Innovation, and Project Management, three groups that share a focus on firmwide strategic initiatives and organizational change. Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change. Together, our study provides measures of and insights into sophisticated genAI use that managers can use to improve outcomes and that researchers can use in future research.

Comment
59 pages, 4 figures, 8 tables. Includes Appendix A (variable definitions), Appendix B (supplementary tables), and Online Appendix C (meta prompts)

Closest papers

Nearest neighbours by meaning across the whole index.

Adoption of Generative AI in the Workplace: Increasing and Shifting the Balance of Productivity and Communication Activity

cs.HC ·2026-08-16 ·Yulin Yu, Yan Chen, Rui Hu, Siddharth Suri, Scott Counts
similarity 0.65
Abstract · This manuscript is being withdrawn because additional legal and internal approval processes are required by the company before public dissem…

Generative AI is transforming the workplace by augmenting and automating cognitive tasks, reshaping how organizations work and innovate while raising questions about workplace inequality and the future of work. Despite rapid adoption, empirical evidence on how these tools alter work practices and generate productivity gains remains limited. We examine how AI use affects the quantity and nature of information work using digital trace data from the Microsoft M365 application suite across multiple large international companies. Specifically, we study how generative AI adoption shifts the balance between communication and productivity-oriented activities, such as content creation in Word. Difference-in-Differences analyses show that AI adoption is associated with significant increases in both productivity (21.2%) and communication (7.1%) application actions among users who used the AI system more than 100 times over a 20-week post-adoption period. Among users with 100-500 AI use instances, higher AI usage is also associated with continued increases in both types of activity. The smaller increase in communication represents an overall shift toward individual, documentation-focused work and reflects mixed changes in communication, including decreases in reading and organizing email, compared with more uniform increases in productivity actions. These findings suggest potential efficiency gains and reductions in information overload, while highlighting the need to ensure that AI adoption does not weaken interpersonal communication and the diffusion of diverse information that supports innovation.

0.38

How Organizations Use AI: Evidence from ChatGPT

cs.AI econ.GNcs.HC ·2026-08-12 ·Aaron Chatterji, David Holtz, Neel Rakholia, Prasanna Tambe, Gawesha Weeratunga
similarity 0.63
Abstract

We study how organizations use frontier generative AI by linking ChatGPT Enterprise account records to usage, worker roles, task classifications, and public-company financial data through March 2026. These linked data enable a privacy-preserving analysis of adoption, worker roles, and message-level tasks at scale: for instance, the worker-level sample we analyze at the six-month adoption horizon includes over 1,500 organizations and over 17 million messages. We document four facts about enterprise AI adoption and use. First, ChatGPT Enterprise usage has grown rapidly due to a combination of new firm adoption and growing intensity among existing adopters. Second, U.S.-based public company adoption is concentrated among larger, more valuable, and more R&D- and SG&A-intensive firms. Third, active use within adopting firms spans job functions and seniority levels, with especially high usage intensity among early-career workers. Fourth, ChatGPT Enterprise usage encompasses a broad range of knowledge work tasks, including writing, technical work, communication, and information synthesis. In aggregate, these results suggest that firms differ widely in the speed, breadth and purpose of their enterprise AI adoption, and that they are still actively learning how to integrate AI into organizational workflows.

0.41

Barriers to GenAI Integration During Large-Scale Agile Transformation: An Exploratory Case Study

cs.SE ·2026-11-19 ·Nils Brede Moe, Marthe Berntzen, Astri Barbala, Viktoria Stray
published similarity 0.62
Abstract · 10 pages. Accepted for publication in the Proceedings of the 60th Hawaii International Conference on System Sciences (HICSS 2027)

Organizations undergoing large-scale agile transformations are increasingly introducing generative AI (GenAI) tools into everyday work. While GenAI promises to accelerate software development, less is known about how organizations integrate GenAI into existing large-scale agile ways of working and how it interacts with tensions related to autonomy, alignment, learning, and coordination. We examine barriers to GenAI integration through an exploratory case study of a large Nordic media organization undergoing a large-scale agile transformation. The findings reveal team-level barriers, including diverging AI beliefs, lack of shared practices, and limited experimentation capacity, and organizational-level barriers, including uneven AI maturity, weak organizational learning mechanisms, and cross-unit coordination and alignment challenges. Our findings suggest that GenAI does not create new organizational challenges but becomes intertwined with existing tensions in large-scale agile transformation and may intensify challenges related to learning, coordination, governance, and trust. We contribute to large-scale agile transformation research by showing how GenAI interacts with ongoing organizational change and by identifying conditions for sustainable GenAI integration at scale.

0.47

Orchestrating GenAI for Interdisciplinary Research

cs.HC ·2026-09-24 ·Shirley Anugrah Hayati, Moyan Zhou, Patricia Anugrah Setiani, Ruizi Wang, Joseph Chee Chang +1 more
similarity 0.61
Abstract

As researchers tackle interdisciplinary problems, they face the need to deepen expertise in primary areas while rapidly acquiring knowledge in secondary domains. Generative AI (GenAI) is increasingly positioned to meet this need, from general-purpose chat assistants to Deep Research tools marketed as autonomous research agents. Prior work has examined how researchers use GenAI to support single-discipline or general research tasks. However, we know little about the goals and GenAI practices in interdisciplinary research. We conducted a longitudinal study and semi-structured interviews with 15 interdisciplinary researchers to examine how interdisciplinary researchers actually orchestrate GenAI. Findings show that researchers leaned on GenAI to fill knowledge gaps while maintaining epistemic agency for novelty discovery. We also uncovered an expertise paradox: GenAI outputs were hardest to verify when most needed. Our empirical insights motivate GenAI designs that calibrate verification to researchers' expertise, nudge toward cross-domain synthesis, and adapt prompting and outputs to disciplinary conventions.

0.33

From Producing to Validating: How AI Is Deskilling Freelancers

cs.HC ·2026-08-26 ·Nakul Rajpal
position similarity 0.60
Abstract

Generative AI is promoted as a way to enhance knowledge work, yet its benefits and drawbacks fall unevenly across the workforce. Freelance and gig workers, who commonly lack the upskilling pathways available to traditional employees, face heightened risks to both skill development and job security as AI adoption advances. We review empirical evidence on AI's impact on knowledge-worker workflows and upskilling, then predict the primary and downstream effects of AI adoption among clients and workers in the freelance economy. We anchor this in two cases of the same shift, machine-translation post-editing and software development. We argue that freelancers are the leading edge of a change that also reaches salaried HCI practitioners, and we close with questions for the platforms and clients that mediate this work, and for HCI researchers.

0.31

Beyond Productivity: Measuring Developers' Cognitive Load During GenAI-Supported Software Development

cs.SE cs.HC ·2026-09-29 ·Charlotte Brandebusemeyer, Daniela Gasser, Tobias Schimmer, Bert Arnrich
similarity 0.60
Abstract

Generative AI (GenAI) is changing software development workflows and how developers work. Industry evaluations of GenAI adoption often monitor productivity gains, usage, and output quality, but limited attention is paid to the interaction experience and cognitive load of the actual adopters and drivers of GenAI technology - the software developers. Understanding whether GenAI changes or shifts developers' cognitive demands during everyday development is important for a developer-centered evaluation of GenAI-supported software development. It can inform organizations in designing and evaluating effective AI-supported workflows. In this work, we study how GenAI use and task context relate to professional developers' perceived cognitive load and whether wearable-derived physiological characteristics provide additional information beyond this context. In a four-day industrial field study at two SAP sites, 21 developers documented their tasks, task duration, GenAI use, and perceived cognitive load while wearing an EmbracePlus wristband. The results show that perceived cognitive load is associated with both GenAI use and task context, while physiological measures provide only limited additional information. These findings suggest that developers' perceived cognitive load during GenAI-supported software development should be evaluated in relation to the concrete work context, with wearable physiological data used as complementary rather than standalone information.

0.12

Empirical Insights from AI-Powered Software Engineering in Practice

cs.SE ·2026-10-16 ·Tina Mersch, Jack Johns, Steffen Klepke, Hanya Elhashemy, Artur Signell +2 more
similarity 0.59
Abstract · Submitted to IEEE Software

Generative Artificial Intelligence (GenAI) is becoming indispensable in future software engineering. Its application spans a wide spectrum - from assisted development to autonomous software engineering, requiring organizations to address associated risks and challenges. This paper investigates these issues through three case studies, highlighting practical problems encountered during GenAI integration. Based on these findings, it provides targeted recommendations for managing risks and ensuring reliable outcomes. This paper provides guidance for effective and responsible GenAI adoption in software engineering based on empirical insights.

0.11

The GenAI Who Knew Too Little: Revisiting Transactive Memory Systems in Human GenAI Collaboration

cs.HC cs.CY ·2026-10-20 ·Christian Meske, Tobias Hermanns, Florian Brachten
published similarity 0.59
Abstract · Full paper accepted and presented at the 20th International Conference on Wirtschaftsinformatik (WI 2025), Münster, Germany. To appear in th…

Generative artificial intelligence (GenAI) is rapidly transforming knowledge work, creating new collaborative dynamics that challenge tradi-tional human-human interaction patterns. This study investigates how Transactive Memory Systems (TMS) manifest in human-GenAI collabora-tion. Through qualitative interpretive interviews with 14 knowledge work-ers and thematic analysis, we identified distinct TMS manifestations across its three main dimensions: (1) regarding specialization, expertise recogni-tion exhibited asymmetry. Humans developed awareness of GenAI capabili-ties while GenAI demonstrated only temporary recognition of human exper-tise; (2) in terms of credibility, humans formed ambivalent trust relation-ships with GenAI, simultaneously attributing expertise to the latter while implementing additional verification processes; and (3) concerning coor-dination, where humans primarily directed more passive GenAI systems lacking initiative, resulting in hierarchical interaction patterns and unidi-rectional information flow. Our findings reveal that TMS manifests differ-ently in human-GenAI compared to human-human collaboration, necessitat-ing theoretical adaptations including trust markers, acknowledging GenAI's absent metamemory, and identifying its temporary memory.

0.43