CS Paper Radar

Measuring AI Leadership: Development and Validation of a Multidimensional Measure for AI-Native Organizations

Mustafa Akben, Leslie Coyne

cs.AI cs.CYcs.HC ·submitted 2026-09-16 00:49 UTC · v1

Abstract

AI is changing what leaders must judge, explain, learn, and coordinate, yet existing measures do not capture these behaviors at the level needed to study leadership in AI-enabled work. We develop the AI Leadership Battery, which organizes 36 behaviorally specific subdimensions into 11 theory-specified content families. Following established scale-development procedures, the research used deductive item generation; content validation of definitional correspondence and definitional distinctiveness; exploratory factor analysis and item reduction; confirmatory factor analysis in independent samples; and tests of internal consistency reliability, convergent validity, discriminant validity, and criterion-related and incremental validity. Across the development and validation studies, the analyses provided evidence for the Battery's content, multidimensional structure, reliability, and distinction from selected orbiting constructs. The Battery also contributed additional information beyond orbiting constructs across organizational growth, decision speed, customer/stakeholder response capability, AI-enabled team performance, AI-enabled work experience, AI security and risk management, and AI adoption and integration. The resulting measure provides researchers with a behavioral framework for examining how leaders in AI-enabled work regulate judgment, learning, adaptation, transparency, and accountability.

Closest papers

Nearest neighbours by meaning across the whole index.

Using profiles of cognitive capability to assess AI suitability for workplace tasks

cs.AI cs.CYcs.HC ·2026-08-26 ·Jonathan Prunty, Marko Tešić, Patrick Quinn, José Hernández-Orallo, Lucy Cheke
benchmark similarity 0.65
Abstract

Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an agent's capabilities from performance on a benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.

0.19

The Dynamic Organization of Sustained Human-AI Cognition: From Construct-Level Change to Relational Structure

cs.HC ·2026-09-14 ·Zijian Ru
similarity 0.61
Abstract · 52 pages. This arXiv manuscript corresponds to Version 2 of the preprint posted on Zenodo on September 9, 2026: https://doi.org/10.5281/zeno…

As generative artificial intelligence becomes a routine participant in writing, learning, information retrieval, analysis, decision making, and problem solving, human-AI cognition research must address not only whether AI changes psychological constructs, use intensity, or task performance, but also how human cognitive activity is organized beneath similar aggregate indicators. This article proposes a dynamic cognitive organization framework that shifts analysis from construct-level change to relational organization anchored in the person's current task-cognitive state under sustained AI participation. The framework distinguishes five relational dimensions: execution locus, cognitive governance, representational reorganization, process organization, and reachable cognitive space; it also proposes a path-specific recursive principle whereby interaction outcomes, costs, and experiences may selectively reweight future probabilities of different organizational pathways. Five sets of testable propositions follow: the same overall AI-use intensity can correspond to different cognitive organizations; similar immediate outcomes can arise from different organizations with different predictive value for proximal subsequent outcomes; longitudinal organizational change need not track overall AI-use intensity; expansion of reachable cognitive space and displacement of pre-existing or emerging human-originated pathways may coexist within one episode; and recurrent cognitive organizations may redistribute cognitive practice opportunities, with accumulated differences potentially corresponding to different developmental trajectories in strategies, habits, and abilities. The contribution is an analytic level and five-dimensional relational structure for describing, comparing, measuring, and testing process differences that aggregate indicators or construct-level analyses do not uniquely determine.

0.33

Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking

cs.CY cs.AI ·2026-11-02 ·Victor Barres, Aden Barton, Nicolai J. Foss, Henrik M. Nielsen
position similarity 0.57
Abstract · 43 pages, 6 figures, 9 tables

AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks. Yet, many of these benchmarks treat technical tasks as the unit of analysis, while much economically valuable work takes place within organizations, where tasks are embedded in a system of relations. Within organizations, successful task performance depends not only on solving the technical demands of the task but also on navigating the interdependence of activities, distributed knowledge, differentiated authority, temporal constraints, conflicting interests, and changing environmental conditions while the task is underway. Accordingly, we argue that organizational task difficulty constitutes a distinct and insufficiently recognized dimension of AI capability, and that existing approaches to benchmarking provide an incomplete basis for assessing how effectively AI agents can perform real-world organizational work. By drawing on organization theory, we develop a taxonomy of the structure of work that characterizes the conditions that agents face in organizational work. Applying this taxonomy to analyze existing benchmarks, we find limited and uneven coverage of organizational dimensions, with fewer than one quarter of prominent benchmarks claiming any of them. We therefore argue that AI benchmarks should increasingly move beyond the evaluation of technically difficult tasks toward the systematic assessment of agents' ability to navigate the organizational conditions under which economically valuable work takes place.

0.23

Uneven Self-Reported AI Use: An Embedded Case Study Across Eleven Teams in a Software Organization

cs.HC cs.CY ·2026-11-10 ·Daniele Veri'
similarity 0.57
Abstract · 11 pages, 1 figure, 4 tables

This paper presents a descriptive, embedded single-case study examining self-reported AI use within a software organization. The study draws on 98 structured interview records collected across 11 teams, using ten standardized team-level instruments (n = 65) and one instrument with distinct response anchors (n = 33). Among respondents assessed using standardized instruments, 36 of 65 (55.4%) reported AI-use frequency scores of 4 or 5 on a five-point scale. Team-level proportions varied from 20.0% (1/5) to 71.4% (5/7). A record-level cross-tabulation revealed limited awareness or understanding of organizational AI policies across both frequency groups: 86.1% (31/36) of respondents scoring 4-5 and 96.6% (28/29) of those scoring 0-3 reported either no policy awareness or uncertainty about policy details. In the group assessed using the distinct instrument, 31 of 33 respondents (93.9%) scored 4 or 5. Although the pooled proportion reached 68.4% (67/98), differences in response anchors limit direct comparability, making the standardized-instrument results the primary analytical reference. Interviewer-recorded observations provide qualitative context on reported practices and constraints but do not support estimates of their prevalence or causal explanations for differences in AI-use frequency. The findings characterize AI-use patterns and policy awareness within the interviewed sample, without supporting inferences about organization-wide adoption, policy compliance, or the effectiveness of AI use. The study contributes an empirically grounded organizational case and an explicit account of the methodological boundaries of cross-team comparisons using heterogeneous interview instruments.

0.49

Governing Bring Your Own AI: A Parameterized Maturity Model

cs.CR ·2026-09-04 ·Dare Bello, John Hastings
survey similarity 0.57
Abstract · 8 pages, 3 figures, 5 tables

Employees are increasingly using personally owned generative AI tools such as ChatGPT, Gemini, and Claude for their daily work. This practice is known as Bring Your Own AI (BYOAI), which is a distinct form of Shadow AI in which employee-authenticated personal accounts are used outside of enterprise identity and security controls. Existing frameworks were designed for AI tools managed by organizations, and their coverage does not extend to unmanaged AI tools used with a personal account. In addressing these issues, we developed a governance model through a systematic review of the literature that produces a risk taxonomy and a framework-engagement profile. We also developed a parameterized governance model that measures how much a level of governance maturity reduces residual risk. A five-level maturity ladder is coupled to a technical control architecture through a chain in which the coverage of the control layer influences the security outcomes. Our study of a curated corpus of 30 records (24 research studies and 6 framework documents) indicated that the most prominent categories identified were data exposure and compliance, and framework engagement was inconsistent. Three mutually supporting pillars (technical, governance, and human) were established to support safeguards. Additionally, the results of the model demonstrated that prohibition-based solutions will result in residual risk levels close to those achieved through baseline solutions. Under the specified parameterization, layered control-based solutions substantially reduce modeled exfiltration risk and increase enforceable coverage.

0.36

Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use

cs.HC cs.AI ·2026-09-14 ·Daniele Veri'
hot direction similarity 0.56
Abstract · 21 pages, 4 figures, 4 tables

Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance. We conducted a structured, seeded review of 24 focal empirical publications, starting from the 2024 COSMIN-based review and adding a targeted update through 17 August 2026. We grouped the measures into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. In an exploratory meta-analysis, we pooled three direct subjective-objective correlations from one research program (REML r = .055; Hartung-Knapp 95% CI [-.047, .156]; combined reported N = 2,765). We could not resolve a discrepancy between the largest study's reported correlation and p-value, leaving its weight uncertain. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gave r = .079 (95% CI [-.025, .181]). This sensitivity analysis concerns a broader comparison. From this small evidence base, we cannot establish a population correlation, validate workplace cutoffs, or justify substituting self-ratings for performance scores. We identified tests of foundation knowledge (AICOS-S and GLAT) and measures of verification, reliance, trust, and dependency. We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure; some cover subsets. We propose a four-layer workplace battery with non-compensatory decision rules, but have not tested its thresholds or whether it improves on other assessment approaches.

0.37

The Moral Check: Strategic AI Governance for the Pacing Problem

cs.CY cs.AIcs.HC ·2026-09-19 ·Zaid Amin, Rahma Santhi Zinaida, Nazlena Mohamad Ali
similarity 0.56
Abstract · 60 pages, 9 figures, 10 tables. Preprint submitted to Computers in Human Behavior

Technology cannot steer itself. Strategy provides that steering, establishing the rule that purpose and judgment must precede compute capital. As the frontier artificial intelligence (AI) ecosystem accelerates exponentially, the pacing problem induces severe cognitive tunneling in engineering teams, prioritizing scalar throughput over human judgment. A calibrated pacing rate is imperative to check unchecked scaling, guarantee safety, and build models in whose alignment society can place warranted confidence. Traditional oversight fails through retrospective checklists, a pathology of performative governance exhibiting high procedural maturity but alarming scientific immaturity. We deliver a dual contribution: a PRISMA 2020 review synthesizing 130 empirical studies (MMAT-appraised across 18 benchmarks; total corpus N = 130 empirical studies across 178 reference foundations), and the Strategic AI Governance Ex-Ante Framework (SAGE-X). Our synthesis exposes two systemic vulnerabilities: the Recursive Assurance Paradox (correlated, ungrounded evaluator confidence) and the Durability Deficit (guardrail decay under multi-turn shifts). Grounded in MIT Strategic Computing doctrines, SAGE-X operationalizes Four Strategic Mindset Pillars: (1) Intent over Execution (mitigating velocity myopia); (2) Ruthless Trade-offs (deterministic tripwires eliminating moral hazard); (3) Outcomes over Outputs (auditing empirical hazard endpoints); and (4) Proactive Alignment (synchronizing ex-ante gates with runtime telemetry). Governed by a calculable Moral Check Index (MCI) with an unbypassable tripwire, SAGE-X delivers an operational Enterprise Lifecycle Audit Instrument (the "Moral Check Audit Card") on Stanford WebProtégé, ensuring exponential progress never outpaces deliberative moral judgment, human agency, and societal trust.

0.24

RAIL: An Automatic Classifier of the Artificial Intelligence Readiness Level

cs.AI ·2026-08-13 ·Juan Irving Vasquez, Juan Terven, Laura-Ivoone Garay-Jimenez
similarity 0.56
Abstract · Under review at journal

Assessing the maturity of artificial intelligence technologies is essential for investment decisions, project management, and policy monitoring, yet the available readiness frameworks are heterogeneous and difficult to apply automatically: the adaptation of Technology Readiness Levels to AI lacks AI-specific gating criteria, the Machine Learning Technology Readiness Levels presuppose access to internal process artifacts, and AI/data readiness dimension models employ scales that resist direct comparison. This paper makes two contributions. First, we unify these three frameworks into the Unified AI Readiness Level (AIRL), a nine-level ordinal scale built on an environmental evidence ladder and complemented by dimensional caps (covering specification, data existence, data quality, data legality, expert knowledge, and algorithmic maturity) together with a generality-anchoring rule and explicit assignment disciplines, so that a readiness level becomes decidable from a natural-language description of the work alone. Second, we propose RAIL (Readiness Assessment via Independent LLM-experts), a panel-of-experts classifier that operationalizes the scale: one evidence agent and six independent dimension agents, each a large language model with a narrowly scoped mandate, deliver verdicts that a deterministic minimum rule aggregates and a chief expert reviews under asymmetric authority, confirming or lowering the panel's recommendation but never raising it above the caps. The method was tested in the analysis of several research works showing consistency and avoiding overestimation from monolithic LLM classifiers.

0.06