CS Paper Radar

Building a Method: Challenges and Lessons from Developing an Impact Evaluation Framework for Public Investment in Open Source Software

Laia Domenech Burin

cs.CY ·submitted 2026-11-04 10:22 UTC · v1
publishedcode/data

Abstract

Modern societies and institutions depend on an invisible layer of open digital infrastructure: foundational, openly accessible digital technologies that support a broad range of social and economic activities. Open source software is a core component of this infrastructure, underpinning everyday technologies and critical systems in various critical sectors. As open source software becomes increasingly recognised, funded, and governed as open digital infrastructure, policymakers face growing demands for accountability, transparency, and credible evidence of impact. However, methods for evaluating public investment in open source software remain underdeveloped, with no widely accepted methodological standard. This paper describes the Sovereign Tech Agency's transition from qualitative reporting and funding-recipient self-assessment towards a mixed-methods approach that incorporates quantitative, goal-oriented evaluation to inform funding decisions. Drawing on the development of this framework, it examines its initial design, the practical and structural challenges encountered, and the responses adopted. The work identifies three central challenges: evaluation indicators must reflect heterogeneous project goals and contexts; quantitative assessment relies on fragmented and fragile data infrastructure; and observed changes before, during, and after investment do not establish causal attribution without additional research designs. The paper argues that evaluating investment in open source software is not only a methodological task but also a governance practice, dependent on sustainable open research infrastructure, community knowledge, and ongoing methodological refinement.

Comment
Working paper presented at the Open Technology Research Symposium 2026

Closest papers

Nearest neighbours by meaning across the whole index.

A curated global dataset of open science initiatives and policies

cs.DB ·2026-10-26 ·Abdelghani Maddi, Marc-Andr{é} Simard, Isabelle Dorsch
similarity 0.64
Abstract

Inventories of open science initiatives differ in scope, geographic coverage and classification, making it difficult to compare infrastructures, services, networks and policies. We developed COSMI Hub within the OPENIT project to bring these sources together in a curated dataset. The release archived on 8 October 2026 contains 1,186 records and 71 variables covering initiative identity, source provenance, geography, record level, operational status, thematic activities, governance, funding and external identifiers. We identified candidates through structured directories, policy documents, official websites and suggestions from the open science community, then checked identities, reconciled duplicates and assessed supporting evidence. The dataset distinguishes standalone initiatives, services, deployments, umbrella organisations and policy instruments. Records can be assigned several thematic dimensions, with uncertainty retained where evidence is incomplete. We describe the collection and coding methods, the data structure and file-level checks, together with considerations for reuse. The Zenodo release is fixed, whereas the companion dashboard is updated separately. Because we assembled the collection purposively, it cannot be used to estimate the worldwide number of open science initiatives.

0.28

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

cs.SE ·2026-11-05 ·Ronivaldo Ferreira, Vinicius Costa Girard, Cleidson de Souza, Igor Steinmacher, Carla Aguiar +2 more
similarity 0.57
Abstract · 11 pages, 4 figures, 4 tables

Governments increasingly use machine learning (ML) to handle data volumes beyond their analytical capacity, yet little is known about how requirements arise and evolve as an ML experiment moves toward operational use. Requirements Engineering (RE) for ML often assumes that requirements can be elicited upfront; in the public sector, however, demands arrive vague, stakeholders speak different professional languages, and what counts as a ``good'' result is unknown in advance. We report a 15-month case study of a thematic classification pipeline for Brazil's national digital participation platform, which holds over 60,000 citizen proposals, triangulating 43 project artifacts with interviews of six project participants. Institutional demands arrived as broad goals, endorsed but not sponsored, and only 40\% of the 95 reconstructed requirements and criteria were ever formalized. The pipeline's key requirements emerged largely from experiments, model outputs, and integration work, shaped by technical decisions and institutional constraints; the central one, the classification categories, was settled by a government taxonomy chosen for its institutional legitimacy. Acceptance relied on technical metrics, domain judgment, and political assessment, without shared thresholds or validation of usefulness in practice. In public-sector ML, technical and institutional artifacts thus act as elicitation instruments, and experimentation and integration belong to the requirements process.

0.39

Regulating for an AI Commons: Adopting, sustaining, and defending people-first innovation trajectories

cs.CY ·2026-10-16 ·Abhineet Nayyar, Mallory Knodel, Kimberlee Weatherall, Gunn Jiravuttipong
position similarity 0.56
Abstract

Technological principles of openness, interoperability, and decentralization are integral to building an AI economy that is accountable to the public and truly responsive to local needs and constraints. However, Big Tech's firm control over essential elements of the stack (such as chips, cloud, models, data, and talent) limits valuable alternatives from reaching the scale necessary for sustained growth. Worse still, hyperscalers continue to enclose and extract value from publicly funded and volunteer-driven technologies without adequately distributing these returns. We see consolidated power within a few entities and "enshittification" of once-transformative innovations. Embedding ethical micro-level tools (like open data repositories or small-language models) into workflows is promising, but without a consistent innovation strategy, they also often end up perpetuating and being appropriated by the same exploitative trajectories. Instead, this whitepaper outlines a three-pillared holistic and intentional policy approach that i) actively pushes for the adoption of cooperative technologies in existing and emerging systems, ii) sets up collective institutional structures to sustain and maintain them in the long-term, and iii) introduces regulatory proposals that limit private entities from disempowering them through uncontrolled enclosure and extraction. Through desk research and purpose-specific case studies, we argue for the importance of such an approach in the Global South, where structural constraints around infrastructure, financing, and policy governance make it difficult for AI commons and cooperative AI tools to emerge and survive. The paper concludes with key open questions for campaigners, policy advocates, researchers, and grassroots organizers to consider and explore while working towards peoples' solidarity within and between Global South countries.

0.15

You can contribute if you... An Empirical Framework of AI Contribution Policies in OSS

cs.SE ·2026-09-07 ·Gregorio Robles, Daniel M. German
code/data similarity 0.55
Abstract

Artificial intelligence is reshaping open source software (OSS) contribution by lowering the cost of producing code, documentation, issue reports, and review interactions. This creates opportunities for broader participation, but also disrupts how maintainers assess contributor effort, competence, and accountability. In response, OSS projects are beginning to regulate AI-mediated contribution through contribution guidelines and other project documentation. This paper presents an empirical study of these emerging policies. We analyze project policies on AI-mediated contributions by evaluating their underlying rationales, rules, and expectations. Our analysis shows that these policies seek to protect scarce maintainer attention, preserve accountability, sustain meaningful review interactions, address legal and quality concerns, and maintain pathways for newcomer learning. Based on these findings, we introduce the AI Contribution Governance Framework, which organizes recurring concerns and governance mechanisms across projects. The framework helps OSS communities develop AI contribution policies and provides researchers with a vocabulary for studying how AI is changing collaborative software production.

0.46

From Open Standards to Openly Governed: Standards-Setting Organizations as Stewards of Openness amid Platformization and Digital Sovereignty

cs.CY ·2026-09-01 ·Begoña G. Otero, Stefaan G. Verhulst
similarity 0.55
Abstract

Open geospatial standards let data, services, and systems work across platforms. But openness is not just a property of specifications. It also depends on the institutions that produce them and the infrastructures in which they operate. Standards may function as digital public goods and, once embedded in public systems, as digital public infrastructure. Those roles can diverge. This paper asks when open geospatial standards remain public infrastructure rather than becoming channels of enclosure. It makes two claims. First, standards-setting governance matters: institutional design shapes whether open specifications retain their public-good character. Second, geospatial coordination is shifting toward proprietary location stacks, APIs, and cloud platforms, where roadmaps, pricing, and service terms increasingly set the rules. Sovereign-cloud and sovereign-AI strategies relocate this power rather than remove it. These dynamics reinforce each other. Weak governance pushes coordination toward the implementation layer; platformization, in turn, reduces incentives to invest in open standardization, weakening portability and FAIR reuse. Standards bodies cannot reverse platformization, but they still control the specification layer. Where legal safeguards are limited, internal governance becomes key. We frame this through four dimensions: Purpose, Principles, People and Processes, and Policies and Practices (the 4Ps framework) while stressing the limit that internal governance cannot solve alone an external structural problem.

0.28

An Evaluation Framework for National AI Regulation

cs.CY cs.AI ·2026-08-15 ·Kaushik Sanjay Prabhakar, Tarun Adarsh R S, Amal Dhivyan Gregory, Sreeparvathy Sajeev, Utkarsh Tomar +1 more
similarity 0.54
Abstract

Governments use laws, institutions, funding programs and nonbinding guidance to shape how AI is developed and used. Comparing these national approaches is difficult. A binding rule and a detailed voluntary framework can address the same problem but create different duties. The resources needed to carry them out also differ by jurisdiction. This paper develops an evaluation framework for the documented design and implementation readiness of national AI policy. The comparison covers China, India, Japan, Singapore, South Korea, the United Kingdom and the United States. The European Union is included as a supranational comparator. The framework evaluates a versioned portfolio of official instruments rather than one prominent law or strategy. Its criteria ask whether the portfolio governs serious AI risks and whether responsible institutions can implement its commitments. They examine coverage across the AI lifecycle and the protections available to people affected by AI systems. Public benefit and responsible innovation remain a separate part of the assessment. Each sub-criterion is scored through ordered anchors and tied to the provision that supports the judgment. The protocol also records the source search, missing evidence, included instruments and cutoff date. The result is a traceable comparison of policy content that keeps category differences visible. It evaluates what a portfolio provides on paper. It does not estimate enforcement success or policy outcomes.

0.04

From Maturity Models to Ground Truth: Reconciling Cybersecurity Capacity Frameworks with Household-Level Governance Realities in the Global South

cs.HC ·2026-08-28 ·Wael Albayaydh, Ivan Flechais
similarity 0.54
Abstract · 3 tabvles

National cybersecurity and digital-governance capacity frameworks, most prominently the Cybersecurity Capacity Maturity Model for Nations (CMM), shape hundreds of millions of dollars in donor-funded governance investments across the Global South. Yet a persistent, under-theorised gap remains between state-level institutional maturity and the governance actually experienced by end-users. We term this the last-mile governance gap: the structural space between what a maturity assessment can see - laws, agencies, standards, awareness campaigns - and what a household living under that formal architecture can access, understand, or enforce. Drawing on a dual vantage point combining CMM assessment experience with peer-reviewed empirical fieldwork on smart-home privacy governance in Jordan, corroborated against studies from Kenya and China, we identify four mechanisms by which national capacity fails to reach the household: legibility, intra-household power distribution, accessibility of redress, and infrastructure-affordability constraints. We map these mechanisms explicitly onto the CMM's five dimensions, propose four concrete, low-cost last-mile indicators pilotable within existing CMM deployments, and outline a staged adoption roadmap with responses to anticipated objections. With AI-enabled devices entering homes across the Global South faster than institutional capacity can adapt, the stakes are rising: this dynamic risks converting a measurable maturity gap into an invisible one. We conclude with actionable implications for the GCSCC, the ITU, the World Bank, and other institutions relying on maturity scores to prioritize investment.

0.19

Open Science, Closed Models: How Funding Shapes AI in Science

cs.DL ·2026-09-21 ·Ana Trišović, Janakan Sivaloganathan
published similarity 0.54
Abstract · Accepted to AAAI/ACM Conference on AI, Ethics and Society (AIES) 2026

How funding shapes AI engagement in science is poorly understood despite its structural importance. We analyze 104,226 papers (2018-2025), linking funding acknowledgments to how each paper engages with foundation models: whether it extends a model (fine-tunes or builds on it), uses one without modification, or references models only peripherally. We report three findings. First, both private and public funding are associated with greater AI engagement. Public funding is associated with more off-the-shelf and open-weight use, while private funding is associated with greater model extension. Industry cloud credits are associated with less open-weight use, consistent with greater reliance on closed models. Second, industry involvement is associated with model extension, suggesting that corporate resources reach science through coauthorship, though this channel is concentrated in Computer Science and Global North collaborations. Third, the gap across research systems is not one of AI adoption but of control. Off-the-shelf use is rising across fields and country groups, and the least publicly funded group, the Global South, now has the highest share of API-only engagement (0.4% to 10.0% of engaged papers). China shows the opposite profile: the highest public-funding share and lowest API-only engagement, with extension remaining stable as it declines across the Global North. Standard capacity indicators show convergence in AI adoption even as dependence on proprietary infrastructure diverges.

0.23