CS Paper Radar

Dynamic Evidence Collection Ecosystem for Assessment Integrity and Authentic Competence

Rajan Kadel, Bellal Hossain, Samar Shailendra, Bushra Naeem

cs.HC cs.AI ·submitted 2026-08-17 02:19 UTC · v1
hot direction

Abstract

Generative Artificial Intelligence (GenAI) can produce high-quality essays, code, and design artefacts, challenging the validity of conventional assessments that rely on single-point submissions and product-only grading. This paper proposes a design framework called "Dynamic Evidence Collection Ecosystem" that shifts assessment toward continuous, authentic, multi-source evidence of student learning over time. The framework collects process evidence through iterative artefacts, design logs, activity rounds, self-reflection, and peer collaboration, supported by an AI-enabled layer for learning analytics, formative feedback, and transparency. The approach is grounded in recent assessment-redesign scholarship in AI-rich contexts and aligned with contemporary views of authenticity in assessment. This paper builds on the hypothesis that academic integrity is strengthened when it is treated as an assessment design rather than as an AI detection problem. The tools have limitations and risks of use that carry academic penalties. This paper presents an implementation scenario to support institutional adoption.

Comment
ISET 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Making the Invisible Visible: A Framework for Reflective AI Use in Software Engineering Education

cs.SE cs.CYcs.HC ·2026-09-28 ·Ali Shakiba, Thomas Chaffey
publishedhighly cited similarity 0.72
Abstract · 9 pages, 2 figures, Accepted for publication in the Proceedings of the 37th Annual Conference of the Australasian Association for Engineerin…

Generative AI (GenAI) is increasingly embedded in software engineering education, supporting activities such as requirements development, design exploration, documentation, and prototyping. However, educators often have visibility only into final artefacts, with limited insight into how students evaluate, verify, and refine AI-generated outputs during the learning process. This creates challenges for assessing evaluative judgement and responsible AI-assisted practice. This paper introduces the AI Journal, a structured reflection framework designed to make student-GenAI interaction visible in first-year software engineering education. The framework combines execution tracking, which records prompts, outputs, intent, and interaction context, with cognitive auditing, which captures verification strategies, intervention decisions, confidence judgements, critical learning moments, and reflections on AI-supported work. Deployed in a first-semester software engineering course, the AI Journal enabled visibility into aspects of student learning not observable through artefact-based assessments alone. Preliminary observations suggested variation in verification practices, intervention strategies, and perceptions of AI-supported work. Critical learning moments frequently occurred when students evaluated contextual suitability, feasibility, and requirements alignment rather than identifying obvious errors. The AI Journal demonstrates a practical, lightweight, and model-agnostic approach for making AI-assisted learning processes visible. By foregrounding verification, intervention, and reflection, it shifts attention from product-focused assessment toward evaluative judgement and responsible AI-assisted practice.

0.51

From Policing Provenance to Designing for Evidence: Testing Four Propositions on Generative AI Against Research and Institutional Policy

cs.CY ·2026-10-31 ·Samar Ansari
similarity 0.69
Abstract

Generative AI has made it hard to establish whether a student produced a piece of assessed work. Building on the argument that validity matters more than cheating, this paper tests four propositions drawn from a strongly stated position: that redesign is more sustainable than detecting or banning AI; that asking students whether and how much AI they used yields low-value information; that unaided ability should be tested directly in controlled conditions; and that the same logic applies to peer review. The propositions are tested against two purposively sampled, coded corpora: 41 research and sector sources and 55 university and sector policy documents. Coding was by a single coder and the policy sample is UK-weighted, so counts describe the sample, not the sector. The propositions hold unevenly. Redesign over detection is well supported, but only as redesign plus assurance: 4 of 55 documents prohibit AI by default, though about 10 of the 55 are publisher or editorial policies that govern authors rather than student assessment; of the 23 that take a position on detection, 16 reject detectors and one requires them. Testing unaided ability is supported as one lane within a programme, not as a return to invigilation. The disclosure proposition is complicated: institutions treat disclosure as valuable, yet the limited evidence suggests many students do not complete required declarations. The peer-review extension is plausible but under-examined. The field is converging on graded permission scales and two-lane designs, and 31 of 55 documents define named categories of use. From these findings the paper builds a nine-step decision chain that extends outcome-first design logic: the role of AI is decided after the evidence an outcome requires. It also separates design moves that add temporary friction from those that provide assurance. A worked example and the full policy matrix are included.

0.19

When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence

cs.SE cs.HC ·2026-10-26 ·Ali Shakiba
similarity 0.68
Abstract · 11 pages, 1 figure, 4 tables

Generative AI can produce code, tests, documentation, and technical explanations that resemble artefacts traditionally used as evidence of student competence, weakening the inference from artefact quality to the engineering capability behind it. We report the redesign and first deployment of a 13-week, AI-permissive, software-intensive engineering course in which students may delegate substantial work to AI while retaining responsibility for planning, verification, review, integration, and explanation. The redesign couples an engineering accountability workflow, which structures human-AI work and preserves human decision authority, with an assessment evidence architecture comprising six complementary channels: product, process/origin, verification, explanation, reflection, and individual assurance. A collect--cross-check--escalate (CCE) reasoning logic connects these channels rather than treating any single artefact or trace as sufficient evidence of competence. By the end of Week 11, the deployment involved 87 students organised into 17 teams, with 957 individual weekly Git repositories and 34 group-project Git repositories supporting repeated assessment and engineering activity. The first deployment yielded four practical lessons: AI assistance changes where rubric criteria carry evidential weight; multiple evidence channels require deliberate cross-checking and meaningful complementarity; the evidential strength realised in practice may differ from that intended in the assessment design; and automation scales evidence collection more readily than interpretation. The contribution is a conceptually grounded, deployed design for making engineering responsibility and competence more visible when AI can perform substantial artefact production.

0.20

The 5P Reflection Model for Education in the Generative Artificial Intelligence (GenAI) Era

cs.CY ·2026-09-03 ·Rajan Kadel, Samar Shailendra, Islam Mohammad Tahidul, Urvashi Rahul Saxena, Aakanksha Sharma +1 more
similarity 0.68
Abstract

Contributions: A reflection model suitable for the era of Generative Artificial Intelligence (GenAI) is introduced. The proposed model is an integrated model that extracts features from various existing models and also incorporates technological aspects of GenAI. Background: Universities worldwide are facing challenges in adopting GenAI into their curricula, as it has impacted academic integrity and the scholarship of teaching and research. Traditional reflection models are struggling to authenticate student reflection as GenAI is incorporated in education. This requires a GenAI-aware model to enable the opportunities that address the associated challenges with GenAI. Research Questions: Does the academic ecosystem require a GenAI-aware reflection model to adopt GenAI into education? How to make a reflection model structured to ensure student authenticity and cognitive engagement within a GenAI-aware learning environment? Methodology: This study employs a design-based research methodology, assisted by a critical inquiry approach, to analyse existing reflection models in the era of GenAI and examine the technological aspects of GenAI. Additionally, it identifies a gap and the absence of a comprehensive reflection model that enhances reflection in the era of GenAI and supports the effective integration of GenAI in education. Findings: The literature survey indicates there is a greater need for reflection in the era of GenAI to ensure learning, but the existing reflection models lack proper strategies to manage the problem introduced by GenAI. A standard GenAI-aware reflection model, called 5P (Purpose, Process, Product, Pitfalls and Plan), is proposed i) to manage greater demand for reflection, ii) to address the limitations of the existing reflection models, iii) to consider technological aspects of GenAI.

0.08

Evaluation in the Age of AI: Output as Evidence of Learning

cs.CY ·2026-08-23 ·Md Zarzees Uddin Shah Chowdhury, Samin Rahman Khan
position similarity 0.68
Abstract · 12 pages, 1 figure, 1 table

The rapid adoption of artificial intelligence (AI), particularly large language models (LLMs), has fundamentally disrupted how learning is demonstrated and evaluated in higher education. Tasks that once served as proxies for understanding-such as writing essays, solving problem sets, or producing computer code-can now be generated superficially by AI systems with minimal human effort. This paradigm shift raises a critical ethical question: how should learning be evaluated when traditional indicators of competence are easily outsourced? This paper examines the ethical challenges of educational evaluation in the age of AI from a university-level perspective. We argue that the core problem extends beyond academic dishonesty to a deeper misalignment between assessment practices and the learning outcomes they are intended to measure. Evaluation regimes that rely on artificial constraints risk measuring compliance, access, or concealment rather than genuine understanding, reasoning, or judgment. By analyzing institutional responses and presenting empirical survey data, we highlight the need for alternative assessment models that emphasize process over product. The goal is to establish ethically informed assessment strategies that preserve student agency and accountability in an automated age.

0.23

Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning

cs.SE cs.CY ·2026-09-24 ·Qusay H. Mahmoud
similarity 0.68
Abstract · 22 pages, 1 figure, 3 tables

Generative artificial intelligence has moved from a disruptive novelty to a recurring part of software-development and computing-education workflows, while software agents are beginning to act across repositories, command lines, browsers, tests, and other tools. The educational problem is no longer whether students should be allowed to generate code, but whether software-engineering (SE) programs can preserve and assess human understanding while preparing students to work responsibly with increasingly capable AI systems. This paper presents a structured integrative review of research and practice from 2023 through 23 September 2026, supplemented by established work on AI literacy, technical debt, and human-AI collaboration. The evidence supports a conditional conclusion: GenAI can improve access to explanations, feedback, practice, and short-term task completion, but learning outcomes depend on prior knowledge, scaffolding, verification, task design, and assessment. We examine student help-seeking and authorship, faculty assessment and policy demands, and the wider SE risks created when generated artifacts persist across repositories, teams, architectures, and deployed systems. We extend the concept of comprehension debt: the deferred learning and maintenance cost that arises when AI-assisted production outpaces a learner's or team's ability to explain, test, modify, and justify the resulting software. We then refine the AI-Augmented Software Engineering Education (AASEE) framework into five non-linear integration levels and four cross-cutting evidence obligations: explain, verify, modify, and account. The framework links delegation to evidence, governance, and recovery mechanisms appropriate to its consequences.

0.33

Using Evidence-Centered Design to Guide Prompt Engineering in Measurement: An Illustration with Automated Essay Scoring

cs.AI cs.HC ·2026-10-10 ·Jiawei Xiong, Cheng Tang, Hong Jiao, Hanna Choi
similarity 0.66
Abstract

Generative artificial intelligence (GenAI) models, particularly large language models (LLMs), are increasingly used in educational settings for tasks such as item generation, automated scoring, and feedback delivery. The quality of these outputs depends not only on the AI model but also on how users construct the prompts that guide model behaviors. Current prompt engineering practices are largely ad hoc, relying on trial and error rather than principled design. This lack of structure introduces validity threats including construct misrepresentation, construct-irrelevant variance, and limited reproducibility. We propose that Evidence-Centered Design (ECD), a framework in educational measurement for structuring valid inferences from observed performance, can serve as a principled foundation for prompt engineering in educational applications. We map the three core ECD models, the student model, the evidence model, and the task model, onto the components of educational prompt design, and describe a workflow for ECD-based prompt engineering. We derive five design principles from this mapping and present an empirical study contrasting ECD-designed prompts with state-of-the-art (SOTA) prompting baselines on an automated scoring task. Our empirical study demonstrates the usefulness of this framework on automated scoring. Results suggest that treating prompting as assessment design rather than informal instructions strengthens the validity, reliability, and fairness of AI-generated educational outputs.

0.12

ProofPaths: An AI Agent Framework for Assessing Student Reasoning

cs.AI ·2026-11-02 ·Mohsen Hariri, Vipin Chaudhary
similarity 0.65
Abstract · Accepted at the 8th Workshop on Education for High Performance Computing (EduHiPC 2026). 6 pages, 3 figures. https://proofpaths.github.io

Students can use large language models to complete assignments, but a correct solution does not establish whether they understand the reasoning behind it. Individualized interviews can probe that understanding but require recurring staff effort. We introduce ProofPaths, an AI agentic framework that combines rubric-based grading with individualized and interactive interviews. Multiple grading agents assess a submission against instructor guidance, exchange critiques, and revise their judgments. Their scores determine which review supplies the grade and feedback. An interview agent asks students to justify steps in their submission and uses their answers to guide follow-up questions. Each question is linked to a retrieved submission excerpt, so instructors can compare the student's explanation with the cited passage. The system tracks assessment progress and stores grading reviews and interview transcripts for each submission. Several deterministic tests of the assessment controllers check source handling, score aggregation, and limits on correction attempts. Instructors can inspect all reviews and transcripts and override grades within the assignment workflow.

0.27