CS Paper Radar

Actionable Insights from Observational Data: The Case of Advanced Classes in K-12 Education

Nabit Bajwa, Seth B. Hunter, Sanmay Das

cs.LG ·submitted 2026-09-20 19:36 UTC · v1
published

Abstract

A fundamentally challenging question in K-12 education is about the effects of taking more advanced or challenging classes. It is particularly complex because students (and/or their parents) choose whether to enroll in these classes, making causal analysis challenging. In this paper, we begin to tackle this question by taking advantage of a novel dataset from a public school system in the US. This dataset records students' course enrollment decisions, prior academic histories, demographics, and subsequent outcomes around the time of a district-wide change that introduced optional open-enrollment advanced middle-school courses in subject areas. This is a rich observational dataset, but enrollment in advanced classes is driven by student characteristics and choices rather than random assignment. This creates a core identification challenge: the same factors that influence enrollment in advanced courses are also predictive of academic outcomes. As a result, simple comparisons between enrolled and non-enrolled students are confounded, and naive estimates may reflect underlying differences in student ability, motivation, or support rather than the impact of coursework itself. Our analysis shows that enrolling in advanced English courses has a net positive but modest effect on student achievement outcomes. However, these benefits are unevenly distributed: some students with relatively large predicted gains ("middle achievers" in prior years) are less likely to enroll than others. Some other groups (e.g. Black students and those with lower socio-economic status) also demonstrate significantly lower propensity to enroll. This gap between predicted benefit and observed enrollment illustrates how careful data analysis can extract actionable insights from large observational datasets, including identifying students who appear well-positioned to benefit but do not select into advanced options.

Closest papers

Nearest neighbours by meaning across the whole index.

What fidelity metrics miss: a structural check on synthetic educational data

cs.CY ·2026-09-23 ·Hitoshi Inoue, Koichi Yasutake
similarity 0.55
Abstract · Accepted for presentation at 1st Workshop of Trustworthy Educational Data Sharing and Secondary Use: ReLEAF Data Challenge, The 34th Interna…

Secondary use of educational records is increasingly mediated by platforms that share a differentially private synthetic version of a dataset and validate specific findings against the real data on request. The synthetic version is evaluated by comparing summary statistics of each variable, yet reported confirmation rates suggest that such comparisons do not predict which findings survive. We propose a structural check: the number of connected components of a weekly proximity graph over learners, tracked across a term. Across four annual cohorts of lower-secondary study-habit logs, the synthetic versions reproduced the level of this quantity and the shape of the weekly partition, but its variation across the term was between 2.6 and 4.9 times smaller than in the real data at a common working point, without exception, and those changes fell in different weeks: the synthetic cohorts single out the term's examination weeks and the real cohorts do not. We also show that a routine rule for setting the graph threshold makes naive comparisons between two datasets invalid, and illustrate this with an error of our own. The real curves are also distinguishable from marginal-preserving surrogates of themselves in all four cohorts, where three of the four synthetic ones are not, a comparison that needs no real data; these differences trace to what the generator was given.

0.31

Causal Modelling of Support Interventions for Student Competency Assessment

cs.AI ·2026-08-25 ·Francesca Mangili, Alessandro Antonucci, Rafael Cabañas
similarity 0.55
Abstract

Accurate assessment of student competencies is essential for enabling educators to identify individual needs, design targeted interventions, and evaluate the effectiveness of educational strategies. Empirical assessment procedures are typically grounded in psychometric models, such as item response theory, which relate student competence levels to performance on assessment tasks. In this paper, we advocate adopting a structural causal modelling approach to educational assessment, moving beyond probabilistic belief updating toward a framework that explicitly supports interventional and counterfactual reasoning. We propose a corresponding protocol for its construction and analyse the practical relevance of forms of reasoning that remain inaccessible to standard associative models, including the explicit modelling of interventions such as hints and the related counterfactual scenario analysis. Although our protocol requires the structural equations to be elicited from experts, the necessary information is purely logical and does not rely on probabilistic, less tenable assumptions. We illustrate the approach using data from an assessment that employs complex tasks designed to measure compulsory school student algorithmic skills.

0.17

Scalable and Versatile Identification for Hierarchical Structural Causal Models: A New Look at Project STAR

cs.AI stat.ML ·2026-08-25 ·Janis Aiad, Aghiles Drali, Aymen El Ouadrhiri, Anass Ettahiri, Yasser Oufqir +4 more
benchmarkcode/data similarity 0.54
Abstract

The STAR (Student-Teacher Achievement Ratio) experiment (1985, Tennessee, USA) is a landmark hierarchical dataset designed to assess the impact of class size on student outcomes, with observations nested within classes. To encode class-level interventions in such hierarchical settings, we develop a complete, scalable, open-source pipeline for Hierarchical Structural Causal Models (HSCM) that bridges symbolic identification and practical estimation. Our approach integrates graph transformations, pyAgrum's do-calculus for automatic identification of causal effects, adaptation of symbolic expression into closed-form HSCM formulas, and numerical estimation from fitted local probability models. A key innovation is our adapted Abstract Syntax Tree (AST), which decomposes pyAgrum's identified formulas into independent density, expectation, and marginalization tasks, enabling parallel and scalable computation. We validate the pipeline on canonical HSCM motifs and benchmark scenarios with known ground truth, then apply it to STAR kindergarten mathematics outcomes. The results show that flat baselines (ignoring hierarchy) recover associations but fail to encode class-level interventions, and that symbolic identification alone is not enough for practical Hierarchical Structural Causal inference; scalable estimation and numerical stability checks are central parts of the scientific object.

0.38

Better Plans, Better Outcomes: Integrating Historical Data into Student Planning in a Computing Course

cs.CY cs.HC ·2026-11-03 ·Jiaen Yu, Jinya Jiang, Soohyun Nam Liao
published similarity 0.52
Abstract · 7 pages, 5 figures. Accepted for publication at the 58th ACM Technical Symposium on Computer Science Education (SIGCSE TS 2027)

Effective planning is critical for academic success, but in many computing courses students often lack accurate benchmarks for assessing the difficulty and time required for academic tasks. This phenomenon is more pronounced in large courses where instructors have limited capacity to provide individualized guidance on planning and task estimation. Although many planning tools exist, still, they rarely leverage task-specific, empirically grounded data from actual course contexts. Thus, we explored whether providing task-specific, data-driven information---assignment previews and cohort-level historical statistics---could support more effective planning. We present a lightweight and data-driven planning intervention embedded in the programming assignment workflow of an undergraduate computing course. We conducted quantitative and qualitative studies with 252 students across five programming assignments. Our results indicated that, in authentic course settings, the intervention led to more conservative and executable plans, with significantly higher plan adherence. Students with stronger adherence also achieved better performance on both programming assignments and exams. Qualitative findings also showed both practical planning support and mental benefits from intervention. These results highlight the potential of data-driven intervention to support student planning in larger courses settings where individual guidance from an instructor is further limited.

0.36

From Early Participation to Later Completion: Evidence from a Large-Scale Self-Paced Learning Programme

cs.HC ·2026-09-28 ·Sakshi Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade +3 more
similarity 0.51
Abstract · 12 Pages, 3 figures, 3 tables

Large-scale learning programmes generate records that make learner participation observable across different activities. Participation points are commonly used to record and encourage such participation, but their value may extend beyond the activities for which points are awarded. Existing evaluations often examine gamification outcomes within the activities or learning environments in which the game elements are implemented, providing limited evidence about whether early participation points contain information about later participation outside the points system. This study examines whether early participation points can provide information about learners' later participation in a self-paced learning track that does not award participation points. Using anonymised records from 876 learners in a large-scale remote software-upskilling internship, we examined participation points generated from live-session attendance and poll responses against later self-paced course completion. The primary analysis used the 438 learners who earned at least one point during the first week, while the full cohort was retained for the no-point analysis. Week-one participation points distinguished learners who later completed a self-paced course with an AUC of 0.89, increasing to 0.95 by the fourth week. Similar AUCs were observed at both stages of the self-paced course sequence, while the absence of week-one points identified learners who did not start or did not complete a self-paced course with 95% precision. These findings indicate that early participation points can provide information about later participation outside the activities that generate the points. Such information can help large-scale learning programmes identify learners who may require timely attention while learning is still in progress, without treating participation points as a measure of overall learner engagement.

0.40

Rules, Choice, and Chance: A Welfare Evaluation Framework for Student-School Matching

cs.CY ·2026-11-08 ·Reetahan Mukhopadhyay, Clara Strasser Ceballos, Augustin Plantureux, Christoph Kern, Julia Stoyanovich
code/data similarity 0.51
Abstract · 42 pages, 26 figures, with 14 page main section

Centralized student-school matching systems, adopted in the United States, Chile, the Netherlands, and the United Kingdom, assign students to schools based on their submitted preference rankings and schools' priority policies. Yet welfare in these systems is typically measured through a narrow lens: a few aggregate statistics published by education authorities, such as the share of students unmatched or assigned to a top-ranked choice. A single aggregate cannot reveal how outcomes vary across students, or why. Welfare depends on three interacting forces: the rules policymakers set (algorithm, tie-breaking, list length limits); the choices students make (which and how many schools to rank); and chance (lottery draws and students' residential location). A rule that reduces unmatched rates may simultaneously lower top-ranked assignment rates for specific groups, and evaluating such trade-offs in advance is challenging because individual-level preference data are rarely available. We present Trace (Trade-offs in Rules, Agency, Chance, and Equity), an open-source framework that synthesizes individual-level preference profiles from aggregate data using mixtures of Mallows models and evaluates alternative rules across multiple welfare metrics, disaggregating outcomes by choice and chance. We apply Trace to two real-world systems: NYC high school matching, where we study minimum list length requirements, and Chilean school assignment, where we compare the current multiple tie-breaking rule with a counterfactual single tie-breaking rule. Both reveal substantial, heterogeneous welfare trade-offs, underscoring the value of simulation-based evaluation before implementing school choice reforms.

0.27

EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics

cs.AI ·2026-09-11 ·Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, Tanja Käser
benchmark similarity 0.51
Abstract

Large language models (LLMs) are increasingly deployed as tutors, but it is unclear whether they support all students equally well. We introduce \textbf{EduFair-Bench}, a benchmark for auditing the pedagogical fairness of LLM tutors---whether tutoring quality varies systematically with student demographics. EduFair-Bench pairs a multi-domain question bank (mathematics, physics, chemistry) with a controlled simulation in which a fixed LLM student interacts with each tutor across nine demographic levels spanning four dimensions: gender, immigration background, first language, and socioeconomic status (SES). Tutoring quality is scored on five turn-level pedagogical metrics and four conversation-level dimensions, using an LLM judge validated against three-annotator consensus on 180 tutor turns. Bias is measured via paired Wilcoxon signed-rank tests and bootstrap effect-size confidence intervals. Two ablations (demographic cues conveyed through names; conflicting demographic information between tutor and student) disentangle tutor-driven from student-driven bias. Across five tutors, we find that model capability and demographic fairness are largely orthogonal: the smallest model is the most consistent while the four more capable tutors all exhibit wide demographic gaps with no clear capability-to-fairness ordering, pedagogy-specific RL training redistributes rather than removes bias, and language- and immigration-related cues produce larger gaps than gender- and SES-related cues.

0.22

OBER+: Continuity-Aware Reporting and Traceable Continuous Improvement in Outcome-Based Education

cs.LG cs.CLcs.CY ·2026-09-03 ·Elakkiya Rajasekar
unusual similarity 0.50
Abstract · 14 pages, 6 figures, 7 tables. Submitted to IEEE Transactions on Learning Technologies

Institutions practising outcome-based education compute learning outcome attainment routinely, while reviews of curriculum analytics report an absence of evidence on how that computation informs decisions. This paper presents OBER+, an extension of a deployed institutional attainment platform that computes the step from a measured shortfall to an evaluated corrective action. Five connected stages accumulate attainment across deliveries of a course, signal a shortfall and a persistent shortfall, grade it on cutoffs the regulator already uses, record the decision against a catalogue of practices annotated with their evidence, log the change, and quantify the subsequent movement in the shortfall. A further rule compares successive statements of an outcome, so attainment is never read as a series across a point at which the outcome changed. Applying the rules to the live record of two real courses produced three results. Every outcome of a core course was substantively redefined between consecutive deliveries, with subject matter moving between outcome numbers, so a naive reading would have reported a twenty-five point collapse between quantities that do not refer to the same learning. Recomputing the platform's figures from its documented rule showed six of ten differing by more than rounding explains, in a pattern that identified a defect since reported to the institution. Across fifteen statement pairs from three transitions, five were identical character for character, and among the ten that were not, the outcome carrying a given number was nearest to a differently numbered earlier outcome in six, a result resting on an ordering of similarities and requiring no threshold and no labelling. The contribution is a computational design for outcome-based reporting, stated as rules any attainment platform can implement, with evidence of what they make visible in a live institutional record.

0.31