CS Paper Radar

Two Fault Lines: Latent Polarity Geometry in X Community Notes

Andreas Andreou, Michael Sirivianos

cs.SI ·submitted 2026-09-18 08:48 UTC · v1
position

Abstract

Community Notes is X's crowdsourced fact-checking system. A note is published beneath the post it corrects only when raters who usually disagree both rate it helpful, a design called bridging. To apply that rule, the system learns who disagrees with whom from the ratings alone, placing every rater and note on one line, the polarity axis. Every scorer in the production pipeline uses a single axis. Refitting the base model these scorers share on the full public data (212.9M ratings, 2.33M notes, 1.07M raters), we find that one axis is too few. The space is at least two-dimensional. The first axis is left/right politics, while the second, which we interpret as trust in institutions, is largely independent of the first. A held-out test confirms that the second axis improves prediction of unseen ratings, while a third adds little. A second rater dimension learned from one set of topics predicts how raters judge COVID and Ukraine notes excluded from the fit, so it does not merely restate subject matter. Among heavily rated notes that barely divide raters politically, the published share falls from 71.5% to 11.7% as second-axis disagreement grows. A one-axis fit records these notes only as weakly polarised and less helpful; the information that raters at one end of the second axis support them is lost. Authors write notes matching their own position on both axes (r = 0.538 and 0.358), and a small minority of raters cast most ratings (Gini = 0.718). Fewer notes are published in the smallest language communities, but the shortfall is in ratings received, not in how the rule treats them. Keeping ratings per note constant, only Hindi stays below the global rate of 10.85%, and Greek moves from 7.76% to 11.68%. We argue for a bridging model with more than one axis of disagreement, and for recruiting raters in the languages the current design reaches least.

Closest papers

Nearest neighbours by meaning across the whole index.

Needs Your Help: Understanding Platform-Directed Rating Participation in Community Notes on X

cs.HC ·2026-09-16 ·Shuning Zhang, Changxi Wen, Jiuchang Wang, Dai Shi, Gabriele Lenzini +2 more
similarity 0.64
Abstract

Community-based fact-checking is promising in countering misinformation, yet its scalability is constrained by slow rating accumulation. To address this challenge, platforms such as X implement platform-directed rating mechanisms, specifically through ``Needs Your Help'' algorithmic prompts, to target unresolved notes. Using a dataset of over 220 million rating contributions -- including 1.9 million platform-directed ratings -- on X, we examine contributors' response to note prompts, notes' resolution, and raters' spillovers. We found (i) at note level, population-sampled ratings concentrate on recent notes with certain helpfulness and high disagreement. Once sampled, population-sampled rating was associated with faster and more transitions to resolved statuses. (ii) At rater level, following raters' first observed population-sampled rating, raters exhibit significant yet modest increases in daily ratings, rating pace and tag usage, while other behaviors show no change. These highlight the promise of algorithmic nudges to guide volunteer attention toward contested content, accelerating consensus while sustaining rater engagement.

0.27

Whose Polarization Is It? Auditing Fairness in Network Polarization Measures

cs.SI cs.MA ·2026-10-15 ·Stanisław Stępień, Michalina Janik, Akrati Saxena, Radosław Michalski
similarity 0.60
Abstract

A polarization score is supposed to describe a community, not its surroundings. Yet when a minority community is embedded in a larger network, its measured polarization can rise or fall with the size and structure of the majority around it. We ask whether polarization measures are therefore fair to minorities and develop an audit built around two criteria: measurement invariance (MI), which asks whether otherwise equivalent majorities and minorities receive equivalent scores, and structural exogeneity (SE), which asks whether a minority's score changes when its surrounding majority changes. Resolution sensitivity (RS) is treated as a prerequisite for meaningful fairness assessment. We audit seven polarization measure families spanning the three network-operationalizable dimensions of polarization, introducing Network-Weighted Distance (NWD) to fill an unserved extremeness-separation combination. Across controlled synthetic networks, mechanism-isolating diagnostic topologies, and real political and communication networks, no measure passes all applicable checks. Minorities are systematically scored as more polarized than equivalent majorities. We trace this dependence to two separate channels: majority structure can alter the opinions that opinion-reading measures consume, and, on a different path, opinion-orthogonal structural change can affect the arithmetic of structural measures. Reading opinions is thus double-edged, letting contamination in when structure moves opinions but shielding the score when structure changes and opinions do not. We close with practical guidance for reporting polarization scores without inheriting these biases.

0.28

Before You Poll with LLMs: A Deliberative Diagnostic Framework

cs.CL cs.AIcs.CY ·2026-09-14 ·Ahmed Wali, Hassaan Tayyab
top venuebenchmark similarity 0.58
Abstract · 17 pages, 2 figures. Accepted to EMNLP 2026 Main Conference. Note: Web abstract is abridged to meet arXiv character limits. See PDF for the …

Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This is critical for silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether personas hold the right opinions -- a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this. We introduce the Deliberative Polling Diagnostic Framework, which compares human and LLM belief shifts after identical informational interventions. Grounded in deliberative polling, it surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we find that every model fails, each in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this signature self-sycophancy: conformity to the model's internal stereotype of the persona rather than reasoning from the information provided. Our framework offers a concrete protocol: run the deliberative diagnostic before trusting LLM personas to mimic revised beliefs.

0.39

Why LLM Agents Favor Their Group: Stakes, Observed Norms, and Reputation

cs.AI physics.soc-ph ·2026-10-07 ·Yujiao Chen
similarity 0.55
Abstract

Language-model agents favor their own group because they have watched their members favor each other. The group label alone does little once the decision has a cost; what drives favoritism is observed behavior, and an individual's own record can override it. We test this in small societies with arbitrary group labels, ten rounds of point sharing, and matched one-shot decisions across fifteen OpenAI models and three Claude models, about 4,400 societies and 3.3 million audited model calls. First, the large effect of a bare group label reported in earlier work appears only when giving others points costs the agent nothing; once the agent can keep points for itself, that effect collapses on every model that shows it. Second, under a stake, interaction history becomes the main source of favoritism: the history effect is statistically positive on 13 of 15 models, reaches about 3.5-8 points out of 10 on 11, grows with the number of rounds played, and extends to labeled strangers the agent has never met. Third, with scripted histories, favoritism falls to near zero under an egalitarian norm and reverses when the agent's own group is seen favoring the other side; stronger models side with an individual's record when it conflicts with the group. Group favoritism is thus conformity to observed group behavior, carried to strangers by the label and overridden by individual reputation. The same account predicts responses to betrayal, scandal, and a free offer to change group: public reprimand repairs betrayal better than apology or restitution, allocation punishment remains confined to the offending member, and a formed group cannot be bought but can, on weaker models, be invited away.

0.10

When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

cs.CL cs.LG ·2026-09-10 ·DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang, JinYeong Bak
publishedcode/data similarity 0.55
Abstract · 15 pages, 4 figures. Accepted at W-NUT 2026. Code: https://github.com/dong4918-skku/Fable

Large language models are increasingly used as judges to measure social bias in text, yet the passages they judge are often noisy, containing typos, informal spelling, and broken punctuation. The consequences of such surface noise for social bias measurement remain unclear. To investigate this question, we apply five realistic noise conditions at multiple intensity levels to 3,822 stereotype-related responses and compare the resulting bias judgments with those on the original text. We find that such surface noise does not degrade bias measurement symmetrically: it is far more likely to turn neutral judgments into biased ones than biased judgments into neutral ones, by up to a 120x margin. We further observe two non-obvious effects across four LLM judges: in the most fragile judge the distortion is at its purest at mild, realistic noise levels, where erasure is scarcest, and as judges grow robust it attenuates toward parity rather than reversing. Bias measured on noisy text is therefore systematically overestimated, most in the categories that matter most for fairness.

0.47

Endorsement Without New Evidence: How Sequential Voting Inflates Mandates in Online Community Governance

cs.SI cs.CYcs.HC ·2026-09-08 ·Zihan Chen, Lei Nico Zheng, Di Zhu
position similarity 0.55
Abstract · 11 pages, 2 figures, 5 tables. Under review

Online communities often treat large support margins in public elections as strong mandates. We argue that such margins can overstate the independent scrutiny behind a decision. Using 198,275 free-text rationales from Wikipedia admin elections, we introduce vote-text divergence, a measure that flags a decisive vote paired with a thin, deferential rationale. Divergence rises as voters arrive later, even after controlling for voter and election fixed effects. The pattern is consistent with information saturation: once prior text is accounted for, arrival order no longer predicts divergence, while accumulated prior evidence does. The effect is strongest among peripheral voters in the co-voting network. Yet divergence does not predict worse post-promotion outcomes, such as administrative activity or survival. Public tallies can therefore weaken the scrutiny signal even while selecting capable administrators: a margin may appear to reflect more consensus and support than it actually contains.

0.33

Accounting for Bias Enables Sustainable LLM Evaluation

cs.AI cs.LGstat.APstat.ME ·2026-09-25 ·Harshita Katoch, David Antony Selby, Gerrit Großmann, Sebastian Vollmer
published similarity 0.54
Abstract · 8 pages, 2 figures; SuRE'26: Workshop on Sustainability and Resource-Efficiency of Artificial Intelligence at IJCAI-ECAI 2026

LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

0.55

From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals

cs.AI cs.HC ·2026-11-03 ·Sachit Mahajan, Javier Argota S{á}nchez-Vaquerizo, Arthur Capozzi, Dirk Helbing
similarity 0.54
Abstract

Large language models (LLMs) are increasingly being used to turn citizens' opinions into collective proposals. These proposals can make assignments: who should act, what must be done, how much one should spend and by when. However, if the assignments change when the same opinions are processed again, publishing a single proposal can hide alternatives that participants never get to see. We examined this issue in four LLM mediators (three open-weight and one commercial) using opinions from DeliberationBank and Polis. For each task, a mediator drafted a proposal from thirty opinions five times, varying either the random seed or the order of the opinions. In 17% of five-run sets from the open-weight mediators and 20% from the commercial one, some runs contained assignments while others did not. Reordering the opinions changed the assignments almost as often as replacing them with opinions from different citizens. Three human annotators confirmed that the detected differences between runs were real, although the LLM judge overstated how often they occurred. Substantial variation remained under four safeguards: temperature zero, a commercial mediator, draft selection, and draft merging. The mediator's objective also shaped representation: prompts seeking agreement covered minority positions much less often than prompts asking to describe differences. A single proposal can therefore obscure choices in both the assignments it proposes and the views it excludes. We propose audits that make alternative assignments visible and disclose the objective behind each proposal.

0.19