CS Paper Radar

RIDGE: State-Conditioned Reward Blending for Behavioral Coverage in Deep RL Game Agents

AL Muqshith Mohammed Shifan, Kevin Christopher Chua, Ali Neshati, Loutfouz Zaman, Cristiano Politowski

cs.LG cs.SE ·submitted 2026-11-03 20:41 UTC · v1

Abstract

A deep RL agent trained on a fixed persona reward plays one way: an Explorer keeps exploring while starving, a Survivor hoards on an empty map. Since each persona fixes its own reward, automated playtesting trains N agents for N styles. Changing reward composition mid-episode runs into the switching-stability dilemma: PPO degrades under non-stationary rewards. We present RIDGE, a single-agent architecture that sidesteps the dilemma with two design choices: a state-conditioned blender that computes softmax-normalized sigmoid weights from slow-moving state variables, and a multi-head critic that trains one value head per persona on its own stationary signal. On Crafter with four personas, one \ridge agent acquires both branches of the wood-tier crafting/combat split that no fixed-persona baseline solves jointly, and a sharpness ablation confirms state-conditioning drives the gain. We position these results as demonstrations of the blending mechanism's dynamics across n=3 seeds per condition; larger multi-seed sweeps remain future work.

Comment
Conference of Games, 4 pages, 3 figures, reinforcement learning, automated playtesting, procedural personas,reward shaping, PPO, Crafter

Closest papers

Nearest neighbours by meaning across the whole index.

Stack more levels: how to get general and human-like Mario playing

cs.AI ·2026-11-12 ·Shinichi Arimura, Yuchen Li, Julian Togelius
published similarity 0.63
Abstract

Super Mario Bros. is widely used in game AI and procedural content generation (PCG), yet it remains unclear whether a model-free agent can reliably win unseen levels. To address this question, we train Proximal Policy Optimization (PPO) agents on increasingly large sets of PCG-generated Mario levels and evaluate their generalization on 1000 held-out test levels. We then shape competent PPO policies into three gameplay archetypes---runner, killer, and collector. Finally, we fine-tune these PPO policies using Diffusion-Reward Adversarial Imitation Learning (DRAIL) with human gameplay demonstrations. We find that increasing the number of distinct PCG-generated training levels generally improves unseen-level performance, and that fine-tuning PPO policies with DRAIL increases action-level human-likeness while preserving much of the persona induced by PPO pretraining.

0.19

Anchor and Perturb: Lazy Agent Remediation by Exploration Injection

cs.MA ·2026-09-23 ·Chengxi Zhong, Yongzhe Chang
similarity 0.62
Abstract · 12 pages, 4 figures; v2 includes multi-cycle automated benchmarks and Section 5.3 additions

Anchor and Perturb (AnP) is a lightweight framework that resolves multi-agent coordination failures by decoupling exploratory variance injection from recurrent manifold stability. Existing remediation strategies predominantly alter mixing network architectures or enforce simultaneous exploration across the collective, which inevitably precipitates severe temporal-difference penalties in non-monotonic reward spaces. Specifically, AnP isolates underperforming lazy agents and injects an asymmetric exploratory pulse into targeted coordinates whilst anchoring converged teammates to nominal greedy exploitation. Empirical telemetry benchmarks demonstrate that AnP successfully rescues collapsed joint policies (recovering from a 5% evaluation win rate nadir back to 85%) and facilitates escape from suboptimal coordination plateaus, sustaining peak win rates of 90% without requiring structural network modifications.

0.35

Reward Engine: Reward-Centric Self-Evolution for Open-Ended Tool Agents

cs.AI ·2026-10-16 ·Litong Feng, Ji Qi, Yiran Cheng, Shaowu Pan, Xinyu Zhang +1 more
benchmark similarity 0.62
Abstract · 29 pages

Open-ended tool agents can complete the same objective through different tool paths, and their quality depends jointly on the interaction trace, environment state, and final artifacts, so such tasks rarely admit an executable ground-truth reward. The central challenge is therefore to construct a reward from existing experience that exposes the current policy's failures while remaining stationary during optimization. We introduce Reward Engine, which, like a self-motivated student, first works out what success requires and then optimizes toward it. From the task and initial trajectories, it builds a RewardPack of testcases for required correctness and rubrics for graded quality, each executed as a check that returns a judgment with evidence. The frozen RewardPack then guides the agent to revise and select its outputs, and can also serve as a reward for reinforcement learning. On WorkBuddy Bench Office and Claw-Eval, the constructed reward largely agrees with the benchmark graders, not reversing their ordering on 82.3\% and 80.5\% of candidate pairs. Used to revise executions, it raises three-run consistency ($\mathrm{Pass}^{3}$) over Baseline from 45.0\% to 54.0\% and from 78.4\% to 83.7\%.

0.19

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

cs.LG cs.AI ·2026-09-10 ·Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li +2 more
benchmarkhighly cited similarity 0.61
Abstract · 37 pages, 18 figures

Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.

0.45

Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating

cs.LG cs.RO ·2026-10-11 ·Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang +2 more
code/data similarity 0.61
Abstract

VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the advanced exploration incentives based on the generated trajectory are introduced, enabling more efficient exploration toward judgment-worthy trajectories for more VLM reward triggers and thus gaining higher RL efficiency. Extensive experiments in MetaWorld demonstrate VLMeg achieves state-of-the-art from-scratch RL, outperforming advanced VLM-based baselines on 12 challenging reward-free tasks. Furthermore, VLMeg also enables effective RL without any ground-truth reward in 3 challenging real-world experiments. Code will be open after acceptance at https://github.com/Raina915/VLMeg.

0.32

Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach

cs.LG cs.AI ·2026-10-28 ·Ajay Vikram Periasami, Anjaneya Gupta, Ethan Cheng, Ivan Rosales
similarity 0.61
Abstract · Accepted at EXAG 2026 (AIIDE Workshop on Experimental AI in Games)

Balatro, a 2024 roguelike deckbuilder, presents three difficulties that have historically derailed reinforcement learning methods: a 3,000-action state-dependent action space, horizons of several hundred decisions with sparse rewards, and pervasive stochasticity in both card draws and shop generation. Published RL results on the game remain scarce. We introduce an end-to-end pipeline comprising (i) a high-throughput Rust simulator that is automatically state-diffed against the live game client via BalatroBot, (ii) a condensed observation-action interface with dynamic per-step masking of legal moves, and (iii) a multi-agent PPO architecture that decomposes play into a tactical blind policy and a strategic shop policy that share the full game state but act in disjoint phases. Long-horizon credit assignment is partially addressed by per-agent reward routing with explicit cross-agent credit. On the main joint run, training metrics show a sharp phase transition in which the first-blind clear rate converges above 99% and mean episode return increases by roughly an order of magnitude, plateauing at a mean maximum ante of 3 out of 8; single-agent baselines under identical rewards oscillated between greedy and hoarding failure modes and did not stably clear the first blind. These results establish the two-policy decomposition as a strong reactive baseline for Balatro and localise the remaining gap to long-horizon strategic planning rather than tactical blind play.

0.19

Dynamic Shaping: Acquiring Interfaces and Constructing Execution over Frozen Pretrained Components

cs.AI ·2026-11-16 ·Nathaniel Oh
code/data similarity 0.61
Abstract · 12 pages, 2 figures. Code and complete evidence: https://github.com/calysteon/shaping-gap-benchmark

We introduce Dynamic Shaping, a method for constructing executable neural arrangements through acquired response and partial-derivative interfaces over frozen pretrained components. The method learns local descriptions of available computation, ranks component rebindings using task-gradient alignment, transports the current interface across a rebinding, and retains changes through behavioral feedback. Continuous interface acquisition uses proximal policy optimization; the source coefficients remain fixed. We derive an exact identity relating transport error in component responses and partials to changes in action logits and their sensitivities. A pixels-only ViZDoom Basic study evaluates first task acquisition without a competent reference controller, demonstrations or intact response targets. Across three initializations and 100 shared held-out game seeds, initial arrangements succeed on 86/300 controller/episode pairs; joint interface-and-binding construction succeeds on 295/300. Fixed-binding interface acquisition also reaches 295/300. Each policy executes four source fragments, with 110 acquired interface values and fixed random visual and action projections. The artifact includes all candidate trials, interventions and costs; all 1,200 recorded initial/retained outcomes replay exactly. The contribution is an explicit profile, rebind, transport and retain procedure and a reproducible reference-free acquisition case over frozen pretrained computation.

0.54

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

cs.CL cs.AIcs.LGcs.MA ·2026-09-14 ·Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer +2 more
similarity 0.60
Abstract

A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.

0.42