CS Paper Radar

Dense Feature Representation over Sequence Modeling: A Solution to the KDD Cup 2026 UniRec Challenge

Yi Zhang, Weiliang Ji

cs.IR ·submitted 2026-09-17 06:52 UTC · v1
publishedbenchmark

Abstract

We describe our 10th-place solution to the KDD Cup 2026 Tencent UniRec Challenge, industrial click-to-conversion (CVR) prediction over 34.82M records, and we ask which mechanisms actually move held-out AUC. Starting from the official PCVRHyFormer baseline, a 15-step single-variable chain raises test AUC from 0.813237 to 0.827816, and our final submission reaches 0.828535. A leave-one-out ablation from the full model attributes the gain: removing the dense-feature representation stack costs 0.0095 AUC and removing the orthogonalized optimizer costs 0.0028, while no sequence-modeling component (merged single-stream backbone, polarity channel, auxiliary head, per-token FFN) costs more than 0.0005, within or adjacent to a $\pm$0.0004 seed band. We also report a generalization hazard: the row-group train/validation split shares one time window, so validation AUC overstates the leaderboard by about 0.014; anti-memorization and high-cardinality-ID changes even invert sign against it, a divergence that traces to dump-to-dump distribution shift and survives a time-ordered re-split. Dense representation and optimization, not finer sequence modeling, drive CVR AUC at this scale, and verdicts must come from the held-out leaderboard.

Comment
6 pages, 1 figure, 4 tables. KDD Cup 2026 Tencent UniRec Challenge Workshop

Closest papers

Nearest neighbours by meaning across the whole index.

QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge

cs.AI ·2026-09-15 ·Yuanzhe Zhou, Zhaoyang Zeng
similarity 0.61
Abstract

Post-click conversion rate (pCVR) prediction requires jointly modeling feature interactions and sequential user behaviors. The KDD Cup 2026 Tencent UniRec Challenge calls for a unified architecture addressing both. We observe that existing unified architectures often generate query tokens---the central information hub---with projection-based multi-layer perceptrons (MLPs), without explicit token-to-query attention for refining the query side. We propose QueryFormer, centered on a stackable unified field--sequence block that bridges non-sequential multi-field features and behavioral sequences, and provide a latency-aware scaling study over view width $H$, model width, depth, data, and compute. The block generates queries through cross-attention and packs sequence queries into shared-parameter attention. QueryFormer secured 1st place in the Industrial Track, achieving an official test area under the ROC curve (AUC) of 0.83254; a modest post-competition scale-up reached 0.832713. Within our grid, $H$-scaling improves validation AUC from 0.84540 to 0.84615 and beats HyFormer at comparable budgets. Ablation identifies query generation as the largest contributor. Packed shared-parameter cross-attention keeps H=8 inference latency to only 1.89x that of H=1, positioning the bridge as an efficient stackable unified block.

0.21

When Are Predictions Enough? An Evaluation Protocol for Frozen Expert Composition

cs.LG cs.AI ·2026-11-09 ·Alessandro Pereira, Keith J Ransom, Lewis Mitchell
publishedbenchmark similarity 0.60
Abstract · Accepted, NeurIPS 2026, E&D track, poster

Practitioners often combine independently trained models while keeping their weights frozen. These experts are commonly combined through their final predictions, although their internal features may retain useful information that those predictions omit. We ask when combining features improves on combining predictions, and how to make that comparison fair. We introduce a reusable evaluation protocol, apply it to 21 frozen deepfake detectors from DF40 (Yan et al., 2024), and audit how evaluation choices change the conclusions. The protocol uses strong prediction baselines, compares simple and larger feature combiners, varies architecture and training data separately, and reports results by held-out generator with train/test overlap controls. Under leave-one-generator-out evaluation, logistic regression on the features of six architecture-diverse experts improves macro AUROC by 0.081 over the best tested prediction combiner in each fold, with gains on seven of eight generators. The larger network shows no clear improvement over the linear model (approximately 3.5M versus 10K parameters), suggesting that access to richer information matters more than additional combiner capacity on this pool. Features may help when they retain information that the predictions omit and that transfers to new generators. Predictions may be enough when the strongest prediction baseline already performs well, while neither approach reliably compensates for poor expert transfer. Weakening prediction baselines on the overlap-affected standard split inflates the apparent feature advantage by up to 0.172 AUROC. Overlap-aware evaluation also changes the estimated gain, though the contrast does not isolate overlap. Results by generator and expert pool show where the advantage holds, weakens, or reverses. Without these controls, the same benchmark can support different conclusions.

0.30

Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery

cs.LG ·2026-09-11 ·Hendrik Droste, Christian Medeiros Adriano, Kathrin Korte, Holger Giese
similarity 0.59
Abstract · 4 pages, 2 figures

Foundation models are increasingly adapted through fine-tuning, model editing, and alignment procedures while retaining previously acquired capabilities. Understanding the internal computations that support these adaptations is therefore becoming increasingly important for continual model evolution. Sparse autoencoders (SAEs) provide interpretable feature dictionaries for residual-stream activations and sublayer outputs, but it remains unclear how state features and update features interact to produce downstream residual features. In this work, we focus on MLP updates as a first test case. We construct a transition atlas of triples $s_k + u_j \rightarrow t_\ell$, where a residual-state feature and an MLP-update feature jointly predict a target residual feature, and validate candidate triples by ablating the decoded update feature. In a 20M-token Pythia-160M $L_7 \rightarrow L_8$ run, we find 38,125 strong ablation-effect transitions, but 88.0% have both state-target and update-target decoder cosine similarity below 0.7. As a preliminary cross-model check, a run of 20M-token Gemma-3-4B $L_{21} \rightarrow L_{22}$ causally validates only the top 30,000 ranked candidate triples by ablating the decoded update feature, and 53.6% of strong-effect triples have both state-target and update-target decoder cosine similarity below 0.7. The Gemma result is directionally consistent with Pythia, but weaker, since update-target cosine recovers many of the strongest Gemma effects and the run is not a full-atlas causal validation. Ultimately, our results suggest that feature flow atlases can serve as diagnostics of representation-update mechanisms and thereby inform tools for steering model updates. Future work will validate more complex patterns across layers, models, and SAE families.

0.18

Sequence Composition in EHR Foundation Model Pretraining

cs.LG ·2026-10-12 ·Joshua Placidi, Inigo Stibbe, Alfred J. Balston, Marek Rei, A. Aldo Faisal
position similarity 0.58
Abstract · 26 pages, 11 figures, 10 tables

Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.

0.18

STAR: Structured Tokenization and Target-Aware Interest Representation for PCVR Prediction

cs.IR ·2026-08-13 ·Yimeng Xu, Ruihao Zhang, Yingqi Song, Ying Jiang, Lan Ma
top venuecode/data similarity 0.58
Abstract · Accepted to KDD Cup 2026. Code is available at: https://github.com/AIzealotwu/taac_26_academic_rank2_firstround_rank11_secondround

Post-click conversion rate (PCVR) prediction is a core ranking task in industrial recommender systems. Modern ranking models must jointly capture heterogeneous non-sequential features, multi-behavior user sequences, and target-item-aware user interests, while remaining robust to high-cardinality sparse features, missing values, and train-inference inconsistencies. In this paper, we present STAR (Structured Tokenization and Target-Aware Interest Representation), a practical framework for the KDD Cup 2026 Tencent UniRec Challenge. STAR combines structured feature tokenization with target-aware interest representation on top of a HyFormer-style multi-sequence backbone. It introduces high-cardinality signal recovery, explicit user-item interaction tokens, target-aware sequence decoding, and a weighted user-item contrastive auxiliary objective inspired by InfoNCE. We further align the training and inference pipelines by reconstructing feature remapping tables and structural hyperparameters from the saved training configuration. Experiments on the challenge dataset identify the components that most reliably improve ranking AUC, while LogLoss is reported as a calibration diagnostic. The main ablation study shows a large gain from temporal context, with smaller but useful contributions from contrastive alignment, target-aware interest encoding, and high-cardinality sequence feature recovery.

0.51

Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models

cs.LG ·2026-11-02 ·Ryan Swift, Konstantinos Psounis
benchmarkcode/data similarity 0.58
Abstract · 19 pages, 3 figures; Code available at https://github.com/ryanaswift7/follow_the_features

Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.

0.29

What DNA Foundation Models Learn Beyond Sequence Composition

cs.LG cs.AI ·2026-10-11 ·Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto Magi
top venue similarity 0.57
Abstract · Accepted to NeurIPS 2026 (Evaluations & Datasets Track). 9 pages of main text, 5 main figures, with references and appendices

Do genomic foundation models (FMs) provide predictive information beyond simple sequence composition? We evaluate frozen representations from five DNA FMs against normalized k-mer frequencies across 57 genomic classification datasets and two exploratory gene-expression regression tasks. Simple k-mer features remain competitive with most models, whereas NTv3 shows a consistent advantage; the ranking is robust to globally fixed and multi-scale k-mer baselines and alternative downstream probes. To test incremental predictive utility directly, we compare global 6-mer features with their concatenation with each FM embedding under a matched Random Forest protocol. NTv3 improves 46 of 57 datasets (mean $Δ$MCC = +0.072), while Caduceus provides smaller gains and the other models do not improve mean MCC. We separately decompose embeddings into a component linearly predictable from 6-mer frequencies and a ridge residual, finding that linear recoverability does not reliably indicate downstream utility. Exact 6-mer-preserving shuffles further reveal task-relevant sequence-order sensitivity, most pronounced for NTv3, although order sensitivity alone does not guarantee predictive improvements. Our results establish compositional features as essential controls and offer complementary diagnostics for evaluating the added value of frozen genomic representations, without making claims about fine-tuning capacity.

0.47

Pruning DNA Sequence

cs.LG q-bio.GN ·2026-11-07 ·Faias Satter, Shaker Islam, Aminul Islam
code/data similarity 0.57
Abstract · 23 pages, 7 figures, 18 tables

Data pruning can substantially reduce training cost by removing examples that contribute little additional learning signal, but it has not been studied for DNA language-model pretraining. We introduce Sequence Pruning (SP), a dynamic pruning framework that removes genomic sequences as they become easy for the model to predict. SP uses per-sequence pretraining accuracy computed directly from the model's forward pass, requiring no separate scoring model or additional pass over the corpus. We evaluate SP across five pretraining settings spanning two corpora, three model families, and both MLM and CLM objectives, with downstream evaluation on five genomic benchmarks. SP reduces pretraining data processing by up to 96% while largely preserving downstream performance. At moderate savings, pruning improves performance by up to 1.2 MCC points over full-data pretraining; at the highest savings, pretraining uses only 4% of the baseline GPU hours with little performance loss. SP also outperforms the dynamic pruning methods InfoBatch and BPAS across most comparisons. Beyond accuracy, our framework supports alternative criteria such as perplexity and EL2N, although accuracy yields the strongest downstream performance at matched savings. These results show that genomic pretraining does not require repeatedly processing all available sequences and that dynamic sequence-level pruning can substantially improve its data and computational efficiency. Code is available at https://github.com/FaiasPromit/pruning-dna-sequence.

0.38