CS Paper Radar

SynMOE: A Syntax-Aware MOE-Guided Approach for PEFT-Based Code Intelligence Learning

Shuo Liu, Hao Ning, Jacky Keung, Zhen Yang, Xiao Yu, Xing Hu, Yicheng Sun

cs.SE ·submitted 2026-11-04 07:15 UTC · v1

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has emerged as an important approach for adapting Language Models (LMs) in downstream tasks. However, in the domain of code intelligence, the performance of most PEFT methods relatively drops compared to Full fine-Tuning (FT), hindering their broader applicability. One of the main reasons is that current PEFT methods are generally based on assumptions about the low-rank nature or sparsity of parameter updates, lacking explicit modeling of code-related information. To tackle this limitation, we propose SynMOE, a syntax-aware MOE-guided approach for PEFT on code intelligence learning. Specifically, SynMOE contains two stages: syntactic information extraction and syntax-aware MOE-guided adaptation. It first obtains the syntactic labels of source code from Abstract Syntax Trees (ASTs), and then aligns the labels with the input tokens of code LMs. In the second stage, syntactic labels are integrated into the MOE module, participating in the routing decision of MOE for a more powerful syntactic-aware token representation. We evaluate the effectiveness of the proposed approach by applying it to two typical PEFT methods, LoRA and $(IA)^3$, across three diverse code intelligence tasks. Experimental results show that SynMOE achieves consistent and significant improvements. Even compared to FT, SynMOE achieves comparable or superior performance while being more computationally efficient. In addition, we also conduct ablation studies, hyperparameter analysis, and evaluation in the low-resource scenario to provide a comprehensive investigation of the proposed approach.

Closest papers

Nearest neighbours by meaning across the whole index.

PassSum: Leveraging Paths of Abstract Syntax Trees and Self-Supervision for Code Summarization

cs.SE ·2026-10-11 ·Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang +1 more
publishedpositionhighly cited similarity 0.67
Abstract · Journal of Software Evolution and Process, 2024

Code summarization is to provide a high-level comment for a code snippet that typically describes the function and intent of the given code. Recent years have seen the successful application of data-driven code summarization. To improve the performance of the model, numerous approaches use Abstract Syntax Trees (ASTs) to represent the structural information of the code, which is considered by most researchers to be the main factor that distinguishes code from natural language. Then, such data-driven methods are trained on large-scale labeled datasets to obtain a model with strong generalization capabilities that can be applied to new examples. Nevertheless, we argue that state-of-the-art approaches suffer from two key weaknesses: (1) inefficient encoding of ASTs; (2) reliance on a large labeled corpus for model training. As a result, such drawbacks lead to (1) oversized model, slow training, information loss and instability; (2) inability to be applied to programming languages with only a small amount of labeled data. In light of these weaknesses, we propose PassSum, a code summarization approach that addresses the aforementioned weaknesses via (1) a novel input representation which contains an efficient AST encoding method; (2) introducing three pre-training objectives and pre-training our model with a large amount of (easy-to-obtain) unlabeled data under the guidance of self-supervised learning. Experimental results on code summarization for Java, Python and Ruby methods demonstrate the superiority of PassSum to state-of-the-art methods. Further experiments demonstrate that the input representation we use has both temporal and spatial advantages in addition to performance leadership. In addition, pre-training is also shown to make the model more generalizable with less labeled data, and also to speed up the convergence of the model during training.

0.59

Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

cs.LG cs.AIcs.CLcs.PL ·2026-10-01 ·Nickil Maveli, Antonio Vergari, Shay B. Cohen
benchmarkpositioncode/data similarity 0.62
Abstract · 30 pages (preprint)

Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as \textit{parametric code retrieval}: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at https://github.com/Nickil21/AlgoREval

0.32

Structure-augmented LLMs for High-Level Synthesis Pragma Optimization

cs.AR ·2026-09-29 ·Haocheng Xu, Ye Qiao, Phyo Pyae Moe Aung, Alok Mishra, Pavana Prakash +5 more
top venue similarity 0.61
Abstract

Pragma insertion drives the quality of high-level synthesis (HLS) designs. Choosing the right directives demands expert knowledge and reasoning about loop nesting, data dependences, and memory layout. While existing large language models (LLMs) show promise in code generation, they lack explicit program-structure awareness, limiting their ability to suggest effective pragmas. We present PRISM, a novel structure-augmented LLM that closes this gap by adding compiler-grade structural reasoning to a pretrained, frozen code LLM. It combines three hierarchical program representations, Abstract Syntax Tree (AST), Control-Flow Graph (CFG), and Data-Flow Graph (DFG), injecting them into a specific transformer layer while keeping original code tokens in a separate stream. The cross-attention gate at the injection point allows falling back to the pretrained representation when its structural signal is unhelpful. On zero-shot evaluation in HLS-Eval, PRISM synthesizes 3.5\times as many kernels as Llama3-8B (26.9\% vs. 7.7\%), and on the kernels where it does succeed, it produces designs that are 2.31\times faster (geomean) than GPT-5-mini's. In the agentic flow, the PRISM codegen outperforms other baselines when optimizing complex code and drives the average normalized improvement across the HLS-Eval suite to 26.4\%.

0.47

MAP4CS: A Multi-dimensional Data Pruning Framework for Efficient Code Retriever Fine-tuning

cs.SE cs.AI ·2026-10-08 ·Yuxuan Chen, Mingwei Liu, Guangsheng Ou, Zekai Zhang, Zike Li +2 more
similarity 0.60
Abstract

Retrieval-Augmented Generation (RAG) has become a cornerstone in software engineering for enhancing Large Language Models (LLMs) with domain-specific knowledge. However, adapting retrievers to evolving code repositories remains challenging due to the noise and redundancy inherent in massive code corpora. Standard fine-tuning on the full corpus is computationally expensive and often leads to sub-optimal performance due to negative transfer from low-quality samples. Conversely, simple random sampling fails to guarantee data representativeness. To address these challenges, we propose MAP4CS (Multi-dimensional Awareness Pruning for Code Search), an adaptive data pruning framework. MAP4CS identifies a small, high-quality core subset by integrating syntactic structure, semantic diversity, and distributional representation, followed by a rigorous rule-based filtering pipeline. Extensive experiments on two large-scale datasets demonstrate that MAP4CS consistently outperforms random sampling baselines using only 5% of the training data. Remarkably, it achieves performance comparable to, or even superior to, fine-tuning on the full dataset, validating the ''less is more'' hypothesis in data-centric AI. Furthermore, linguistic analysis reveals an adaptive optimization mechanism: MAP4CS automatically functions as a de-duplicator for redundant corpora and a denoiser for chaotic ones, constructing a training corpus that is both lexically diverse and information-dense.

0.13

Temporal Adaptive Fine-Tuning for Code Comprehension

cs.SE ·2026-10-14 ·Alex Wolf, Pooja Rani, Harald C. Gall
similarity 0.60
Abstract

Repository-level code comprehension question answering (QA) supports developer reasoning over functionality and usage distributed across codebases. However, repository-specialized large language models (LLMs) can become misaligned as code evolves. We study this challenge as temporal repository adaptation: acquiring new knowledge, suppressing obsolete, retaining still-valid, and preserving semantically neighboring knowledge. Existing retrieval, fine-tuning, and model-editing methods address only parts of this problem. We propose Temporal Adaptive Fine-Tuning (TAFT), a novel adaptation approach that treats repository transitions as supervision. TAFT derives temporally grounded QA data from repository transitions, pairs newly valid with obsolete knowledge through preference optimization, and replays transition-conditioned retention examples to preserve still-valid knowledge. We evaluate TAFT on an augmented temporal SWE-bench dataset comprising 255,295 QA entries across ten temporal slices, comparing deferred (Baseline), per-slice retrained (Baseline-Batched), and variants with knowledge anchors along four dimensions: updated knowledge, obsolete-answer suppression, retention, and neighbor stability. TAFT consistently outperforms both the deferred and batched baselines, with an average relative improvement of 18% in updated/suppressed knowledge accuracy, doubled retention accuracy, and roughly 1.5 times the neighbor accuracy. With directly supplied knowledge anchors, TAFT-KANT further improves mean updated/suppressed, retention, and neighbor accuracy over TAFT by 3.55%, 8.35%, and 13.00%, respectively. Overall, TAFT demonstrates its effectiveness across all evaluated areas, effectively enabling code comprehension QA alignment with repository evolution.

0.18

SpecCoder: Specification-Aware Code Generation with Curriculum Dual-Task Reinforcement Learning

cs.SE ·2026-09-05 ·Yixuan Li, Mingxuan Huang, Jiajing Wang, Weidong Yang, Xinyi Liu +2 more
similarity 0.60
Abstract · 47 pages, 12 figures, 8 tables

Large language models (LLMs) have made substantial progress in code generation but still struggle with challenging programming tasks that require understanding rich natural language requirements. These requirements often specify problem goals, input/output formats, constraints, examples, and edge cases. Overlooking even one may produce executable but functionally incorrect code. Existing training-free methods mainly rely on prompting or agent-based workflows, while training-based methods typically optimize final code outputs. However, existing approaches provide limited supervision for learning the intermediate mapping from raw requirements to structured specifications and for grounding them in concrete implementation behavior. Consequently, models may omit critical constraints, and even when an explicit specification is produced, the implementation may fail to reflect it consistently. Motivated by this gap, we propose SpecCoder, a specification-aware two-stage training framework for code generation. SpecCoder first employs specification-guided SFT to train LLMs to derive structured specification analyses and generate code conditioned on them. It then introduces curriculum dual-task GRPO, which jointly optimizes specification-guided generation and discrimination to encourage stronger correspondence between specifications and code behavior. Experiments on APPS, CodeContests, and xCodeEval demonstrate the effectiveness of specification-aware training, with SpecCoder consistently improving both standalone code generation and agent-based workflows. Additional evaluations on BigCodeBench-Hard and ClassEval, alongside human evaluation and perturbation studies, further validate the role of structured specifications in guiding code generation and discrimination.

0.41

CodeHID: Learning an Addressable Hierarchical Code Index for Generative Code Retrieval

cs.IR ·2026-08-25 ·Zhen Li, Yuhong Chen, Wenhao Xu, Xiaodong Li, Hui Li
similarity 0.60
Abstract · 10 pages, 4 figures

Code retrieval models have predominantly relied on a flat matching paradigm that treats code snippets as independent candidates, making them less capable of distinguishing similar code candidates. Generative retrieval offers a solution by constructing a learnable index over the code corpus, guiding the retriever to better understand how code candidates are semantically organized and addressed. However, naively applying generative retrieval in the code retrieval task may result in operating over an identifier space whose prefixes do not correspond to meaningful code-semantic regions. In this paper, we propose CodeHID, a generative code retrieval framework that reformulates the code retrieval task from flat candidate matching to coarse-to-fine semantic address generation. CodeHID relies on two core components. First, Pseudo-Neighbor Guided DocID Learning constructs a globally static hierarchical index by applying multi-level residual quantization and $k$-nearest-neighbor pseudo-supervision, ensuring that semantically related code snippets share prefixes while preserving target-level separability. Second, Dual-Phase DocID Generation Guidance reliably navigates this fixed index by combining training-side ranking enhancements, using hard negatives and rank distillation, with inference-side candidate constraints and prefix-aware decoding. Extensive experiments on CoSQA and ProCQA benchmarks demonstrate that CodeHID outperforms existing sparse retrieval, pre-trained code models, dense code retrieval, and generative retrieval baselines by a large margin in most cases, achieving particularly strong improvements in rank-one retrieval metrics.

0.24

Token Signatures of Code: Comparing Coding Behaviors Across Large Language Models

cs.CL cs.LGcs.SE ·2026-08-12 ·Junpeng Wang, Yuzhong Chen, Menghai Pan, Uday Singh Saini, Yiwei Cai
similarity 0.59
Abstract · 11 pages, 9 figures

The evaluation of large language models (LLMs) on coding tasks has primarily focused on performance metrics such as pass@k. As LLMs continue to advance, many models now meet baseline performance requirements, reducing the discriminative power of performance-based evaluation alone. Yet a key question remains largely unexplored: how do LLMs differ in their coding behavior? We propose CLIC (Code Learning for Identification and Comparison), a visual analytics approach that characterizes LLM coding behavior through token-frequency analysis. CLIC represents each code sample as a feature vector of token frequencies and trains an interpretable decision tree to separate two LLMs' code sets. Beyond classification accuracy, we define two new metrics: robustness, which measures whether the two LLMs remain distinguishable as their most-discriminative tokens are progressively removed, and concentration, which measures whether the difference is driven by a few dominant tokens or spread across many. Interpreting numerous pairwise comparisons (across LLM pairs, tasks, and tokenization levels) and tracing the full analytical chain form an inherently multi-scale, hypothesis-driven exploration task. We therefore develop an interactive visual analytics system to navigate the comparison landscape, identify pairs of interest, and drill down into discriminative tokens and their code contexts. Case studies comparing 10 LLMs across 22 Kaggle ML tasks reveal actionable insights for LLM selection and prompt engineering.

0.10