CS Paper Radar

Imagine Before Acting: Prospective Skill Identification for LLM Agents

Shuo Liu, Yutong Yang, Haohao Xiao, Mouxing Yang, Xi Peng

cs.IR ·submitted 2026-08-28 13:19 UTC · v2

Abstract

Effective skill identification is crucial for LLM agents to reuse task-solving experience for complex task execution, which however remains challenging as agents must accurately identify useful skills from a large skill gallery and invoke them on demand. Existing studies have explored various strategies for skill retrieval and invocation, but largely overlook the natural discrepancy between task descriptions and skill documents. To be specific, our empirical study reveals that task descriptions mainly specify task objectives, whereas skill documents describe the required capabilities and procedural guidance for achieving the objectives. Such a so-called Task--Skill Misalignment (TSM) problem would cause desirable skills to be neglected during retrieval or remain unselected during invocation, thus hindering task execution. As a remedy, inspired by human prospective cognition, we propose SkillDreamer, a prospective skill identification framework to mitigate TSM. In brief, SkillDreamer first infers execution-required capabilities and imagines their realizations as pseudo skills, thereby translating task objectives into execution-oriented guidance for skill retrieval. As execution proceeds, SkillDreamer reevaluates the capabilities required by emerging subtasks to determine which skills should be invoked, thereby aligning skill invocation with evolving execution needs. Extensive experiments on three representative agent skill benchmarks not only verify the effectiveness of SkillDreamer in mitigating TSM, but also demonstrate its generalizability across diverse retrievers and LLM agents. The code will be released upon acceptance.

Comment
26 pages, including supplementary material. Substantially revised and extended version: expanded from skill retrieval to skill identification covering both retrieval and invocation, with additional experiments and analyses

Closest papers

Nearest neighbours by meaning across the whole index.

Think with Skills: Pseudocode Planning Before Execution for Advanced Reasoners

cs.AI ·2026-11-02 ·Boshen Zhou, Boxuan Yue, Xiang Li, Shanchao Yang
similarity 0.72
Abstract · 10 pages, 3 figures, 3 tables

Language model agents often struggle with complex tasks even when equipped with expert-written skills. Such skills encode declarative knowledge, whereas successful execution requires procedural knowledge tailored to the task at hand. Natural language skills may leave conditions and dependencies implicit, and tasks involving multiple skills further require agents to coordinate intermediate results and execution order. We introduce Think with Skills, a method that trains reasoning models to turn relevant skill descriptions into task-specific pseudocode plans before execution. The pseudocode explicitly represents operations, conditions, intermediate results, and dependencies, providing a structured bridge from skill descriptions to agent actions. We construct training data that pairs tasks and skills with pseudocode plans and execution trajectories, use supervised fine tuning to initialize this behavior, and further optimize end-to-end task performance with Group Relative Policy Optimization. On SkillsBench, Think with Skills raises task success from 35.0% to 62.0%. These results show that proceduralizing expert knowledge before execution substantially improves skill use.

0.10

SkillFocus: Evolving Agent Skills via Capability Decomposition

cs.AI ·2026-09-28 ·Ning Wang, Zhiren Gong, Bingdong Li, Peng Yang, Aimin Zhou
similarity 0.70
Abstract

Agent skill evolution seeks to improve reusable procedural guidance for large language model (LLM) agents through iterative revision. Existing methods base each revision mainly on execution trajectories or feedback, leaving recurring behavioral requirements across tasks implicit and tying revision to the behavior of the current skill. We introduce SkillFocus, which decomposes recurring task requirements into a capability space that remains fixed as the skill evolves, separating what tasks require from how the current skill behaves. SkillFocus maps current task outcomes to this space to identify the capability that leaves the most tasks unresolved, then uses that capability to determine what to revise and which evidence to use. Across four benchmarks spanning heterogeneous tasks, SkillFocus achieves the best held-out accuracy on all four, outperforming the strongest competing result by 5.7 points on average while using 24\% fewer evolution tokens on average than the closest iterative baseline. Controlled studies further show that capabilities derived from recurring task requirements outperform task-semantic and execution-derived alternatives, while randomizing task--capability assignments reduces final accuracy by up to 20.2 points. Matching evidence to the selected capability increases candidate gain by 4.4 points under prioritized revision.

0.17

Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

cs.AI ·2026-10-13 ·Jinlin Liu, Wei Chen, Xiaojin Zhang
code/data similarity 0.69
Abstract · 22 pages, including references and appendices. Code: https://github.com/jinlinll/lost-in-implicit-routing

Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.

0.35

SkillSandbox: Skill Verification via Dynamic Scenario Synthesis

cs.AI cs.CL ·2026-10-07 ·Serin Kim, Kwangwook Seo, Dokyung Song, Jinyoung Yeo, Dongha Lee
similarity 0.68
Abstract

Self-evolving agents distill task-solving experience into skills for future reuse, but these skills can encode incorrect procedures or non-transferable knowledge. It is therefore critical to verify each skill's reusability: whether its guidance remains useful beyond the experience from which it was distilled. Such verification requires observing how a skill affects execution in new tasks, yet existing tasks may not expose the situations where the target skill can actually be exercised. To construct such situations, we propose SkillSandbox, a framework that dynamically synthesizes a task and its environment for each skill that are skill-relevant yet novel. A Proposer specifies the conditions to preserve and the source-specific details to vary, a Builder constructs an executable scenario, and a Verifier compares executions with and without the skill. The Verifier assesses executability, utility, and efficiency to assign a Keep or Reject verdict, determining whether the skill enters the library. Across ALFWorld and WebShop with three models, SkillSandbox consistently yields the strongest downstream performance and improved execution efficiency. Further analyses examine whether these gains reflect accurate assessment of skill reusability and identify which components of SkillSandbox contribute to them.

0.23

Skill2Env: Capability-Oriented Environment Synthesis from Skills for General Agents

cs.AI ·2026-09-27 ·Weiyi Xu, Xiaowen Yang, Wen Da, Hang Xu, Canwei Li +5 more
similarity 0.67
Abstract

Executable environments are critical for post-training agents on tasks that require tool use and multi-step interaction, but constructing executable tasks together with their environments remains difficult to scale. Skills provide reusable domain knowledge, operational procedures, and tool-use instructions, but a substantial gap remains between the information contained in a skill and a concrete, challenging task with a complete executable environment. To address this gap, we introduce Skill2Env, a capability-oriented framework that starts from a skill and uses agent capability demands to guide task and environment synthesis. Skill2Env represents these demands through reusable difficulty patterns and instantiates them into task blueprints that specify objectives, challenges, environment facts, information boundaries, and acceptance criteria. These blueprints guide the joint construction of task instructions, execution substrates, workspaces, and rubric-based evaluators around source skills. We further propose Iterative Task Hardening, which uses solver execution evidence to identify insufficiently challenging task designs, strengthen or extend their difficulty-pattern instantiations, and revise the corresponding blueprints and environments. Using 1.5K high-scoring trajectories generated from Skill2Env environments for supervised fine-tuning, we observe consistent improvements across a broad range of agent benchmarks, demonstrating the effectiveness of capability-oriented environment synthesis for agent post-training.

0.15

Demystifying Agent Skills: Why They Work-Until They Don't

cs.AI ·2026-08-14 ·Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao +4 more
highly cited similarity 0.67
Abstract

Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \emph{\textbf{When do skills help, why do they work, and where do they fail?}} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.

0.43

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

cs.AI cs.CL ·2026-08-20 ·Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, Jiawei Zhou
similarity 0.66
Abstract · 34 pages, 28 figures, 8 tables

Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent's performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.

0.44

When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents

cs.CL ·2026-08-23 ·Zihan Lin, Zhenyu Chen, Jiawen Wei, Xiaohan Wang, Jie Cao +4 more
top venuehighly cited similarity 0.66
Abstract · Accepted by EMNLP2026 Findings

Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents. Prevailing agent self-evolution paradigms typically rely on a core assumption: equipping LLMs with skill memories derived from successful trajectories will monotonically improve their problem-solving capabilities. However, probe analyses reveal that extracting skills solely from successful trajectories traps the model in a \textbf{Skill Imitation Trap}. For tasks that resemble past successes but require different tools, retrieving more skills paradoxically increases the model's confidence in wrong tool calls---procedure skills raise the wrong-tool margin by $47\%$ over a memory-free baseline. To overcome this limitation, we propose \textbf{Boundary-Aware Skill Memory} (BASM), which augments each skill with explicit boundary fields---applicability conditions, risk cues, avoidance rules, and recovery notes. These fields transform each retrieved skill from an unconditional action template into state-conditioned guidance: the agent applies the skill when its conditions hold, suppresses inapplicable tool calls when they do not, and issues targeted repairs when execution fails. Across three agent benchmarks and four model scales, BASM consistently outperforms success-distilled skill-memory baselines: it improves task success rate by up to $23.8\%$ on AppWorld, accuracy by up to $5.0\%$ on BFCL, and reduces attack success rate by $4.6\%$ on AgentDojo, while simultaneously reducing average AppWorld steps by up to $6.6\%$ relative to the memory-free baseline.

0.63