CS Paper Radar

State-Aware Generation of Deployable Websites from Visual Prototypes and Requirements

Ruwei Pan, Junlei Shen, Linhao Wu, Zixiong Yang, Qisheng Zhang, Yifan Li, Tianyu Ma, Yakun Zhang, Lu Zhang, Hongyu Zhang

cs.SE ·submitted 2026-10-13 07:23 UTC · v1
hot direction

Abstract

Generating deployable websites from visual prototypes and textual requirements is increasingly feasible with multimodal models, yet keeping generated projects consistent with their prototypes remains challenging. Prototypes specify layouts and visual elements, while requirements describe interaction workflows and deployment constraints. Existing UI-to-code methods and coding agents often treat layout reconstruction, interaction implementation, and project editing separately, allowing visual elements, files, workflows, dependencies, and render evidence to lose their correspondence across generation, verification, and update. We call this problem prototype-to-project state drift. We propose VisStateCoder, a state-aware framework that maintains these correspondences in a Project State Graph. The graph connects four types of state nodes through typed links and supports prototype grounding, graph-guided generation, and render-evidence-guided scoped updates that map failed checks to affected files while preserving files outside the update scope. We evaluate VisStateCoder on Vision2Web using five foundation models across Static Webpage, Interactive Frontend, and Full-Stack Website tasks. Averaged across models, VisStateCoder improves overall scores over the strongest baseline for each model by 26.1 and 22.6 points on Interactive Frontend and Full-Stack Website tasks, respectively. It also reduces verified-state drift, measured by checks that pass in one update round but fail later, compared with alternative state representations. An annotation study of 50 tasks finds that the constructed graphs achieve 95.6% edge-level F1.

Closest papers

Nearest neighbours by meaning across the whole index.

RepoAtlas: Guiding Coding Agents via Evolving Multimodal Repository Views

cs.SE cs.AI ·2026-09-15 ·Yunxiang Zhang, Haiquan Wang, JiaWei Guo, Hanyang Xia, Yan Chen +3 more
similarity 0.66
Abstract

Large language model (LLM)-powered coding agents have made rapid progress in automating software engineering tasks, yet repository-level issue resolution remains challenging. Beyond generating a plausible patch, an agent must localize relevant code across interdependent files and maintain repository context that is both sufficient and focused. Code graphs expose non-local relations, but linear text interfaces obscure their topology; rendering the full repository graph yields visual representations that are too dense to perceive reliably, whereas a one-shot local view becomes stale as exploration proceeds. We present \textbf{RepoAtlas}, a training-free module that maintains evolving multimodal repository views through a \emph{select--project--refresh} loop over a repository code graph. RepoAtlas combines evidence from the issue with the agent's current exploration state to select a task-relevant region under a fixed budget, projects the selected structure into complementary visual and textual representations, and refreshes the view when changes in the exploration state render it outdated. We evaluate RepoAtlas on SWE-bench Verified, where it improves the resolve rate by 2.4 points while reducing input tokens and model calls by 5.8\% and 7.8\% on average, relative to the strongest multimodal graph baseline, with consistent gains across three models of different families and scales.

0.10

SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents

cs.SE cs.AI ·2026-08-12 ·Ruitao Wang, Yuwen Hao, Menglin Yang
similarity 0.66
Abstract · 31 pages, 9 figures

Web agents often struggle to generalize to unseen websites because they lack website-specific supervision. Recent exploration-based data synthesis methods reduce manual annotation, but they still face two key limitations: they often fail to cover the full functionality of a website, and without sufficient website prior knowledge, they tend to propose hallucinated tasks, which in turn limits the diversity and efficiency of downstream trajectory synthesis. We present \textbf{SynWeaver}, a website-prior task-trajectory co-synthesis framework designed to address these challenges. SynWeaver first performs structured website exploration and constructs a website map that covers a broad set of functionally distinct page states and executable interactions on the target website. It then derives page-level and transition-level supervision from this map to train a UI-aware model with website-specific priors, enabling more grounded task proposals. Finally, SynWeaver performs collaborative task-trajectory synthesis, jointly updating the task and execution trajectory when they become inconsistent, and then verifies and repairs the collected results to produce executable, semantically aligned supervision. Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms strong synthesis baselines and yields more effective supervision for both in-domain and out-of-domain generalization.

0.10

Toward Executable Multi-framework Front-end Code Generation with Self-Correction

cs.LG cs.AIcs.SE ·2026-11-01 ·Linxiao Li, Haoran Ma, Chenyue Wang, Jiaye Lin, Haochen Sui +1 more
top venuecode/data similarity 0.65
Abstract · Accepted at NeurIPS 2026. 30 pages

Generating codes from UI screenshots has recently benefited from MLLMs, yet existing methods centered on HTML targets and transfer poorly to multi-frameworks such as React, Vue, and Angular. However, unlike HTML, which does not entail compilation issues, generating executable code in multi-framework settings is more challenging due to framework-specific differences in syntax and failure modes, as well as the need to maintain global consistency across files. To address this problem, we study executable multi-framework front-end code generation and propose MESCoder, a tree-structured self-corrective generation framework. Our method first extracts a structural scaffold from the screenshot and then constructs sub-trees and assembly relations step by step, transforming large project generation into a sequence of localized decisions. On top of this scaffold, we introduce a unified Project Agent that continuously revises its own partial project under runtime sandbox feedback and dynamically chooses between local repair and cluster level repair. For training, we adopt a two-stage strategy that first initializes the policy offline to learn basic multi-framework generation and repair priors, and then refines it with online reinforcement learning to optimize long horizon executability and final page quality. The resulting framework models multi-framework executability as a conditioned project state transition process and improves the stability and quality of complex front end generation through self correction. Experiments show that our method improves executability, while narrowing the visual-fidelity gap to HTML-based generation. Code is available at https://github.com/leo-lilinxiao/MESCoder.

0.40

MaLiang-Harness: A Programmable Path to Image and Video Generation

cs.CV ·2026-09-28 ·Haoyu Zhao, Zihao Zhang, Xudong Wang, Jiaxi Gu, Zuxuan Wu +2 more
code/data similarity 0.65
Abstract · 22 pages, 11 figures

Executable programs offer explicit control over how images and videos are constructed, but generating runnable code is only the beginning of visual creation. A program can execute correctly while violating the requested composition, appearance, or motion. We define this discrepancy as the Program-to-Visual (P2V) gap and introduce MaLiang-Harness, a unified framework for organizing MLLM-driven visual generation into a persistent process of construction, inspection, and revision. Its central design is to make the evolving visual program, its construction history, and its verification share a common revision reference. We define the Persistent Executable Generation (PEG) state as preserving programs and task context. Traceable Generation Process (TGP) connects edits to rendered evidence, and Revision-aware Editing and Verification (REV) supports restoration and checks the current revision before completion. Together, these mechanisms coordinate planning, execution, and visual feedback across rendering backends. We evaluate 11 powerful closed-source MLLMs on MaLiang-IBench and four on MaLiang-VBench, measuring generation success, visual quality, and computational cost. GPT-6-Astra achieves 100% generation success on both benchmarks, with 96.0% of image tasks and 76.9% of video tasks meeting all quality thresholds. The comparison also reveals a mismatch between general capability scores and visual generation performance, with similarly scored models differing substantially in their ability to satisfy visual requirements. MaLiang-Harness provides a systematic basis for studying how MLLMs translate executable code into visual outcomes, exposing both the potential of programmable generation and the limitations of general benchmarks as predictors of this ability. The project is available at https://github.com/gulucaptain/MaLiang-Harness.

0.70

PLAYGROUND: Progressive Layout Generation with Render-Grounded Planning

cs.CV ·2026-10-13 ·Jaeho Lee, Junhwan Heo, Junghyun Park, WonJun Moon, Eunju Yang +3 more
similarity 0.65
Abstract

Generating editable graphic designs requires arranging visual and textual elements into a coherent composition while preserving their content and editability. Although recent MLLM-based approaches employ iterative generation and visual refinement to reflect the real-world design workflows, their high-level plans are predetermined prior to execution, limiting their adaptability as the design evolves. To address this limitation, we propose PlayGround, a progressive layout generation framework that dynamically adapts to the current rendered canvas. PlayGround establishes a progressive design loop comprising a Progressive Planner, a Layout Generator, and a Visual Verifier, guided by a Structure-Aware Retriever. Given an initial set of unordered elements, the Structure-Aware Retriever initially infers a coarse layout structure to retrieve structurally similar reference designs. Subsequently, at each turn, the Progressive Planner examines the current rendered canvas alongside the remaining elements, forms the next functional group, and plans its composition. The Layout Generator then predicts editable layout attributes, while the Visual Verifier evaluates rendered candidates and provides feedback when refinement is necessary. To train the Progressive Planner with step-wise supervision, we construct a two-stage data generation pipeline that extracts functional groups and turn-level planning trajectories from completed designs. Experiments on Crello and LICA demonstrate the effectiveness of PlayGround over existing methods. Our code and weights will be publicly released.

0.32

From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

cs.CV cs.LG ·2026-08-12 ·Houston H. Zhang, Tao Zhang, Li Gu, Linfeng Ye, Yuanhao Yu +3 more
published similarity 0.64
Abstract · ECCV2026 (MUCG Workshop)

Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.

0.49

Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development

cs.CV ·2026-09-02 ·Yilong Guo, Hanqi Chen, Zixiao Ye, Guanzhong Wang, Chen Yu +1 more
similarity 0.64
Abstract

Multimodal large language models have achieved remarkable progress in front-end web development, generating interactive webpages from multimodal references such as screenshots and interaction videos. However, existing work largely emphasizes visual metrics such as aesthetics and layout similarity, while overlooking the more critical validation of interactive functionality. We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback. RILA introduces an Action Interaction Verification (AIV) module that replays the reference interaction trajectory on the generated webpage to collect grounded execution-aware observations, and an Execution-aware Rendering Score (ERS) that jointly measures interaction correctness and visual fidelity to guide iterative optimization. We further build an execution-verified data synthesis pipeline that produces diverse, high-quality training data, offering gains complementary to inference-time optimization. On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models. Notably, with our training pipeline, RILA lifts the compact Qwen3.5-9B backbone from 40.40% to 57.52%, surpassing far larger one-shot generators, including the 1T-parameter Kimi-K2.6 (55.61%) and the proprietary GPT-5.5 (55.74%).

0.20

LiveWorkspace: Externalizing Visual Understanding as Persistent Shared State for Screenshot-to-Code Agents

cs.SE ·2026-11-09 ·Shijun Xiao, Zonghui Yang, Xinrong Chen, Hao Liu, Wensheng Zhang +1 more
similarity 0.64
Abstract · 17 pages, including references and appendices

Screenshot-to-Code requires coding agents to sustain visual understanding across code generation, execution, and revision. We introduce LiveWorkspace, a visual coding harness that externalizes this understanding as Shared Visual State (Shared VS), an implementation-independent representation of visual hierarchy, node attributes, and layout relations. LiveWorkspace constructs and retains \textit{Target} from screenshots and recovers \textit{Current} from browser execution in the same schema. Structured Diff and implementation bindings guide code modifications; user edits update \textit{Target} and reuse the alignment loop. Using GPT-5.2 for both methods, LiveWorkspace improves end-to-end Avg by 1.8 points on Design2Code and 2.0 on ScreenBench over a control with matched model-call counts and stagewise budget limits. Further experiments demonstrate reuse of constructed Shared VS across coders and improvements in iterative repair. LiveWorkspace also achieves 93.0\% edit success across 100 controlled editing cases. These findings support treating visual understanding as a persistent shared working state throughout code generation, execution, and modification.

0.20