CS Paper Radar

From Sketch to Design: A Cross-scale Workflow for Procedural Generative Urban Design

Jin Gao, Sayjel Vijay Patel

cs.HC cs.GR ·submitted 2026-11-08 19:04 UTC · v1
published

Abstract

Urban designers use sketches and symbols to express spatial intentions, while many generative urban design tools rely on numeric parameters that are difficult to relate to this visual process. We propose a cross-scale workflow and a web-based prototype that translates symbolic sketches into controllable procedural urban generation. A vocabulary of shapes, informed by examples of urban design diagrams and sketching exercises with graduate urban design students, encodes attributes such as network structure, density, and program. The prototype uses the \$1 Unistroke Recognizer to map gestures to symbols and translates their locations and sizes into tensor-field attractors and weighted parameter grids. Grid and radial tensor fields guide street-network generation, while the parameter grids control building heights and programs. Implemented in JavaScript and TypeScript with React and Three.js, the prototype supports interactive sketching, visualization, and urban massing generation. We describe the methodology and illustrate prototype interactions, then discuss downstream integration with architectural-scale generative tools for further refinement and analysis. The work connects designers' symbolic input with an explicit procedural generation process.

Comment
7 pages, 6 figures. Author version of a paper published at CAADRIA 2024
Journal ref
Accelerated Design: Proceedings of the 29th CAADRIA Conference, Volume 1, pp. 343-352 (2024)
DOI
10.52842/conf.caadria.2024.1.343

Closest papers

Nearest neighbours by meaning across the whole index.

MaterialVellum: Steering Generative AI by Leveraging Expressive Material Affordances to Form Design Constraints

cs.HC ·2026-11-01 ·Chu Li, Nicolai Marquardt, Hugo Romat, Ken Hinckley, Nathalie Riche +2 more
top venue similarity 0.66
Abstract

We present MaterialVellum, an interaction approach for steering generative AI through material-inspired manipulation of design constraints. In urban design, for instance, zoning bounds can be defined by the stretch of elastic bands, building density redistributed by molding clay, and functional relationships organized by the pull of magnets. MaterialVellum departs from current interfaces that rely on natural language prompts, and instead leverages users' existing knowledge of physical materials to represent and manipulate design constraints. We introduce a framework that maps material affordances---such as elasticity, malleability, or attraction---to spatial, relational, and proportional constraints that underpin design. These mappings are operationalized in user interfaces through familiar direct manipulation multi-touch gestures, including stretching, sculpting, and clustering, whose effects feed as control parameters into generative AI pipelines. We demonstrate this approach through technology probes, including an urban design, a flower arrangement design, and a graphical design scenario.A qualitative study (N=10) shows that material-based interactions improve accessibility of constraint expression, support exploratory workflows, and lower barriers for non-experts, while effectiveness depends on alignment with users' mental models.

0.43

SurfScaff3D: Bridging Geometric and Semantic Representations for LLM-based 3D Object Generation

cs.CV ·2026-11-08 ·Anay Loya, Rishi Gupta, Arnav Samal, Anirban Chakraborty
publishedposition similarity 0.65
Abstract · Accepted to WACV 2027

Creating high-quality 3D content has traditionally required specialist expertise and experience with complex tools. The growing ecosystem of AI-assisted creativity demands accessible and intuitive 3D generation systems. Recent advances in Large Language Models (LLMs) offer promising directions, yet extending their generative capabilities to 3D remains fundamentally challenging because existing 3D representations are not naturally compatible with language-based reasoning. Current approaches either depend on expensive fine-tuning or impose restrictive structural priors that sacrifice expressiveness and exclude organic shapes. We argue that the mismatch between LLM-native reasoning and conventional 3D parameterizations is a core bottleneck of LLM-driven 3D generation. To address this, we introduce the Hierarchical Patch Scaffold (HPS), a representation designed around semantically meaningful, interpretable part-level abstractions that are naturally understood by both humans and LLMs, enabling expressive, editable, and open-domain 3D generation without imposing restrictive geometric priors or expensive fine-tuning. Building upon HPS, we introduce SurfScaff3D, an interactive framework for text and image-to-3D generation. HPS inherits the flexibility of B-spline patches, while exposing an interface that leverages open-world capabilities of foundation models to reliably generate 3D. Its hierarchical part-level semantic labels enable targeted, natural-language edits at any level of granularity without re-running generation. Further, HPS offers explicit per-patch control over surface detail, enabling more expressive geometry capable of representing arbitrary objects. Extensive experiments against state-of-the-art LLM-based 3D generation methods demonstrate strong geometric fidelity, semantic alignment, and flexible editing across diverse object categories.

0.28

ProxyBuild: Text-Guided Structured 3D Building Generation with Mesh-Anchored Procedural Proxies

cs.CV cs.GR ·2026-09-20 ·Xiang Tang, Ruotong Li, Xiaopeng Fan
dataset similarity 0.64
Abstract · 15 pages, 9 figures

Text-guided 3D building generation holds tremendous application potential, yet existing generative models typically output inseparable single meshes or non-interactive rendered representations. While procedural modeling can generate editable buildings with hierarchical structures, rule authoring is laborious, and even with the aid of large language models (LLMs), it remains challenging to effectively solve procedural rules under geometric constraints. In this paper, we propose ProxyBuild, a novel hybrid framework for structured building generation. We introduce the Mesh-Anchored Procedural Proxy (MAPP) as a novel intermediate representation, which tightly anchors building components onto geometric shells, thereby decoupling the generation task into two phases: proxy prediction and proxy-to-asset instantiation. First, we construct a building dataset with MAPP annotations to train our designed face-edge bigraph encoder. By explicitly modeling the feature interactions of topological elements on heterogeneous mesh graphs, this encoder accurately infers the semantic roles of faces and edges. Subsequently, conditioned on textual styles and attribute parameters parsed by LLMs, we accomplish high-precision asset retrieval and assembly by integrating a spatial placement logic with hard constraints. Extensive experiments show that ProxyBuild not only significantly mitigates common issues in building generation such as over-smoothing, component collisions, and structural corruptions, but also accurately parses semantic-free shells from diverse sources. Outperforming prior baselines across various metrics, our method can robustly generate structurally clear, detail-rich, and post-editable 3D buildings from text, thereby providing a reliable and interactive content foundation for downstream applications such as virtual reality and digital twins.

0.19

ReVision: Supporting Designers' Interpretation and Exploration of Visuals in Concepts and Forms

cs.HC ·2026-09-27 ·Yaqing Yang, Mei-Xi Chia, Aniket Kittur
top venue similarity 0.59
Abstract · UIST'26 Poster

Visual designers get inspiration from references to expand their design space. They decompose what makes a reference evocative into conceptual and visual elements, ranging from explicit attributes such as objects and colors, to less readily articulated concepts and visual motifs. They then create different visual forms to explore how the selected elements could be combined differently. Novices often struggle with these moves, instead focusing on surface features or producing limited visual variation, thus becoming fixated on the reference. Existing tools support editable visual attributes and high-level themes, but provide limited control over how conceptual interpretations relate to expressive visual motifs or how their combinations can be systematically re-expressed. We present ReVision, an AI-based tool that decomposes visual and textual references into editable conceptual interpretations and visual motifs, enables their recombination across conceptual and visual spaces, and renders each direction as divergent visual-form variations, supporting more divergent exploration during the creation process.

0.27

PolyGandr: Steering Generative Models via Spatial Navigation through Latent Space

cs.HC ·2026-10-27 ·Mingwei S. G. Li, Susie S. Y. Li, Isabel Moreira de Oliveira, Ryan P. Adams, Remco Chang
similarity 0.58
Abstract

Generative models support content creation across text, images, videos, and design disciplines, yet their interfaces rely primarily on natural language. While effective for specifying high-level intent, language can be inefficient and imprecise for fine-grained control, iterative refinement, and open-ended exploration. We present PolyGandr, a visual interface for exploring relationships between latent representations and generated outputs, complementing natural-language model steering. PolyGandr formulates generative control as high-dimensional navigation, enabling users to define starting points, steering directions, and step sizes through a unified map-and-mixing-palette interface. Combining global exploration with local refinement supports both broad design-space navigation and fine-grained output manipulation without repeatedly translating visual intent into prompts. We evaluate PolyGandr across three use cases: StyleGAN2 face generation, Stable Diffusion chair design, and rotational kirigami engineering. A formative study indicates that PolyGandr complements existing workflows by supporting real-time editing while simultaneously providing an overview and means to explore the broader design space.

0.07

ForagerSim: Evaluating Embodied Map-Based Reasoning in Large-Scale Dynamic Worlds

cs.RO cs.LG ·2026-10-19 ·Tomáš Musil, Martin Kotík, Michal Vavrečka, Karla Štěpánová, Sebastian Scherer +1 more
code/data similarity 0.58
Abstract · Code and data: https://github.com/MrTomzor/foragersim

Animals build cognitive maps of vast environments, use them to perform map-based navigation, and might even be reusing the same circuitry used in spatial reasoning for creating and using maps of arbitrary task-spaces. In contrast, current embodied AI systems either rely on hand-engineered mapping pipelines, which break in the presence of unmodeled dynamics and are restricted to metric space, or on learned policies that rarely form map-like representations at all. Developing architectures that could learn to build and use map-like representations as flexibly as animals is, however, difficult because few embodied AI benchmarks actually require map-based reasoning. To address this, we present ForagerSim --- an interdisciplinary research platform and simulation environment focused on embodied tasks that require map-based reasoning, with scenarios ranging from mobile-robot search and rescue to animal foraging in both complex mazes and vast, potentially infinite open areas. ForagerSim is highly modular and allows modifying independent axes of difficulty related to perception (abstract 2D sensing to full RGB or LiDAR), scale (room-scale to multi-km scale), topology (open-fields to complex mazes), dynamics (smoke, debris, sensorimotor failures and others) and tasks (from simple 2D coverage to foraging-inspired well-alternation tasks). We compare reinforcement learning baselines with a simple engineered robotics stack and a human player on 8 example worlds which show how different difficulty axes affect existing methods. Our results show that, when sensing is simplified to 2D, learned policies can perform multi-object search and well-alternation tasks surprisingly well in simple topologies, but they fall short of human performance in larger or more complex-topology environments where map-based baselines perform well, suggesting that map-based reasoning is the bottleneck in these scenarios.

0.33

ViSculpt: Visual-Centric Agentic Geometry Editing

cs.CV cs.GRcs.HC ·2026-08-25 ·Bo Pang, Jiaqi Pan, Xiaocheng Zhang, Jiacheng Xu, Guoping Wang +1 more
benchmark similarity 0.58
Abstract

3D geometry editing is a critical yet labor-intensive part of the graphics pipeline, requiring artists to translate creative intent into precise operations in complex professional software. Large language models (LLMs) have shown promise for script-based 3D creation, but script generation is less suited to perception-driven editing of arbitrary existing meshes, where execution must remain visually grounded and untouched regions should be preserved. We present a \emph{visual-centric}, training-free multi-agent system that edits existing 3D meshes directly in Blender by emulating the iterative workflow of human artists. Rather than generating scripts or regenerating geometry, our system operates through the Blender GUI: multimodal LLM agents observe the viewport, reason about the current mesh state, and execute localized edits through simulated user interactions. Experiments on a curated benchmark provide initial evidence that this agentic approach can follow natural language instructions, perform representative localized mesh edits, and preserve the overall identity of the input asset. Our results highlight a complementary regime for language-driven 3D editing: direct in-place modification of existing meshes within the native 3D editing workflow. We view this work as an exploratory step toward visual-centric agentic geometry editing in professional graphics software.

0.17

Visual-to-Executable Procedural Reconstruction of Buildings

cs.CV cs.AI ·2026-11-02 ·Zichong Zhu, Zhihao Liu, Andrei Sharf, Zhanglin Cheng
similarity 0.58
Abstract · 27 pages, 15 figures

Procedural modeling offers compact, editable, and reusable representations for architectural content, but building such models still depends heavily on manually authored grammars. We study image-based procedural building reconstruction from in-the-wild images, with the goal of recovering an executable procedural description that can be instantiated as an editable 3D building asset. Because a single image provides only partial and view-dependent evidence, we focus on recovering the dominant visible facade(s) and attach the recovered facade program to a simple mass model for building-level instantiation. Our method decomposes this task into two structured prediction stages. First, a Global Structure Layout Tree recovers facade-scale split-and-repeat organization through a fine-tuned layout model with render-guided visual-symbolic reflection. Second, a Component Parameterization schema maps representative facade tiles to structured procedural attributes for local architectural component synthesis. These two representations are then deterministically compiled into executable Computer Generated Architecture (CGA) programs, avoiding the brittleness of direct low-level program generation. Experiments show that the proposed framework handles both regular facades and challenging in-the-wild images, while maintaining high structural consistency, procedural executability, and fine-grained architectural detail. The recovered programs further enable controllable and reusable 3D building asset generation.

0.13