CS Paper Radar

Consequence-guided clarification: releasing minimum-compliance topology-optimized designs from incomplete specifications

Shaoliang Yang, Jun Wang

cs.CE ·submitted 2026-10-26 07:23 UTC · v1
code/data

Abstract

A converged topology optimization can solve the wrong problem when loads, supports, or the domain are underspecified. Assuming defaults, or asking the problem's author about every detail, ignores how much each answer changes the design. We propose consequence-guided clarification (CGC), which bases acceptance on computed mechanical consequences. CGC computes a reference design for each of a finite list of complete specifications, called readings. A design is accepted only if its compliance is at most 5% above the reference at every reading consistent with the answers. Otherwise, CGC asks the question whose worst answer leaves the smallest worst-case excess, and withholds acceptance at its question limit. To save optimizations, the screening variant CGC-S postpones some references until a candidate must be checked. We developed and evaluated both on 36 arXiv-derived tasks, treating each of 134 readings in turn as intended and simulating correct answers. In 26 tasks, the default-reading design failed another reading. Apart from always-asked questions on load direction, sign, and material budget, asking everything took 188 questions; CGC and CGC-S each took 105. The acceptance test supplied most of this saving: applying it to the default design alone took 117. Relative to asking everything, CGC-S requested 4.6 extra reference designs per avoided question, each from two optimization starts, counting optimization time only. References are not certified optima: better ones pushed three accepted cases above tolerance, and finer meshes changed some decisions. Within these limits, the test shows which readings a design covers and which details block acceptance.

Comment
24 pages, 9 figures, 7 tables. Supplementary Information (40 pages) and the two author adjudication records are ancillary files. Code and data: https://github.com/nbbllxx0/consequence-guided-clarification

Closest papers

Nearest neighbours by meaning across the whole index.

From topology-optimization brief to solver-checked release: separating intent, admissibility and acceptance knowledge for language agents

cs.CE ·2026-11-02 ·Shaoliang Yang, Jun Wang
code/data similarity 0.62
Abstract · 60 pages, 13 figures, 20 tables, including the supplementary appendix. Machine-readable supporting records are ancillary files. Code and dat…

An engineering solver can converge even when a language agent has misread the user's problem. AgenticTO makes this distinction inspectable by keeping three records separate: the proposed problem specification, deterministic formulation checks, and numerical acceptance of the solved design. We evaluated the workflow on 50 author-built topology-optimization briefs, injected faults, five numerical problems, and complete agent sessions. The tests used linear-elastic models with one load case. Geometry tools raised the primary provider's correct specifications and refusals from 37 to 44 of 50, while also increasing wrong specifications delivered as finished. Input-format feedback alone recovered most of the accuracy gain. The wrong specifications passed the formulation rules, which also missed most injected faults that changed the intended problem. Clarification corrected the remaining errors when an oracle supplied reference facts, but also prompted questions on fully specified briefs. Perfect accuracy under that oracle therefore does not establish performance with engineers. Numerical trials reached 73.7 million elements, although released designs used finer grids than requested. The records expose how a proposed problem becomes an accepted numerical result; agreement with the brief still needs a separate judgment.

0.52

Calculation and rejection diverge in language models tested at known mechanical limits

cs.CE cs.AIcs.CL ·2026-10-28 ·Shaoliang Yang, Jun Wang
code/data similarity 0.60
Abstract · 94 pages, including Supplementary Information. Data and code: https://github.com/nbbllxx0/Certified-feasibility-margins

When a language model returns engineering quantities with the status "solved", software that trusts it passes them on, even when the requested design is physically impossible. Such failures could reflect poor calculation, how the answer is requested, or a failure to judge feasibility. We set beam loads and disk speeds at known percentages above or below plastic collapse or burst, measured each model's limit calculation separately and tested 12 models in 4,860 calls, including follow-up experiments on new designs. We find that a model can recognize an impossible design when asked only whether it is feasible, yet accept the same design, reporting "solved", when asked for a full answer. Three models accepted beams 10 to 30% beyond collapse despite precise limit calculations in separate questions, and many accepted replies stated the violation. Asked only about feasibility, Opus 5 and Gemini 3.8 Flash rejected every impossible new design they were given, including those accepted in full answers, without false alarms. Rewording when to decline or changing the answer format raised rejection on the original beams, but not consistently on new designs. Software and benchmarks using a model's status should test feasibility judgment and full-answer rejection separately, and report the exact prompt.

0.58

Eliciting Specifications: When Does Interactive Clarification Help?

cs.SE cs.AIcs.CL ·2026-10-05 ·Mowafak Allaham, Aditya Vempaty, Ashish Jagmohan, Prasenjit Dey, Ravi Kokku
published similarity 0.58
Abstract · Accepted to NeurIPs 2026 Workshop on AI for Verifiable Coding

LLMs can generate Lean specifications from task descriptions, but those descriptions are often underspecified or ambiguous and may omit verifier-relevant edge cases, yielding imperfect formal specifications. We study whether an elicitation agent (EA) can improve verifier-checked Lean specifications by seeking clarification from a reference-grounded simulated developer before formalization. The agent decomposes the task, identifies unresolved decisions, asks bounded clarification questions, and uses the answers to write a refined natural-language specification that an LLM formalizer then translates into Lean. On 189 Verina Lean 4 specification-generation tasks, we compare single-shot generation with elicitation under question budgets of k=3 and k=10, allowing at most three or ten EA questions per task, across same-model and cross-model configurations. The same-model Claude Sonnet 4.6 pipeline has negative strict pass@1 estimates at both budgets, showing elicitation is not a uniform accuracy booster. In contrast, the Sonnet 4.6-to-GPT-5.6 and Sonnet 4.6-to-Gemini 3.8 Flash pipelines have positive strict estimates. The largest strict result reaches 65.6%, a +6.3-point change (+12 tasks), in the Sonnet 4.6-to-GPT-5.6 pipeline. Six of eight conditions have positive compile-conditional estimates, with a maximum cross-model difference of +4.9 points among compiling outputs. Treatment estimates also differ across the basic and advanced splits; the largest advanced-task change occurs in the Sonnet 4.6-to-GPT-5.6 configuration at k=10, from 42.0% to 50.6%. Failure analysis identifies postconditions as the largest source of degradation. These findings suggest future autoformalization systems should treat elicited natural-language specifications as intermediate artifacts and optimize jointly for semantic correctness among compiling outputs and reliable translation of clarified information into Lean.

0.30

DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle

cs.SE cs.AI ·2026-11-06 ·Rong Pan, Jianqiang Wang
similarity 0.58
Abstract

Can frontier general-purpose agents understand client intent, explore creative directions, execute revisions, and prepare production deliverables? We introduce DRAFT-Bench (Design Reliability Across Feedback and Tasks), a lifecycle-oriented evaluation of requirement elicitation, creative exploration, constrained editing, iterative revision, and production handoff. Its Requirement-to-Artifact framework links evolving requirements, protected properties, artifact versions, and delivery obligations. We compare native Codex/Astra and Claude Code across 35 design briefs under 43 conditions, alongside a separate 40-scenario RFQ track. Astra completes 43/43 design conditions and Claude 35/43. All scored core runs receive full structural credit, yet on the 28 core conditions scored for both systems, semantic checks flag violations in 11 for Astra and 5 for Claude; these are model judgments, not expert-confirmed failure rates. RFQ coverage is high, but success ordering changes across cohorts. SKU expansion, revision dependencies, and White ink files expose ambiguity among unmet obligations, missing inputs, and evaluator interpretations. These findings separate execution, requirement compliance, and professional acceptance. Unpaired tracks and different tools preclude claims of RFQ-to-delivery success, factory acceptance, or a pure model ranking.

0.27

RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfer

cs.CL ·2026-10-14 ·Yuqi Zhang, Di Zhang
similarity 0.58
Abstract · 21 pages, 2 figures. Includes a size-limited reproducibility subset as ancillary files

A reasoning prompt can improve one deployment and degrade another. RCCM (Reasoning-Control Capability Mismatch) examines these reversals by pairing answers to the same questions with records of how generation ends. Across five open-weight deployments and nine strategies, aggregate accuracy hides substantial question-level gains and harms; a common-budget MATH500 evaluation examines transfer on a separate sample. On 60 additional questions, concise reasoning improves Phi's automatic answer matching at a 2,048-token cap across three collections, but sharply reduces Qwen's. Author-reviewed final-answer commitments give a smaller, statistically unresolved Phi gain. Reserving an answer stage also improves matching, with additional input and call costs. These results show why a prompt benefit must be interpreted alongside its execution and evaluation: matching a reference answer need not mean delivering it, and an apparent compression benefit may admit another explanation. RCCM records this evidence for reproducible transfer comparisons.

0.33

Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models

cs.AI ·2026-10-31 ·Zhengshuai Yang, Changyun Li
similarity 0.57
Abstract · 10 pages, 1 figure, 4 tables

Automated prompt compilers search for or rewrite prompts under the implicit assumption that optimization consistently improves downstream task performance. We evaluate this assumption on frontier reasoning models through rigorous paired, same-day measurements. Across four optimizer families (genome search, profile-guided mutation, GEPA, and ESPO) and eight tasks, structured search yields no statistically significant accuracy gains on strong reasoners. However, a discrete structural genome provides essential architectural stability, preventing catastrophic bimodal seed-collapse (where unconstrained search collapses toward the judge floor in 3/5 runs). Through 190-problem evaluations on knowledge-intensive benchmarks, we establish a three-dimensional headroom typology: prompt optimization cannot compensate for intrinsic knowledge deficits or physical token truncation, but successfully resolves protocol and formatting errors. On gpt-6, while free-text reflective search (GEPA) yields a +.14 gain on a task where structured search remains flat, qualitative ablation reveals that ~.035 stems from judge-rubric reconstruction while ~.105 reflects genuine instruction refinement. Our pre-registered AutoAPC-Select deployment gate matches the oracle policy in 5/7 groups and rescues seed-collapse arms with zero regret. Finally, we expose silent backend-mixing in commercial APIs during outage windows and provide automated per-call audit verification tools to ensure empirical evaluation fidelity.

0.27

Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

cs.LG cs.AIcs.CL ·2026-10-13 ·Chinmoy Mitra, Tarek Mahmud
similarity 0.57
Abstract

Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records prompt changes throughout optimization. We evaluate RFDPO across five benchmarks covering law, knowledge, instruction following, mathematics, and privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% ($p=0.024$) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.

0.27

Formal-Verification-Guided Bayesian Optimization for Non-Intrusive Last-Mile Coverage Closure

cs.AR cs.LG ·2026-10-12 ·Chengjia Liu, Priyadarsi Mishra, David Kebo Houngninou, Michael Quinn, Aakash Tyagi +1 more
published similarity 0.57
Abstract · 21 pages, 3 figures, 7 tables. Accepted for publication in ACM Transactions on Design Automation of Electronic Systems (TODAES)

Last-mile functional coverage closure for modern chip designs remains costly because a mature UVM environment exposes many controls while direct hits on the remaining bins are sparse. We present a non-intrusive workflow that closes pre-defined bins in an existing, human-authored functional coverage model; it does not synthesize verification objectives from the design specification. The workflow first uses abstract-syntax-tree (AST) parsing and cone-of-influence (COI) tracing to map each residual target to existing testbench knobs. Under a matched 20-simulation budget, Bayesian optimization (BO) is compared with continued constrained-random verification (CRV) and random search over the same AST/COI-reduced space. Formal reachability then removes only targets supported by complete unreachability proofs, after which retained targets are grouped by formal difficulty and processed by group-specific BO under a second matched 20-simulation budget. Cleanup and grouping never add numerator bins; only simulator coverage artifacts do. On OpenTitan AES, first-pass reduced-space BO reaches 241/271 bins; formal cleanup removes two denominator bins; the integrated post-cleanup stage - difficulty grouping, policy routing, and group-specific BO - adds 20 simulator-confirmed hits, yielding 261/271 raw coverage (96.31%) and 261/269 effective coverage (97.03%). On Entropy Source, first-pass reduced-space BO reaches 2117/2277 bins; formal cleanup removes ten denominator bins; the integrated post-cleanup stage - difficulty grouping, policy routing, and group-specific BO - adds 93 simulator-confirmed hits, yielding 2210/2277 raw coverage (97.06%) and 2210/2267 effective coverage (97.49%). These results characterize the integrated diagnosis-and-search workflow; the present evaluation does not isolate the causal contributions of difficulty classification, policy routing, and post-cleanup BO.

0.36