CS Paper Radar

Razavi-Bench: Tracking AI Progress in Analog Circuit Reasoning

Zhishuai Zhang, Oscar Toomey, Taekwang Jang

cs.AI ·submitted 2026-11-16 11:45 UTC · v1
benchmarkcode/datapopular code

Abstract

We propose Razavi-Bench, a benchmark built from the 50 questions in Razavi's two Analog Design Experiments With AI articles and graded by GPT-6 Luna against reference answers and rubrics, and use it to track how fast AI models improve at analog circuit analysis. Across 73 multimodal large language models released between May 2024 and November 2026, the top score rose from 28.83% to 98.33% in 30 months, mostly before the reference answers were published. The leading models now answer almost every textbook question correctly; the hardest questions show a circuit that resembles a textbook one but works differently, or ask about behavior the obvious analysis misses. Within two and a half years, a set of questions that exposed the limits of AI models has become one they answer almost perfectly.

Comment
6 pages, 3 figures, 2 tables. Project page: https://razavi-bench.tokenzhang.com ; code and data: https://github.com/Arcadia-1/razavi-bench

Closest papers

Nearest neighbours by meaning across the whole index.

CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits

cs.AI cs.CV ·2026-08-10 ·Xinqi Yang, Kang An, Tengyue Wang, Zhongyu Yang, Chenxu Du +6 more
benchmarkcode/data similarity 0.65
Abstract

Electrical circuit analysis requires more than recognizing components in an image. A solver must ground symbols and labels, recover latent topology, select a physical model, formulate coupled equations, propagate intermediate quantities, and preserve units, signs, directions, and phase conventions. We introduce \benchmark, a benchmark of 1,000 authentic textbook problems for evaluating this complete long-horizon visual-to-symbolic reasoning process. Each problem pairs one or more circuit diagrams with a self-contained question, a typed or semantically specified answer, and a reference worked solution. An evidence-first construction pipeline aligns questions, figures, and solutions, while a reasoning-oriented taxonomy organizes problems by circuit type and dependency depth. Evaluation combines conservative typed scoring with identity-blinded multi-model semantic consensus, retaining every problem in the denominator. Across three commercial chatbot systems and six open-source multimodal large language models, the highest-scoring system reaches 84.8\% accuracy. However, performance consistently deteriorates on long-horizon problems, and qualitative analysis exposes persistent failures in topology-to-target binding, physical conventions, and late-stage output propagation. \benchmark{} provides a focused testbed for measuring whether multimodal models can transform technical visual evidence into sustained, physically valid symbolic reasoning. Code are available at GitHub - CircuitReason/CircuitReason1K.

0.20

AnalogPortBench: Benchmarking LLMs on Analog Circuit Porting

cs.AR cs.AI ·2026-10-21 ·Saeid Rajabi, Aly Moussa, Vishal Saxena, Chengmo Yang, Satwik Patnaik
benchmarkcode/data similarity 0.61
Abstract · 5 pages, 3 figures, 4 tables Code and data: https://github.com/saeidrajabii/AnalogPortBench

Analog circuit design still relies heavily on expert knowledge, and porting proven designs to new process nodes is a recurring bottleneck. As a lower supply reduces available voltage headroom, a working topology may require redesign or fail to meet specifications. While large language models (LLMs) have begun to generate and optimize analog circuits, existing benchmarks focus on one process node and assume feasible specifications. To bridge this gap, we introduce AnalogPortBench, a benchmark of 54 tasks for porting ten circuits, from current mirrors to telescopic amplifiers, from a 0.18 $μ$m process to the open-source SKY130 PDK. Tasks range from a direct port through a reduced supply and harder specifications to infeasible ones, and every feasible task has a reference design verified at five process corners. Evaluating six frontier LLMs at two reasoning levels, with and without simulation feedback, we find that models port simple circuits but rarely re-design headroom-sensitive amplifiers, seldom recognize infeasible specifications, and gain more from simulation feedback than from more samples or reasoning. Code and data are available at https://github.com/saeidrajabii/AnalogPortBench.

0.34

PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research

cs.CL ·2026-10-23 ·Qituan Zhang, Guo Chen, Linfeng Zhang, Wenlong Zhang
benchmark similarity 0.60
Abstract

Large language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for theoretical-physics reasoning with 63 Reproduce and 43 Discover problems. Reproduce tests derivation of published results; Discover tests consequences of modified theoretical conditions. FormulaForest constructs the tasks with reference solutions and subquestion-level rubrics. Across five models, whole-problem accuracy ranges from 49.21% to 68.25% on Reproduce and from 18.60% to 74.42% on closed-book Discover. With tools, four models complete more Discover problems, but conceptual errors and missing arguments persist. Besides, our experimental results demonstrate the value of FormulaForest as synthetic data. On FrontierScience, a benchmark developed by OpenAI, it raises the accuracy of a Qwen-35B-A3B model from 61% to 72%.

0.35

Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability

cs.LG ·2026-10-01 ·Chuqin Geng, Li Zhang, Haolin Ye, Mark Zhang, Luke Zhang +1 more
benchmark similarity 0.59
Abstract · 35 pages, 3 figures

Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. Yet standard intervention-based evaluation replaces signals outside a circuit to isolate its effect, changing the computational environment in which the retained components operate. We call this change context distortion. This raises the question of whether faithfulness rewards recovery of model behavior or compatibility with the altered context. We show that intervention-based faithfulness can prefer an equally sized circuit that reproduces the model's behavior less well. We call this reversal misranking, an objective-level recovery gap. To isolate evaluation from discovery, we construct equally sized candidates through controlled edits to human-identified circuits in pretrained language models, including IOI and Greater-Than, and benchmark circuits with known mechanisms. We find misranking even without a discovery algorithm. Misranking also occurs in real automated circuit-discovery settings. Among outputs of EAP, EAP-IG, ACDC, and Edge-SP, KL faithfulness under ordinary resampling prefers the behaviorally worse candidate in 13.2%-24.4% of candidate pairs across these methods on four pretrained-language-model tasks. Finally, restoring selected intact-model signals reverses 96 of 100 selected persistent misrankings, demonstrating that context distortion contributes to these failures. In light of these findings, we call for rethinking both how circuit discovery is performed and how its results are evaluated.

0.17

SAFTAC: Simulation-Augmented Fine-Tuning of Open-Source LLMs for Analog Circuit Design

cs.AR cs.AI ·2026-11-05 ·Junsheng Huang, Yifan Sun, Zhuoer Zhang, Ning Wei, Yening Liu +5 more
top venue similarity 0.57
Abstract · Accepted to NeurIPS 2026

Circuit design for analog integrated circuits (ICs) is challenging because circuit topology, device sizing, and circuit performance are tightly coupled. Existing large language model (LLM)-based methods either rely on inference-time correction around frozen models or train on limited circuit types without directly using simulation outcomes as training signals. A key bottleneck is that existing datasets do not provide sufficient task information for training LLMs on end-to-end specification-conditioned design. To fill this gap, we first construct a large-scale simulation-grounded dataset with 8,626 design tasks across 10 different types of analog circuits, where each task includes a textual description, labeled I/O ports, loading condition, target specifications, a testbench, and a reference netlist. Using this dataset, we present SAFTAC, a simulation-augmented fine-tuning framework that trains LLMs to generate and revise analog circuit design. SAFTAC combines three-stage supervised fine-tuning, which progressively builds the capabilities required for analog circuit design and feedback-guided correction, with simulation feedback-augmented reinforcement fine-tuning, which further optimizes models using simulation-grounded rewards. We evaluate SAFTAC under single-pass generation and two-pass generation with simulation feedback, showing improvements over the strongest closed-source LLM by 15.9% and 7.8%, respectively.

0.39

THEIA: A Multimodal Dataset and Benchmark for Vision-Language Analysis of Layout

cs.LG ·2026-09-28 ·Giuseppe Chiari, Michele Piccoli, Federico Viola, Davide Zoni
top venuebenchmark similarity 0.57
Abstract · 10 pages, 10 figures, 14 tables, to be published in NeurIPS 2026

The integration of artificial intelligence into computer-aided design frameworks has sparked a shift in the design of analog integrated circuits (ICs), transitioning the field from using manual and algorithmic-based solutions to adopting automated and intelligent paradigms. In this scenario, the GDSII file represents the industry-standard database containing the ultimate and most accurate source of information of the analog circuit, encapsulating the complex physical geometries and parasitic realities that define tape out performance. This paper proposes THEIA, a novel dataset containing thousands of layout images paired with question-answer conversations, along with a benchmark that employs a fine-tuned vision-language model (VLM) to analyze GDSII files of analog circuits, enabling designers to interact with and query physical layouts as intuitive, meaningful entities. Experimental results using thousands of analog designs across five realistic tasks demonstrate that the proposed fine-tuned VLM outperforms state-of-the-art general-purpose VLMs by a significant margin (up to 73%), highlighting a fundamental gap between general-purpose multimodal reasoning and domain-specific layout understanding.

0.34

Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks

cs.AI ·2026-09-27 ·Analog Design Bench Team
benchmarkcode/data similarity 0.57
Abstract · 25 pages, 10 figures, 6 tables

Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured. We introduce Analog Design Bench, a long-horizon agentic benchmark of 50 transistor-level design tasks contributed by 17 chip designers. Agents work with an open-source simulator, while an isolated verifier evaluates the submitted circuit using specification-based electrical tests. We evaluate 15 agent configurations across 2,250 two-hour attempts and observe full-specification pass rates from 8.0% to 78.0%. Coding-benchmark performance correlates with analog results but leaves much of the performance spread unexplained. Our failure analysis shows that most unsuccessful submissions have no recorded legality rejection but fail electrical acceptance, identifying electrical closure as the dominant endpoint challenge. We test time, reasoning effort, agent harness, and supplied design knowledge as interventions. Longer budgets and higher reasoning effort improve performance, while general skill documents provide little benefit and sometimes reduce performance. Supplying a task-matched reference topology, an idealized form of circuit-IP retrieval, raises DeepSeek V4 Pro by 18.7 percentage points and mainly accelerates GPT-5.6 Sol.

0.27

Inference for Reasoning Models: Practical Lessons from Evaluation and Serving

cs.AI ·2026-11-07 ·Luv Verma
publishedbenchmark similarity 0.57
Abstract · 217 pages. Book, first edition, version 1.0 (August 2026). Published on Zenodo

Running a language model involves more than its trained weights. The same model can receive very different benchmark scores, produce different answers, or cost more to serve depending on how it is evaluated and run. In one experiment in this book, changing only the maximum response length moved accuracy on a math benchmark from 0.08% to 93.18%. This book explains how inference settings shape the results we report and the systems we deploy. We begin with evaluation, including generation limits, answer extraction, scoring methods, and quantization. We then examine why outputs can change across batches, attention backends, and caching, even when sampling is disabled. The next chapters cover serving performance, including batching, memory capacity, latency, throughput, and cost. We also study structured outputs, tool calls, reasoning budgets, and long-context tasks, showing what these controls and benchmarks do and do not establish. The book concludes with practical guidance for recording inference runs, checking evaluations, estimating capacity, and comparing systems fairly. Written for researchers and engineers working with language models, it aims to make inference results easier to understand, reproduce, and trust.

0.24