CS Paper Radar

Testyard Benchmarks: a Dynamic RISC-V-based Benchmark Framework including DfT insertion, Structural and Functional Test Flows

Francesco Angione, Nicola di Gruttola Giardino, Gabriele Filipponi, Giusy Iaria, Irith Pomeranz, Paolo Bernardi

cs.AR ·submitted 2026-11-10 15:25 UTC · v1
benchmarkcode/data

Abstract

The main challenge of modern computing manufacturing is to produce silicon products capable of operating under the worst possible conditions, both in data centers and under stringent functional safety constraints. These requirements impose exceptionally high quality standards on silicon vendors, while testing costs keep escalating as transistors continue to shrink. Evaluating new testing strategies, however, presents a dual challenge for researchers. Standard academic benchmarks, such as the ITC-99 suite, remain valuable for fast test prototyping, but they lack the structural complexity of modern VLSI devices. Assessing the true computational effort of academic methodologies on industrial scale designs is equally difficult without accessible and scalable models. A further obstacle is the absence of a unified framework able to abstract the commands imposed by each EDA tool and the configurations tied to each technology library, forcing researchers to spend time adapting and debugging fragile scripts. This work addresses both needs by proposing the Testyard framework, built on top of the Chipyard framework, to derive a set of open-source, industrial scale benchmarks. It also extends the Hammer VLSI flow abstraction framework with hardware testing capabilities. Three new functionalities are introduced into Hammer: Design-for-Testability (DfT) insertion, ATPG, and functional tests fault simulation. Each functionality is driven by high-level descriptions, while the framework translates these inputs into the commands required by each EDA tool. The extended framework is applied to 13 representative SoC configurations from the Chipyard ecosystem, reporting the first comprehensive structural and functional test results across three technology libraries and multiple fault models, including stuck-at, transition delay, and small delay faults.

Closest papers

Nearest neighbours by meaning across the whole index.

An Open-Source Benchmark Suite of 3D-IC Testcases

cs.AR ·2026-08-25 ·Rohan Soni, Jooyeon Jeong, Alexander Graening, Anthony Foo, Richard Chen +1 more
benchmarkcode/data similarity 0.64
Abstract

The physical design community has benefited from standardized, publicly available benchmark suites, which have enabled reproducible evaluation and driven significant advances in 2D place-and-route algorithms over the past three decades. However, the emergence of 3D heterogeneous integration technologies, including through-silicon vias (TSVs), hybrid bonding, and chiplet-based architectures, has introduced new physical design challenges that are not captured by existing planar benchmarks. Although several 3D-IC design examples have been reported, publicly accessible and scalable benchmark suites that enable reproducible evaluation across different 3D physical design problems remain limited. In this paper, we present an open-source suite of 3D-IC benchmark testcases derived from representative chiplet-based case studies in CATCH, an open-source framework for estimating the cost of heterogeneous integration architectures. The proposed benchmark suite provides reusable virtual chiplet models covering compute, memory, I/O, analog, and substrate components. Each testcase captures essential physical design characteristics of 3D systems, including heterogeneous die integration, inter-die connectivity, and technology-dependent design constraints. By publicly releasing these benchmarks, we aim to establish a common evaluation platform and accelerate community-wide research progress in 3D heterogeneous integration.

0.32

Efficient Hardware Information-Flow Tracking for Pre-Silicon Security Testing

cs.CR cs.LO ·2026-09-06 ·Yu-Wei Fan, Yuheng Yang, Christine Guo, SooHyuk Cho, Thomas Bourgeat +2 more
code/data similarity 0.60
Abstract

Register-Transfer Level (RTL) simulation is widely used to test hardware before it is fabricated. To allow testing for security related information flow properties, such as confidentiality and integrity, taint logic can be automatically added to the design to track how information flows through it. However, taint logic instrumented by the state-of-the-art, such as CellIFT, makes simulation-based testing prohibitively expensive: On our evaluation of Mega-BOOM (136K cells), it increases the instrumented design to 5.81x the original cell count and causes a 143.72x simulation slowdown. The taint logic could be simplified to improve simulation speed, but it will inevitably trade off its precision. This lightweight, imprecise taint logic will introduce false positives and may eventually result in even more overhead to check these false positives. This paper explores the research question of where precision is actually needed in the design to overcome the overhead of false positives. It presents CEGAR-T, a framework that automatically synthesizes taint logic that minimizes the taint-logic instrumentation overhead while guaranteeing no false positives (relative to the precise CellIFT baseline). We have implemented CEGAR-T and evaluated it on the safe instruction set problem for timing side-channel security across open-source RISC-V cores. Over all evaluated cores, CEGAR-T reduces both instrumentation and simulation overhead, in geometric-mean, from 5.64x to 1.42x and from 34.65x to 1.79x, respectively, without compromising the precision benefit of the CellIFT baseline.

0.29

VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation

cs.PL cs.ARcs.CL ·2026-10-04 ·Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav, Ehsan Degan, Vandana Mukherjee
publishedbenchmarkcode/data similarity 0.59
Abstract · 7 pages

Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages. However, most existing benchmarks focus on Verilog, with limited evaluation of VHDL, which remains widely used in industry and academia for FPGA and safety-critical systems. To address this gap, we introduce VHDL-REPOBENCH, a large-scale, cross-file, repository-level benchmark for assessing LLM capabilities on realistic VHDL design generation and analysis tasks. VHDL-REPOBENCH curates ~100 open-source VHDL repositories, encompassing ~2.5k VHDL files and ~500 testbenches, and provides structured problem statements, module stubs, and self-verifying testbenches. The benchmark enables comprehensive evaluation across syntax, semantic correctness, hierarchical reasoning, cross-file dependency resolution, and functional verification. We evaluate several state-of-the-art models, including GPT-4o, Llama-3-70B, Qwen2.5-72B, CodeLlama-70B, and multi-step reasoning approaches such as Reflexion and CoDes. Results reveal that while current LLMs achieve moderate line- and block-level accuracy, substantial challenges remain in multi-file reasoning, hierarchical design understanding, and specification-to-module generation. VHDL-REPOBENCH represents the first large-scale VHDL-focused benchmark and provides a valuable resource for the hardware design community to evaluate, compare, and advance LLM capabilities for practical VHDL development.

0.38

GateTruth: Auditing the Rigor of RTL Design Benchmarks via Mutation Testing

cs.AR ·2026-08-12 ·Meet Bhadra
benchmarkposition similarity 0.58
Abstract

Benchmarks for evaluating large language models on register-transfer-level (RTL) hardware design have proliferated rapidly, yet none reports having applied mutation testing, an established hardware-verification technique for quantifying testbench quality, to ask whether its own testbenches are trustworthy. A testbench that never fails is not evidence of a correct design; it may simply never stimulate the logic that is actually broken. We introduce GateTruth, a mutation-testing engine and methodology for auditing RTL benchmark testbench rigor: inject a deterministic, seeded set of semantic mutants into a reference design and measure what fraction the testbench catches. We validate the methodology against our own 68-task, dual-track reference suite -- 60 specification-to-RTL generation tasks and 8 agentic-repair tasks, scored through a pinned, deterministic synthesis-to-timing flow with correctness enforced as a strict gate -- certifying that 46 of 60 Track A testbenches kill at least 95% of injected mutants under sequential, reproducible execution; we disclose why the other 14 do not, including a Goodhart effect on testbenches revised to pass this gate. We then point the same engine, unmodified, at RTLLM v2.0, a widely adopted external benchmark: of 46 auditable designs, 72% fall below the 95% floor our own suite is held to, and three score 0% outright. A comparable audit of NVIDIA's CVDP benchmark is structurally impossible: its public release withholds reference solutions, removing the golden RTL mutation testing requires. Auditing our own instrument also surfaced a second finding: an initially uniform 4096-token output cap silently truncated three of seven evaluated models, and re-running at 16,384 tokens moved one model from fifth place to first. We argue mutation-kill certification should become a standard reporting requirement for RTL-generation benchmarks generally.

0.36

Lessons from the Hardware Hacking Competitions: Verification Techniques, Findings, and Insights

cs.CR cs.ET ·2026-08-23 ·Sudipta Paria, Aritra Dasgupta, Raghul Saravanan, Jayanth Thangellamudi, Sai Manoj P D +1 more
similarity 0.58
Abstract · 11 pages, 5 figures, 9 tables

Hardware hacking competitions have emerged as practical platforms for evaluating security weaknesses in complex System-on-Chip (SoC) designs while promoting security-aware verification and tool development. This paper presents a systematic study of SoC security verification through open-box hardware hacking competitions, focusing on practical vulnerability analysis strategies, observed findings, and lessons for security-aware verification. We present a multi-strategy vulnerability analysis methodology, combining simulation-based verification, formal verification, lint analysis, Large Language Model (LLM)-assisted bug detection, and coverage-guided hybrid fuzzing. Representative vulnerability findings are analyzed to illustrate how different techniques expose complementary classes of security flaws, and we derive practical lessons for pre-silicon security verification. Finally, we discuss how competition benchmarks can support the reproducible evaluation of emerging hardware security techniques and guide future security-aware EDA research.

0.19

GarageWorks: A Shared Testbench Abstraction for Software Simulation and FPGA Testing

cs.AR ·2026-10-11 ·Kazutomo Yoshii
similarity 0.57
Abstract

GarageWorks is a lightweight FPGA testing framework that enables the same cocotb testbench to be used for both software RTL simulation and FPGA-accelerated hardware simulation. A shared abstraction hides backend-specific interfaces, while synthesizable Advanced eXtensible Interface (AXI) dispatchers provide design-under-test (DUT)-specific control and test operations on the FPGA. GarageWorks is intended for rapid bare-metal testing of standalone designs and on-chip processing logic for scientific sensors, rather than full SoC-level prototyping. The framework has been validated on AMD Alveo V80 FPGAs with several internal designs and provides initial support for AI-assisted generation of DUT-specific test components.

0.27

SoK: ARCUS: On the Efficiency and Efficacy of Hardware Fuzzing

cs.CR cs.SE ·2026-08-25 ·Alenkruth Krishnan Murali, Raghul Saravanan, Sai Manoj P D, Ashish Venkat
survey similarity 0.57
Abstract

This work presents a comprehensive analysis of contemporary hardware fuzzing techniques applied across three major abstraction layers: Instruction Set Architecture (ISA), microarchitecture, and Register-Transfer Level (RTL). Our study examines key factors including input stimulus quality, mutation strategies, feedback mechanisms, target platforms, reference models, and achieved coverage. We find challenges, goals, and design trade-offs vary significantly across abstraction layers. We further identify several unmet needs in current hardware fuzzing practices, such as intelligent input generation, reliable and scalable golden reference models, expressive feedback channels, and cross-layer integration. Building on these insights, we outline future research directions, including hybrid fuzzing frameworks, AI-assisted test generation, scalable reference models, standardized evaluation metrics and benchmarks, and human-in-the-loop automation for guided exploration and analysis. Together, they aim to unlock efficient, reliable, and comprehensive hardware verification solutions.

0.32

Ankhdjet: An Open-Source Compiler for Mask-Programmed Ternary Compute-in-ROM on an Open PDK

cs.AR cs.ET ·2026-08-26 ·Mohnish Pai
code/data similarity 0.57
Abstract · 17 pages, 9 figures

Large-language-model inference is dominated by weight movement: every generated token re-reads every weight. Ternary quantization (BitNet b1.58) shrinks each weight to 1.58 bits with reported parity at the 2B-parameter scale, small enough that hardwiring the weights into a read-only mask becomes plausible, and a commercial chip (Taalas HC1) has validated hardwired weights on an advanced node behind closed tooling. This paper asks whether model-specific silicon can be made reproducible with entirely open infrastructure. We present Ankhdjet, an open-source compiler that lowers a HuggingFace ternary checkpoint (BitNet b1.58 and its kin) to a via-mask program of a fixed compute-in-ROM macro on the open SKY130 PDK, verified end to end with open tools. We defend two claims: (1) the first open-source weights-to-mask compute-in-ROM compiler on a fabricable open PDK, taken through full open-toolchain signoff (KLayout DRC zero, netgen LVS zero, clean timing) twice with two different weight matrices through an identical flow in which only the mask differs; and (2) the first compute-in-ROM macro submitted for fabrication on an open PDK (TinyTapeout/ChipFoundry SKY130 shuttle ttsky26c, silicon expected 2027). The compiler's read is fully digital (full-swing bitline sampling into standard cells, so the emission ports to any node by synthesis); an analog comparator readout of the same machine is retained as a measured variant, replaced by the digital readout at one eighth the area. We additionally report the per-level adversarial verification methodology that caught three DRC-invisible silent shorts, a negative result on storing two ternary weights per transistor at 130 nm, and which analog sense techniques survive at which nodes. Read energy is from extracted-parasitic simulation (0.98-1.73 pJ per sensed weight); no energy measurements are claimed, and the fabricated vehicles are demonstrators.

0.42