CS Paper Radar

In Situ Training of Optical Neural Networks by Coding Agents

Ilker Oguz, Colin Christian, Christophe Moser, Demetri Psaltis, Gordon Wetzstein

cs.AI physics.opticscs.LG ·submitted 2026-11-18 05:29 UTC · v1

Abstract

Optical neural networks promise fast, energy-efficient computing, but every device departs from its design and must be trained with the hardware. In situ training relies on an expert-built calibrated digital model of each device or on gradient-free trial and error. Coding agents, large language models that write and run programs, have shown an emergent ability to understand and edit the weights of digital neural networks. Here we introduce optics as code, a method in which a coding agent trains an optical neural network from measurements alone, through a harness that returns camera images of its probes and requires a hypothesis per request. We discovered that the agent forms a physical understanding of the optics and reparametrizes the network with interpretable optical elements tuned on the hardware. It outperforms gradient-free methods in classification and image encoding, approaching backpropagation through a calibrated model at a fraction of the cost. Needing neither per-device calibration nor gradients through the hardware, and improving with more capable models, optics as code offers a practical route to large, scalable optical neural networks and a general tool for calibrating and optimizing optical experiments.

Comment
14 pages, 4 figures. Our project website is at https://ioguz.github.io/optics-as-code/

Closest papers

Nearest neighbours by meaning across the whole index.

Optical Neural Networks Should Be Evaluated by Function, Not Only Physics

cs.ET physics.opticscs.LGcs.NE+1 ·2026-11-17 ·Shi-Yuan Ma, Xingjian Bai, Ryan Hamerly, Mahmoud Jalali Mehrabad
similarity 0.65
Abstract · 28 pages, 3 figures, 4 tables

Optical neural networks (ONNs) are attracting significant attention as alternative platforms for machine learning (ML). In an ONN, an optical module often takes over part of the computation inside a digital ML pipeline. How much of the task does the optical module itself perform, and how much does it contribute to any claimed system-level advantages, such as throughput or energy? This paper offers a simple, function-oriented evaluation methodology. Whatever its internal physics, the optical module appears to the surrounding digital pipeline as a mapping between real-valued tensors. We therefore evaluate it in three steps: (i) define its boundary in the pipeline, as real-valued tensors in and out; (ii) replace it with standard digital modules to identify those that match the optical system's performance on the same task and metric; and (iii) state system-level claims, such as speed or energy, per inference against the performance-matched digital module. We summarize steps (i) and (ii) in an Expressivity Card and demonstrate its use on published ONN systems. This helps isolate the optical module's contribution and provides a concrete digital reference for assessing its claimed system-level advantages, without the need to understand the physics. While we focus on optical systems as the most prominent example, the same approach applies to any physical analog computing platform used in ML, wherever a physical module takes over part of a digital pipeline.

0.36

Architectural Principles for Parameter-Efficient Unitary Transformations in Coherent Optical Neural Networks

cs.LG cs.ARphysics.app-phphysics.optics ·2026-11-16 ·Kazuo Aoyama, Hiroshi Sawada, Shota Kita, Akihiko Shinya, Masaya Notomi
similarity 0.61
Abstract · 28 pages, 28 figures

Coherent optical neural networks based on interferometric architectures offer a promising route toward high-throughput, energy-efficient computation by exploiting the multiple degrees of freedom of light, including spatial, temporal, and spectral modes. However, practical implementations are constrained by the number of tunable parameters due to fabrication complexity, device footprint, and control overhead. In such parameter-constrained settings, achieving high expressive power remains a central challenge. We identify two key architectural requirements for maintaining high expressive power under these constraints: (i) statistical uniformity in the implemented transformations and (ii) strong mixing capability per parameter, which enables rapid and unbiased redistribution of information across modes. To realize these requirements, we develop a logarithmic-depth butterfly-topology architecture whose fundamental building block is a single phase-shifter--beam-splitter pair, in contrast to conventional rectangular mesh architectures based on Mach-Zehnder interferometers. Computational evaluations show that, particularly at restricted parameter counts, the proposed architecture achieves lower matrix-reconstruction errors and higher classification accuracies than conventional meshes in both feedforward and recurrent networks. We further quantify these design principles using graph-theoretic measures, expected squared moduli of transformation-matrix entries, and matrix-reconstruction error. The resulting framework provides a quantitative and physically interpretable basis for parameter-efficient optical neural networks.

0.35

LensDesigner: A Self-Improving Agent for Optical Lens Design

cs.CV ·2026-09-24 ·Lei Sun, Haoran Liang, Dannong Xu, Yao Gao, Yuyu Geng +4 more
benchmark similarity 0.59
Abstract

Optical lens design is a complex, non-convex optimization challenge that relies heavily on human experience and intuition. Existing optimized-based automatic lens design methods struggle to navigate this vast parameter space without meticulous manual tuning. In this paper, we present LensDesigner, an autonomous agent framework that mirrors the problem-solving workflow of expert opticians. To overcome the initial cold start problem, we construct LensLib100K, an extensive optical lens library, and employ Optics-Aware Retrieval to supply physically valid structural seeds. Within an interactive physical simulation environment, the agent executes macroscopic orchestration while receiving immediate optical feedback. Furthermore, we introduce a continuous self-evolving mechanism guided by a curriculum agent. By iteratively solving design tasks with progressively increasing difficulty, the agent autonomously extracts, accumulates, and reuses design heuristics, effectively evolving its optical lens design expertise over time. At the evaluation level, we introduce LensArena, a standardized evaluation benchmark comprising $120$ diverse optical design tasks, covering extreme configurations. Extensive experiments on this benchmark demonstrate that LensDesigner significantly outperforms publicly available baseline algorithms, achieving superior success rates and optimization efficiency. We hope this work sheds light on the emerging field of intelligent optics. The code will be publicly available.

0.39

Optimizing Physical Neural Networks through Device Variable Selection

cs.ET cs.ARcs.CVcs.LG ·2026-10-24 ·Sjoerd van den Belt, Manuel Escudero, Mohamadreza Zolfagharinejad, Wilfred G. van der Wiel, Nikolaos Alachiotis
similarity 0.58
Abstract

Physical neural networks (PNNs) leverage device physics in electronics, optics, and related domains to enable energy-efficient neural networks. Broken-isomorphism PNNs replace linear operations with multivariate physical transformations, modifying how input signals are mapped to device variables. We exploit network architectures emerging from these mappings by treating the routing between signals and input variables of physical devices as trainable parameters. This approach expands device-level tunability of the physical transformations while, at the system level, demonstrating improved expressivity of PNNs with negligible additional inference overhead. This discrete routing is optimized by training a categorical distribution using Gumbel-Softmax reparametrization. We evaluate this approach using experimentally calibrated differentiable models of multivariate electronic RNPUs for MNIST and CIFAR-10 classification. Using variable selection, we improve the MNIST classification accuracy for a small CNN model from 94.5% to 96.9% and the accuracy of CIFAR-10 from 62.8% to 68.0%. Our method uniquely leverages the induced biases of broken-isomorphism PNN architectures, paving the way towards more efficient physics-based neural network inference.

0.16

Interpretable weight-space programming builds transparent transformers and brain encoding models

cs.CE cs.AI ·2026-11-13 ·Yash Mehta, Chandan Singh, Weijia Xu, Richard J. Antonello, Adeen Flinker +4 more
similarity 0.58
Abstract

Neural networks are powerful predictors, yet their opacity limits their usefulness in science, where the goal is explanation rather than mere prediction. In neuroscience, for example, deep encoding models accurately predict brain activity in response to natural stimuli, but struggle to reveal which stimulus features drive those predictions. Here we introduce interpretable weight-space programming (iWSP), a method in which an LLM agent iteratively builds interpretable models by explicitly programming their weights and features in code, without gradient-based training (backpropagation). Without backpropagation to tune parameters, the agent must draw on its prior knowledge and reason over previous trials to hypothesize, hand-craft, and refine human-interpretable computations mirroring the workflow of a human scientist. LLM audits confirm the interpretability of the hand-crafted models. We first validate iWSP by tasking it with constructing transformers for canonical benchmarks such as arithmetic and sentiment classification, finding that LLM agents can successfully craft compact, accurate networks directly. Next, we apply iWSP to build neural encoding models across vision, language, and audition. In the visual cortex, iWSP outpredicts existing interpretable baselines, uncovers distinct computations across early/higher visual areas, and generalizes to unseen data. In language fMRI, iWSP surfaces a confound in the data and in auditory ECoG, it systematically discovers interpretable acoustic features that predict neural activity. Together, these results establish iWSP as a general paradigm for aligning high predictive performance with human-interpretable scientific discovery.

0.27

LCAP: Population-Informed Latent Chip Adaptation from Few Output Probes for Photonic Neural Networks

cs.LG ·2026-09-15 ·Tianyu Gao, Guantian Zheng
similarity 0.57
Abstract · 5 pages, 3 figures, 2 tables

Photonic neural networks (PNNs) offer efficient analog inference, but parameters optimized under ideal device models can degrade after fabrication, creating a persistent simulation-to-hardware (sim-to-real) gap. When many identically designed chips are deployed, calibrating each device from scratch compounds this cost. We propose Latent Chip Adaptation from Probes (LCAP), a population-informed framework that decomposes hardware adaptation into a transferable population correction and probe-inferred latent personalization. LCAP first learns a shared correction from 80 historical chips, then extracts a low-dimensional correction space from device-specific refinements. At deployment, 32 fixed unlabeled output probes infer an unseen chip's latent correction coordinates, enabling feed-forward personalization without target-device optimization. On a three-layer 64-mode MZI simulator with phase variation, beam-splitter errors, quantization, and crosstalk, accuracy improves from 80.4147% under direct deployment to 92.6860% after shared calibration and 93.3617% with LCAP. LCAP improves 27/30 unseen chips and raises worst-device accuracy from 89.18% to 90.54%.

0.31

Wavelength Diversity Expands the Expressive Power of Optical Computing

cs.NE physics.opticsphysics.app-ph ·2026-10-16 ·Shiqi Chen, Che-Yung Shen, Parnian Ghapandar Kashani, Yongkang Cheng, Aydogan Ozcan
similarity 0.57
Abstract · 34 Pages, 10 Figures

Universal function approximation underpins neural networks' capacity to represent nonlinear relationships, but realizing this capability in optical hardware requires engineering the computational features encoded in light. Here, we show that wavelength diversity enhances optical function-approximation expressivity through wavelength-scaled Fourier features generated by a shared phase encoder, without increasing the number of input-encoded harmonics. Multiple wavelengths contribute to the representation of each output function rather than merely serving as independent computational channels. We establish theoretical scaling laws and approximation-error bounds, revealing that, for nondispersive encoders, the spectral endpoints determine the accessible computational bandwidth, while intermediate illumination wavelengths enrich the available harmonics within that bandwidth. Numerical studies demonstrate improved approximation of real- and complex-valued nonlinear functions, with a representative design achieving more than a 100-fold reduction in mean-squared error relative to tested monochromatic optical processors with the same number of trainable phase features. Experimentally, we validate enhanced approximation accuracy and demonstrate the simultaneous approximation of 4,136 distinct nonlinear functions using a compact, single-layer diffractive processor under five-wavelength illumination. These results establish wavelength diversity as an additional physical resource for scaling the expressive power of optical computing and enhancing the accuracy of parallel nonlinear function approximation.

0.24

PISim: A Unified Photonic Inference Simulator for Hardware-Aware Cross-Layer Design-Space Exploration

cs.AI physics.optics ·2026-11-12 ·Andreas Falkenberg
similarity 0.56
Abstract · 8 pages, 2 figures

Photonic neural accelerators offer ultra-low latency linear algebra, yet their usable numerical precision, sustainable throughput, and application accuracy remain tightly bounded by surrounding electro-optical interfaces and physical non-idealities. Pre-fabrication design-space exploration (DSE) requires a cross-layer electronic design automation (EDA) framework capable of evaluating disparate optical topologies under identical neural workloads and physical assumptions. We present \textbf{PISim}, a unified Photonic Inference Simulator that couples pretrained deep neural network execution with physics-based models of six accelerator architecture families, six electro-optic modulator technologies, four optical source organizations, and direct or coherent readout schemes. PISim embeds stage-resolved impairment operators---covering control and analog-to-digital quantization, finite extinction ratio, shot noise, correlated relative-intensity noise (RIN), thermal drift, propagation loss, and channel crosstalk---alongside an analytical cycle-level interface timing model to expose electronic-optical bottlenecks. Using simulated MNIST multilayer perceptron (MLP) benchmarks, we demonstrate how PISim identifies optimal precision operating points under multi-impairment stress (peaking at 6 bits for Mach--Zehnder and microring modulators), quantifies a 52.5 percentage-point accuracy gap between VCSEL-array and parallel-fanout laser architectures arising, under the modeled conditions, from power partitioning and correlated source noise, and models interface-limited throughput scaling across emitter-array widths from 32 to 128 channels. This comprehensive framework establishes an extensible, reproducible foundation for accuracy-constrained Pareto optimization prior to photonic tape-out.

0.25