CS Paper Radar

Memory-Efficient Integer-Only Training of Deep Neural Networks

Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri

cs.AI ·submitted 2026-10-12 07:25 UTC · v1

Abstract

Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural networks, in which every tensor (weights, gradients, and activations) is stored as INT8, and every operation (forward pass, backward pass, weight update, optimizer update) is performed in integer arithmetic. Specifically, tensors are stored as signed integer mantissas paired with explicit power-of-two exponents, on which integer-only operations are performed using five custom composable building blocks (products, sums, divide-by-constant, nonlinear functions, and weight updates). This allows complete implementation of trainable integer layers, including convolutions, batch and layer normalization, softmax, and GELU. A per-group velocity mechanism with independent exponent tracking preserves gradient precision across neurons and channels with diverse magnitudes. As every tensor fits in INT8, training memory is roughly 4x smaller than the FP32 baseline. We validate our training algorithm on CNNs (ResNet-18/34/50/101, MobileNetV2) and Vision Transformers (ViT-Ti / B), demonstrating for the first time that training neural networks with INT8 tensors and integer arithmetic can achieve competitive accuracy on CIFAR-10, CIFAR-100 and ImageNet at a reduced memory budget.

Comment
19 pages

Closest papers

Nearest neighbours by meaning across the whole index.

TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation

cs.LG ·2026-10-07 ·Eliyahu Levy, Adam Teman, Yoni Pugachov
similarity 0.70
Abstract

Post-training quantization (PTQ) enables efficient deployment, yet transformer architectures remain challenging to quantize due to nonlinear layers. While existing methods attribute accuracy loss to insufficient numerical precision, often necessitating floating-point fallbacks, we demonstrate that degradation is actually driven by specific structural error sources. We find that learned scale parameters in normalization layers and compounded approximations in GELU are the primary error contributors, whereas SoftMax remains inherently robust to aggressive quantization. To address these bottlenecks, we introduce TR-PTQ, a unified integer-only formulation using shared Taylor Region (TR) exponential and logarithm primitives. This approach allows computationally expensive operations, including division and square roots, to be performed entirely in the log-domain via standard integer arithmetic. Combined with a calibration-free, outlier-aware optimization for LayerNorm parameters, our method eliminates the need for floating-point hardware units for nonlinearities, achieving less than 1.5\% absolute accuracy degradation across vision and language benchmarks.

0.23

Decode Late, Multiply Once: Rethinking the Dot Product for Quantized Weights

cs.LG cs.AR ·2026-10-22 ·Jonathan Tapson
similarity 0.68
Abstract

Large neural networks are increasingly deployed with 1-4 bit weights and 16- or 32-bit activations, yet their dot products still use one multiplier per element. We show that a length-n dot product with k-bit weight codes can be computed with n+2^k additions and at most 2^k decodes and 2^k multiplications. Each weight code is used directly as an address offset that selects one of 2^k accumulators; each activation is added into the selected accumulator; and the accumulated sums are decoded and multiplied only once, at the end, in a fused decode-and-multiply step. The decoded weight vector is never formed. The refactoring is exact for integer arithmetic, works with integer or floating-point operands and with arbitrary, including nonlinear, decoders, and extends to group-wise scales and to matrix and tensor products. The multiplier count falls by n/2^k, while total arithmetic energy improves by a smaller, bounded factor that we quantify with a cost model. We describe a scatter-accumulate processing element (SA-PE) that replaces the MAC unit in a weight-stationary systolic array, with a load/stream/reduce/drain schedule in which a small, time-shareable multiplier runs only 2^k times per tile. Because decoded weights are not stored, local weight memory shrinks by up to w_d/k for w_d-bit decoded weights. An RTL implementation synthesized for FPGA and compared with a matched MAC array shows that an unshared SA-PE is larger than a MAC PE, and that the saving appears when the reduction multiplier is shared across PEs. Software models confirm exact integer equivalence and floating-point error no worse than the conventional pipeline. We do not claim the underlying identity is new, and we discuss related work on weight sharing and lookup-based kernels.

0.23

MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference

cs.AR cs.AI ·2026-09-23 ·Romain Facq, Sami Ben Ali, Olivier Sentieys
similarity 0.67
Abstract · 12 pages, 7 figures

Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers' forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of $\times2$ for weights and gradients, and by up to $\times9$ for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to $\times7.53$ compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by $\times3.5$ for state-of-the-art object detection models YOLOV8nano and $\times2.2$ for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%.

0.30

AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

cs.AI ·2026-11-06 ·Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan, Charbel Sakr, Tijmen Blankevoort +2 more
similarity 0.67
Abstract

Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.

0.23

FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization

cs.AR cs.LG ·2026-10-21 ·Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang +2 more
similarity 0.66
Abstract · 15 pages, 12 Figures

Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix multiplication. Recent quantizers reach a 2-3-bit average with mixed precision, but hardware has not kept pace: GPUs dequantize weights to FP16 before the tensor-core multiply-accumulate, and bit-parallel FP-INT accelerators run at a single weight precision, so fewer bits save memory but not computation. In this paper, we present FIGFLEX, an algorithm-hardware co-designed mixed-precision accelerator whose bit-parallel datapath scales throughput as precision falls. FIGFLEX contributes (1) a shift-add, radix-4 reformulation of FP-INT multiplication that fuses the multiply-accumulate into a unified, precision-flexible partial-product reduction and roughly halves the partial products, together with a novel INT3 optimization that reclaims the idle datapath radix-4 leaves at that precision; and (2) a mixed-precision quantizer that assigns each channel group its bit-width according to the error GPTQ cannot compensate, with a per-input-channel column scaling whose power-of-two normalization keeps the scaled activations out of the FP16 subnormal range, so that the quantized model maps directly onto FIGFLEX's datapath with the same accuracy as on a GPU. On LLaMA-2-7B, FIGFLEX attains 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, lower than GPTQ, AWQ, SliM-LLM, AMQ, and QuIP# (without fine-tuning), while delivering 1.56x and 1.42x higher whole-chip throughput per area on the linear GEMMs of a 3-bit mixed-precision model than AxCore and FIGLUT, respectively.

0.13

ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator

cs.LG ·2026-10-15 ·Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem Ulkar
code/data similarity 0.66
Abstract

Fixed-function CNN accelerators make edge inference fast and energy-efficient, but they execute only networks that obey a precise numerical contract. On the MAX7800x family, this contract requires power-of-two, per-tensor, symmetric scales, 8-bit activations and biases, integer-only arithmetic, and one statically configured shift per layer. We present ShiftQAT, the quantization-aware training (QAT) flow we developed for this family, which produces networks satisfying the contract by construction. It is the default training path of ai8x-training, the family's open-source training toolchain. ShiftQAT resolves residual, concatenation, and weight-sharing dependencies so that activation scales propagate statically and each layer's requantization reduces to one offline-configured shift. It also trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget. Among the compared methods, only ShiftQAT satisfies all of the contract's scaling constraints, including signed 8-bit activation storage. Even so, on ImageNet it matches the W8A8 accuracy drop of less constrained QAT methods and stays within 2.9 percentage points of FP32 at W4A8. On MAX78002 silicon, ShiftQAT-trained MobileNetV2 and ResNet-18 classify ImageNet images in 14.6 and 8.4 ms at 5.1 and 2.8 mJ per inference.

0.20

ZTA-Q: an Open-source RISC-V Platform for Accurate Quantized CNN Inference

cs.AR ·2026-10-01 ·Yike Li, Ajay Kumar M, Vishnu PS, Dimitrios S. Nikolopoulos, Bo Ji +2 more
publishedcode/data similarity 0.65
Abstract · Accepted for publication at the 2026 IEEE 33rd International Conference on Electronics, Circuits and Systems (ICECS)

Low-precision inference is widely adopted in edge AI to reduce computational cost and memory footprint. However, existing open-source accelerator platforms provide limited end-to-end support for CNNs following the standard TensorFlow Lite integer inference scheme. This paper presents ZTA-Q, an open-source RISC-V-based platform that enables accurate deployment of TensorFlow Lite INT8 models. In addition to extending operator support, ZTA-Q provides a configurable post-processing datapath for studying how circuit-level approximations, including reduced multiplier precision, shared shift scaling, and simplified rounding, affect model accuracy. The proposed system is implemented on a Digilent Arty A7-100T FPGA and operates at 83.3 MHz. Evaluations on representative CNN models show that with LUT, register, and DSP overheads of 26.3%, 12.6%, and 150%, respectively, ZTA-Q limits the degradation in both top-1 and top-5 accuracy to within 0.25 percentage points.

0.35

Enabling Memory-efficient Im2win Convolution with Multi-precision Support on GPU CUDA and Tensor Cores

cs.DC cs.CV ·2026-08-21 ·Xiang Fu, Jixiang Ma, Xinpeng Zhang, Peng Zhao, Shuai Lu +1 more
published similarity 0.65
Abstract · Accepted at the 2026 International Joint Conference on Neural Networks (IJCNN 2026). To appear in IEEE Xplore

Convolution is a principal computational bottleneck in deep neural networks, and its efficiency depends on tight integration between algorithms and GPU hardware. Existing GPU convolution methods suffer from large memory overhead, poor cache utilization, limited effectiveness across kernel sizes, or numerical instability. This work extends the im2win paradigm -- a universal, memory-efficient convolution method with contiguous memory access for all kernel sizes -- to run efficiently in full precision on CUDA cores and half precision on tensor cores. By introducing new kernel designs and optimizations such as zig-zag memory access and asynchronous data movement, im2win efficiently exploits hardware-accelerated half-precision matrix multiply-accumulate operations. Across twelve CNN benchmarks, im2win achieves up to 2.8x higher TFLOPS than its CUDA core implementation, 1.4x higher than cuDNN, and 6.4x higher than GEMM-based convolution with cuBLAS, while using as little as 53% and 35% of their memory, respectively. These results establish im2win as a unified, high-performance convolution framework for modern GPU architectures.

0.50