CS Paper Radar

How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation?

Youssef Moawad, Kathleen West, Vasilis Bountris, Philipp Thamm, Yehia Elkhatib, Lauritz Thamsen

cs.DC ·submitted 2026-08-10 09:39 UTC · v1

Abstract

Distributed batch data processing applications are widely executed on cloud-based resources where restricted user access to node-level hardware energy counters hinders transparent sustainability accounting. Energy and carbon attribution methodologies therefore depend on power models and available resource utilisation traces, yet the accuracy of these estimates has to be validated while direct counters are available. In this work, we use Apache Spark running on Kubernetes as a case-study dataflow runtime and cluster resource manager to compare model-based energy estimates to Intel RAPL package and DRAM energy on an AWS bare-metal cloud and an on-premises cluster, comparing different CPU usage signals and memory coefficients. We show that external monitoring improves signed package-energy error relative to Spark task traces, reducing underestimation from -29.58% to -24.41% on AWS and from -24.00% to -16.22% on-premises.

Comment
12 pages, 3 figures, conference

Closest papers

Nearest neighbours by meaning across the whole index.

Collecting Infrastructure-Level Measures for Energy Analysis in Modern Computing Infrastructures

cs.DC ·2026-11-17 ·Isac Pasianotto, Stefano Cozzini, Niccolo Tosato
top venuedataset similarity 0.72
Abstract · Accepted version. Published in SC Workshops '26 (Chicago, IL, USA, 2026). Version of Record: https://doi.ieeecomputersociety.org/10.1109/SCW…

Modern HPC facilities are complex environments in which all components are tightly coupled; consequently, precisely modeling and accounting for energy consumption requires approaches that go beyond instrumenting individual compute nodes. Aiming for a cluster-wide vision, we designed a scalable data collection pipeline that aggregates fine-grained measurements from multiple subsystems-spanning component loads, software and hardware counters, sensors and energy-ensuring consistent sampling frequencies and synchronized data acquisition across the infrastructure. We present a dataset of comprehensive metrics collected at 1 Hz over one month from a production research data center supporting heterogeneous workloads, including HPC, Kubernetes services, and distributed storage. To illustrate what this approach makes possible in practice, we analyze two case studies-distributed I/O and large-language-model inference-showing how cluster-wide measurement reveals system-level dynamics missed by localized instrumentation. The proposed dataset and collection methodology provide a foundation for future work on predictive modeling, energy-aware resource management, and more efficient computing infrastructures.

0.53

GreenPipe: Power Modeling for Containerized DNN Inference on Kubernetes Edge Nodes

cs.DC cs.PF ·2026-09-04 ·Mengxue Wang, Peini Liu, Amir Taherkordi, Jordi Guitart
similarity 0.64
Abstract · Accepted to Workshop proceeding of Euro-Par 2026; 6th International Workshop on Performance and Energy Efficiency in Concurrent and Distribu…

Distributed DNN inference is increasingly deployed in containerized edge-cloud environments, where workloads run on-device or are exposed to remote clients over the network. Accurate online power estimation on resource-constrained ARM nodes without hardware power counters such as RAPL remains a challenge, and CPU-only models fail to capture multi-resource behavior. We present GreenPipe, an automated profiling-training-validation pipeline that builds multi-resource regression models from external power meter measurements and attributes power to containers proportionally. GreenPipe is evaluated on a Raspberry Pi 4 edge node in a K3s edge-cloud testbed, covering DNN inference with three vision models, multiple precisions, thread counts, and both local and serving scenarios. System-level MAPE is 6.3-9.4%, improving over CPU-stress and utilization-only baselines by 26.9% MAPE on average. We jointly report inference latency and energy per inference, exposing performance-energy trade-offs across workload configurations.

0.17

Joule-Profiler: Profiling the Energy Consumption of Build Automation Tools Made Easy

cs.SE cs.DCcs.PF ·2026-09-25 ·Jérémy Woirhaye, François Gibier, Romain Rouvoy
code/data similarity 0.63
Abstract · 4 pages, 2 figures. Submitted to the ICSE 2027 Tool Demonstration and Data Showcase Track

Build pipelines are integral to modern software development, yet their energy footprint remains largely invisible to practitioners. Existing CI energy tools either rely on model-based estimation (due to hardware access restrictions in cloud runners) or report only total pipeline energy without decomposing it into meaningful phases. Joule-Profiler is an open-source command-line tool for Linux that measures hardware energy consumption via Intel RAPL (CPU), NVML (NVIDIA GPU), and attributes it to user-defined program phases by monitoring standard output. In this tool paper, we demonstrate Joule-Profiler in the context of Maven build pipelines using Google Gson as a case study. By applying a token pattern-matching Maven plugin invocations, we analyze builds across the 5 most recent Gson releases into per-phase energy profiles, comparing cold builds (empty local repository) and warm builds (cached dependencies).

0.30

GreenDirector: carbon- and water-aware workload placement for sustainable computing

cs.DC ·2026-09-11 ·Jaime Iglesias Blanco, Ignacio Heredia, María Castrillo, Andrei Tsaregorodtsev, Mazen Ezzeddine +1 more
similarity 0.63
Abstract

The rapid growth of data center electricity demand, accelerated by AI, makes carbon-only accounting an incomplete measure of computing's environmental impact: low-carbon electricity mixes are often water-intensive, and the resulting harm depends on local, seasonal scarcity rather than on the volume of water consumed. We propose the Environmental Score (ES), a unified, dimensionless index in $[0, 100]$ that jointly captures the carbon footprint and the spatial-temporal, scarcity-weighted water impact of the electricity a workload consumes. It combines real-time, cross-border electricity flow tracing with monthly AWARE2.0 water-scarcity characterization factors, weighting global greenhouse-gas emissions together with the local, seasonal severity of water stress. Building on it, we define the Green Score (GS), a scheduling metric proportional to the useful computational work delivered per unit of real environmental impact, which also accounts for data center power and hardware efficiency. We add both metrics as a green-affinity feature to the GreenDirector schedulers of two production federated infrastructures, the AI4EOSC scientific cloud and the DIRAC workload management system. In AI4EOSC, a cluster-filling experiment over four pan-European providers shows that greener sites are filled first without degrading scheduling latency or end-user experience. In DIRAC, a trace-driven simulation of 133,631 jobs and a preliminary production deployment for the KM3NeT community reduce carbon emissions and improve carbon efficiency by about 40\%, while making the carbon-water trade-off explicit when the lowest-carbon site also carries higher water stress. The results show that hydrological stress can be dynamically weighted into workload placement in live, multi-tenant systems

0.19

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE

cs.DC ·2026-10-23 ·Jie Ren, Ivy Peng
similarity 0.62
Abstract

Memory demands from data analytics and LLM workloads grow rapidly, driving up memory costs in cloud and data centers. Multi-tier memory has emerged as a promising solution to balance performance and cost. However, effectively leveraging it requires a deep understanding of workload memory behaviors. This work introduces a high-resolution memory characterization methodology for analyzing workload-specific opportunities for multi-tier memory usage. In response to the increasing deployment of ARM-based processors in data centers, we implement our methodology using the recent Statistical Profiling Extension (SPE) for ARM. By characterizing eight memory-intensive workloads from CloudSuite and LLM applications, we identify region-specific access patterns, evaluate the effectiveness of cloud-oriented cache hierarchies, and propose new metrics based on memory idle periods and latency profiles to determine suitable data placement for tiered memory.

0.19

Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models

cs.CY cs.LG ·2026-09-27 ·Joshua Horswill, Ross Hunter, Matt Clifford, James Hall
similarity 0.62
Abstract · 25 pages, 12 figures

We describe a methodology for estimating the per-token energy cost of cloud-hosted large language model (LLM) inference, separating between input (prefill) and output (decode) tokens. Graphics processing unit (GPU) energy usage is measured during inference benchmarking with open-weights models on a wide range of text-based tasks. The remaining server energy contribution from non-GPU hardware is estimated from the inference wall time. Bayesian linear regression is used to model the relationship between energy per token and LLM size, request traffic, and hardware deployment configuration. Proprietary frontier LLMs of unknown size and deployment are binned into size buckets based on naming conventions and performance priors, and the space of possible LLM configurations is sampled with Monte-Carlo methods to give a representative average energy per token and uncertainty. We also describe how these energy measurements can be used to estimate the carbon-dioxide equivalent ($\mathrm{CO_2\text{-}eq}$) emissions, both usage and embodied, and water consumed per token of AI inference. This methodology provides actionable data that enables reductions in cost, electricity usage, $\mathrm{CO_2\text{-}eq}$ emitted and water consumed in cloud and Software as a Service (SaaS).

0.10

GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon

cs.CL cs.AIcs.PF ·2026-08-24 ·Rajeswari Kannan, Raj Firke, Shreya Bengle, Srushti Deshmukh
publishedcode/data similarity 0.61
Abstract · 7 pages, 1 figure, 6 tables. Accepted at IEEE ICCUBEA 2026

The rapid proliferation of Large Language Models (LLMs) has raised concerns about their environmental impact during inference. While Green AI research has focused on datacenter GPUs and embedded platforms, the energy profile of LLM inference on Apple Silicon, with its unified memory architecture, remains unstudied. This paper presents GreenBench, a benchmarking framework that evaluates the energy efficiency, throughput, and carbon footprint of five open-source LLMs (3-9B parameters) across three NLP tasks on an Apple M4 Pro with 48 GB unified memory. Using macOS powermetrics for direct power measurement and Ollama's nanosecond-precision timing, we find that the M4 Pro draws only 0.47 W of CPU+GPU package power during sustained inference, with total system power of 8-12 W, achieving 30-40x better energy efficiency per token than datacenter GPUs in single-user deployment. Smaller models (3-3.8B) deliver 2.6-4.2x higher throughput and up to 62% less energy per token than larger models (7-9B). Pareto analysis identifies Qwen 2.5 (7B) as the optimal accuracy-efficiency trade-off at 57% MMLU and 59 tokens/s, while Llama 3.2 (3B) suits latency-critical applications at 175 tokens/s. We provide per-token energy at package and system levels with CO2 estimates for India and US grids.

0.39

Green500: Adventure in Qualifying a Cluster

cs.DC ·2026-11-06 ·Nicolas Richart, Xavier Ouvrard, Julia Paolini, Manuel Cubero-Castan, Gilles Fourestey
similarity 0.61
Abstract · submitted to EESP Workshop 2026

Energy effciency has become a key metric for evaluating High Performance Computing (HPC) systems, as reflected by the Green500 ranking. However, energy effciency alone does not capture the full environmental impact of an HPC cluster. Using the Kuma cluster at EPFL, we report on two Green500 measurement campaigns at Level 2 and Level 3, and extend this analysis to a full carbon footprint assessment using the MITSI life-cycle based methodology. GPU power capping experiments show that energy effciency peaks at 450 W, achieving an 11% performance loss offset by a 20% reduction in energy consumption. When embodied emissions are accounted for, the optimal power cap for carbon effciency shifts to 500 W under Switzerland's low-carbon electrical grid, revealing a tension between minimizing usage and embodied contributions. We discuss the challenges of generalizing this approach into a broader carbon-based HPC ranking, and argue that moving beyond energy effciency is both necessary and feasible.

0.20