CS Paper Radar

FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale

Shripad Nadgowda, Gaurav Mitra, Sing Sing Ma, Vipul Chaskar, Pradipta Ghosh, Canturk Isci, Onkar Deshpande, Edgardo Aviles-Lopez, Ryan Feng, Ján Ondrušek

cs.SE cs.DC ·submitted 2026-11-11 03:15 UTC · v1

Abstract

Pre-production changes are usually tested in isolated environments to prevent their side effects from reaching production. Some record-and-replay platforms allow a production-connected mode: candidate changes receive recorded production requests while selected downstream calls reach production dependencies. This improves fidelity before deployment, but it also introduces reliability risks. At hyperscale, experimental traffic can fan out across shared services, trigger latent defects, or send malformed input to systems serving user requests. In this paper, we show how production-connected experimentation can operate reliably in a large-scale recommendation serving system. To contain downstream effects without sacrificing realistic dependency behavior, the platform uses three complementary guardrails. Isolated tiers redirect replay traffic away from shared dependencies where replication is practical. Dependency monitors observe downstream health and experiment-attributed errors and terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity. Over 90 days, these tiers experienced more than 200,000 abnormal worker exits without directly affecting production-serving workers. During a separate three-month window, dependency monitors cancelled approximately 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering also reduced code-review experiment executions by 25%. These results show that high-fidelity, production-connected experimentation can operate at scale when downstream effects are isolated where possible, bounded where necessary, and continuously monitored and attributed.

Comment
14 pages, 8 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Sustaining In-Network Computation in Satellite Networks: An Architecture for Semantic Continuity

cs.NI cs.DC ·2026-11-14 ·Zhiyuan Ren, Wenchi Cheng
position similarity 0.61
Abstract · 8 pages, 5 figures. Submitted to IEEE Wireless Communications

Satellite computations often outlast the paths and providers initially serving them. We argue that in-network computing should organize changing execution opportunities around one continuing logical computation: workers execute ready operations, while consumers recognize which results already contribute. We present an architecture separating logical computation, stream context, and physical binding, and show in wildfire assessment how retained observations support recomputation while reachable association history stays in place. A 300-node Linux prototype examines placement, dependency assembly, and continuation. Across 120 images, stage-wise placement delivers 82.08% of positions within deadlines versus 76.06% for single-node placement, while traces separate blocked output delivery from dependencies split across providers. Across 50 observations with 20-node isolation, Confirmation-Coordinated Rebinding (CCR) coordinates replanning, source replay, cross-attempt acceptance, and in-band confirmation to complete all 50 observations with verified outputs using seven replays and 13 ignored duplicate outputs.

0.34

AtomicFaaS: Offloading Side-Effect Compensation to the Serverless Platform

cs.DC ·2026-11-04 ·Jonas Heisterberg, Sebastian Koch, Trever Schirmer, David Bermbach
similarity 0.61
Abstract · Accepted for the Second International Workshop on AI and Serverless (WoAIS '26)

Function-as-a-Service (FaaS) simplifies distributed application development by offloading operational concerns to the platform, but developers remain responsible for recovering from failures. Retries require idempotent functions, which means that developers have to write complex, error-prone code that undoes or skips the already completed operations of a failed function. To address this, we propose AtomicFaaS, a platform extension that provides semantic atomicity for the external side effects of a function invocation, whether the function fails or its instance crashes, without developer-written compensation code. AtomicFaaS intercepts outbound calls and records their effects in a fault-tolerant log, from which the platform compensates leaked effects outside the failed instance. Compared with a manually implemented saga, AtomicFaaS eliminates compensation code and leaves no compensatable effects after any tested failure, while adding ~1 ms per intercepted call. Under load, it lowers the sustainable throughput of CPU-bound functions, while its latency overhead becomes negligible once calls to external services dominate the invocation time.

0.25

State Without a Landlord: An Architecture Proposal for Peer-to-Peer Replication of Durable Workflow State

cs.CR cs.DC ·2026-09-15 ·Luca Maraschi, Matteo Collina
similarity 0.61
Abstract · 51 pages, 8 figures

Durable-execution frameworks commonly journal execution steps of a long-running function and reconstruct state by deterministic replay; the journal is therefore the workflow's authoritative state, and in current deployments it typically resides with whichever provider hosts the run. This paper examines what changes if the journal is instead an authenticated, append-only log replicated among the parties to the workflow. We propose an architecture in which runs are chains of single-writer epoch cores; executor succession is governed by quorum-finalized closure certificates; durability, timeout, and checkpoint decisions are carried by explicit certificates rather than implicit trust; and large payloads travel a separate content-addressed distribution plane. A threat model separates what signatures establish (authorship and order) from what they do not (truth of recorded effects). We state the design as a set of typed artifacts and invariants, identify which claims are established by existing systems and which are proposals requiring validation, and define the prototype experiments, including specific adversarial cases, that would validate or falsify the proposal's central mechanisms.

0.38

RESTberus: Trace-Aware Benchmarking of Automated REST API Testing Tools Across Microservice Dependencies

cs.SE ·2026-11-03 ·John Toniutti, Massimiliano Baldo, Marino Miculan, Vincenzo Riccio
similarity 0.58
Abstract

Existing REST API testing benchmarks focus on isolated services, overlooking failures arising from interactions with downstream dependencies. In microservice systems, response bodies and entry-point operations alone cannot reliably distinguish failures, compromising tool comparisons. We present RESTberus, a Kubernetes-based benchmarking platform for black-box REST API testing tools on deployed microservice systems. By complementing API-level evidence with distributed traces, RESTberus can distinguish failures manifesting in the target service from those manifesting in downstream dependencies. Across six tools on two microservice benchmarks, our trace-aware deduplication strategy achieves 87.1% accuracy, outperforming baselines from prior work by 10-13 percentage points. Our empirical study shows that all tools expose downstream failures, accounting for 11.7% of estimated unique failures on average, which isolated-service benchmarks cannot expose. On single-service benchmarks, RESTberus produces statistically equivalent results to RESTgym, demonstrating that integration-level assessments can preserve compatibility with existing tools and comparability with prior evaluations.

0.27

FaaSpot: Fault-Tolerant Serverless Workflow Execution on Spot Instances

cs.DC ·2026-10-19 ·Bandhan Patel, Sushruth Harsha, Kwangsung Oh, In Kee Kim
publishedcode/data similarity 0.58
Abstract · 14 pages, 12 figures, 7 tables. To appear in the 27th International Middleware Conference (Middleware '26), December 14-18, 2026, Tarragona,…

Serverless computing is increasingly used to run complex DAG workflows across diverse applications, and individual function execution times have grown to support them, pushing workflow makespans to hours. Open-source serverless platforms, deployed on a pool of VMs, typically use on-demand instances, and their cost increases with the makespan. Spot instances are far cheaper, but a preemption is likely before the workflow finishes. A preemption at a fan-out node or a task feeding a fan-in node stalls every parallel branch, cascading a single preemption into a workflow-wide failure. Prior systems address this challenge by migrating high-impact tasks to reliable, non-spot resources, sacrificing spot's cost advantage. We present FaaSpot, a serverless workflow execution system that runs every workflow task on spot instances with no on-demand fallback. Our key insight is that cascade-prone positions are determined by the DAG's topology and can be identified before execution. Therefore, FaaSpot analyzes the workflow DAG once at submission time and classifies each task by the impact of its preemption. Selective checkpointing, survival-aware scaling, and lineage-based retry then act on this single classification, with no runtime coordination. We evaluate FaaSpot on a trace-driven spot emulator with eight DAG workflows against four state-of-the-art baselines. FaaSpot completes 89.03% of runs at $0.018 per successful run, 21pp (percentage points) above the average baseline success rate and at a lower cost per success than every baseline, while reducing mean latency by 2.3x.

0.52

Enabling Differentiated QoS Degradation for Replicated Databases under Failures

cs.DB cs.DC ·2026-08-12 ·Belkis Djeffal, Pierre Bourhis, Romain Rouvoy
published similarity 0.57
Abstract

Elasticity is commonly presented as the default response to capacity loss after failures, since replacement replicas can compensate for failed nodes and restore pre-incident service levels. Replacement capacity entails both delay and additional resource commitment, as replicas must be provisioned and synchronized before they can serve traffic. Under fixed budgets or constrained operating conditions, capacity restoration cannot be treated as the immediate recovery path. Failure handling must instead define how the service continues while capacity remains reduced. When the service exposes differentiated service levels, capacity loss cannot be handled uniformly. Degradation becomes part of the service behavior, requiring explicit control over how reduced capacity affects each class without erasing the intended differentiation. We study differentiated QoS degradation in replicated database services with service-class-aware sessions. We present a repair-to-target policy, implemented in PLB, a PostgreSQL JDBC middleware load balancer for service-class-aware routing. When fail-stop failures remove part of the available capacity, PLB updates the role assignment of healthy replicas into Premium, Mixed, and Freemium roles. This keeps the replica pool shared while ensuring that new session assignments continue to reflect the service class. We evaluate PLB under single and cascading replica failures across two deployment strategies: isolated perclass replica pools and shared, priority-agnostic routing. The results show that PLB improves median Premium goodput retention by 26-28 percentage points under a Premium-side fault, achieves more than 2x higher Premium goodput in the most severe cascading-failure phase, and reduces Premium p95 latency by 18.2% relative to shared round-robin.

0.29

AOS-Bench: Can LLM Agents Manage Dynamic Workloads?

cs.AI ·2026-10-23 ·Erle Zhu, Genrui Zhang, Keyang Hu, Mingyu Su, Hongning Wang
similarity 0.57
Abstract

Coding agents are moving from handling one request at a time to serving as persistent assistants to which users hand a stream of work: a build, a data-processing job and a bug fix may all be pending on the same machine at once. Such an agent must decide when to take on new tasks, how to delegate and overlap them, and how to stay within the machine's CPU and memory. Existing benchmarks cannot measure this, because they evaluate each task in isolation. We introduce AOS-Bench, which evaluates an agent system as a workload manager: verified tasks arrive over time and must be delivered under shared CPU and memory limits, and the same tasks run one at a time serve as a per-system reference. Across five model--harness pairs, isolated task success does not carry over evenly. On heavy workloads every system loses 13 to 23 points. On medium workloads, systems span 11 points in isolated success but 26 points in stream success, and two systems 0.6 points apart in isolation deliver 54\% and 68\% of the same tasks. Execution-policy interventions trace much of this gap to task intake: serial dispatch leaves 73\% of heavy-workload tasks unclaimed. Yet more elaborate management does not necessarily help. A fixed concurrency of two comes within four points of the best agent-managed policy on both medium and heavy workloads, and under every policy median CPU use stays below 30\% of the provisioned cores while tasks wait. These results identify workload management as a distinct capability for agents, and AOS-Bench as a way to measure it.

0.17

Retry Amplification in Distributed Systems: A Systematic Analysis of Retry Policies and Their Role in Cascading Failures

cs.SE ·2026-08-26 ·Rishabh Mehan, Jasmit Kaur Saluja
publishedcode/data similarity 0.57
Abstract

Retry mechanisms are a standard component of resilient distributed systems, but their collective behavior, when every tier in a call path retries concurrently, is less well understood than the per-client guidance that produced them. This paper introduces the retry amplification factor (RAF), a metric quantifying the additional request volume that retry policies generate during partial failures. In a study of 200 open-source Python microservice projects, explicit retry logic is detected in 11.5%, and an audit of our own false negatives places true prevalence near 41%. Among the projects detected, 60.9% contain at least one configuration without backoff, and after manual verification exactly one of 113 production configurations randomizes its delay. We then evaluate these policies in simulation (n = 100 trials per strategy). Under correlated failures, a naive standard retry policy reduces the success rate from 55.4% to 41.5% relative to performing no retries at all. We catalog five recurring anti-patterns, propose Adaptive Retry Budgeting (ARB), and show that budget-constrained retries maintain success rates close to the no-retry baseline while still recovering from transient faults. These results indicate that retry behavior should be designed as a system-level property rather than configured locally at each call site.

0.66