CS Paper Radar

Service Health Engineering for Distributed Systems

Siva Rama Krishna Varma Bayyavarapu

cs.SE ·submitted 2026-09-07 22:03 UTC · v1
published

Abstract

Distributed systems support many critical business workflows, but service health is often judged through component dashboards rather than through end-to-end user outcomes. This article presents service health engineering as a practical reliability discipline that connects telemetry, workflow completion, dependency behavior, operational readiness, and recovery validation. Using a document approval workflow as a running example, it describes how service promises, service-level indicators and objectives, watchdogs, incident measures, resiliency testing, and weekly service-health reviews can reveal silent failures and stranded asynchronous work. It also presents a human-reviewed, AI-assisted reporting architecture for assembling service-health evidence without making AI an autonomous decision-maker. The approach brings established reliability practices together around whether user journeys complete as promised.

Comment
9 pages, 1 figure. Accepted for publication in IEEE Reliability Magazine
DOI
10.1109/MRL.2026.3727928

Closest papers

Nearest neighbours by meaning across the whole index.

Making Deployments Safe at Meta: Health Checks for Continuous Change-Safety

cs.SE cs.AI ·2026-08-20 ·Prakash KL, Anton Korenkov, Uttam Thakore, Christopher Hegre
top venue similarity 0.71
Abstract · 6 pages, 4 figures, Accepted to ISSRE 2026 Industry Track

Continuous deployment to large scale production systems creates a tension between release velocity and reliability. Every change is a potential reliability incident, yet every delay is a missed opportunity. This paper describes the deployment time health check infrastructure that Meta uses to mediate this tension across thousands of heterogeneous services. We summarize the architecture of this prevention based distributed system's service called Service Health Checker, explain how check authors compose templated metric queries, thresholds, and workflow predicates; and discuss how the system is integrated with tiered and phased rollouts so that regressions trigger automatic rollback. We then describe the operational problems that emerged at scale, such as noise, alert fatigue, drift, and uncovered regressions, and the program of measurement, tooling, and improved defaults we deployed to address them. We close with lessons learned from years of operating deployment health checks at Meta, and the directions we are exploring next, including AI assisted health check tuning. Index Terms: deployment safety, continuous deployment, monitoring, software reliability, release engineering, software reliability engineering, AIOps, anomaly detection

0.36

Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

cs.CL ·2026-08-31 ·Junyi Yao, Baichuan Li, Zihao Zheng, Jiayu Long
highly cited similarity 0.66
Abstract · 4 pages, 3 tables

Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging, and care coordination. Yet many evaluations remain limited to static medical question answering or one-shot generation, under-representing longitudinal state, interruptions, and human handoffs. We introduce an episode-level evaluation protocol for healthcare NLP agents. The protocol separates evidence across model, agent, and simulated-workflow behavior; specifies a five-field episode schema; and defines annotation and scoring for state continuity, evidence traceability, and escalation decisions. It is instantiated as four task templates: documentation update, evidence retrieval, patient messaging, and triage handoff. The protocol does not claim to measure clinical outcomes or deployment value. Instead, it supplies a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.

0.33

Diviner: Agentic RCA with Human-Governed Adaptation for Large-Scale Infrastructure

cs.AI cs.MA ·2026-10-15 ·Pradipta Ghosh, Gaurav Mitra, Shripad Nadgowda, Akshil Shah, Junyu Liu +5 more
similarity 0.65
Abstract

Root-cause analysis (RCA) in large-scale distributed systems is challenging because a single symptom can cross service, data, configuration, compute, and organizational boundaries. Responders must assemble incomplete evidence and coordinate experts who hold different parts of the causal path. Generic language-model agents do not solve this problem through tool access alone: they can use stale or mistimed context, interpret failed queries as absent symptoms, report unsupported system state, and overload observability services designed for human-paced use. We present Diviner (Domain-Informed Verifiable Incident Navigation through Evidence and Review), an agentic RCA system operating at Meta and organized around a four-stage reliability lifecycle. Encode combines a large corpus of reviewed incidents with service-owner expertise to produce scoped, versioned skills, workflows, typed tools, and an incident knowledge graph. Execute uses an adaptive Lead and context-isolated workers to apply relevant knowledge to point-in-time evidence and maintain evidence-linked hypotheses. Evaluate compares saved investigation trajectories with responder-reviewed outcomes to measure exact cause, useful direction, and Groundedness. Adapt uses recurring gaps and on-call feedback to derive focused operational- knowledge updates, admitting them only after replay, quality checks, and owner approval.

0.18

Developing JEEVHITAA: Designing and Evaluating a Human-Centered AI Ecosystem for Collective Care

cs.HC cs.ET ·2026-11-15 ·Shyama Sastha Krishnamoorthy Srinivasan, Harsh Pala, Shahnawaz, Mohan Kumar, Pushpendra Singh
hot direction similarity 0.63
Abstract · 38 pages, 5 figures, 7 tables, Preprint of a Submitted manuscript

Health management is often a collective activity, involving family members and close relations in monitoring, interpreting, and coordinating care. Yet, most digital health technologies remain individual-centric, offering limited support for the distributed responsibilities and negotiated information sharing that characterize collective care. We present JEEVHITAA, a human-centered AI ecosystem designed to address this gap. Drawing on existing literature on collective health practices, we identify key requirements and translate them into a canonical ontology and a versioned data schema that underpin a mobile ecosystem integrating wearable data, role-aware sharing, granular access controls, auditable histories, and AI-assisted health summarization and verification. We first evaluated core system primitives using 1,000 ontology-driven synthetic care circles to test data visibility and information-processing workflows. We then conducted a controlled lab study with 13 subject--care-actor dyads (26 participants), combining task-based interactions, semi-structured interviews, the System Usability Scale, and component-level usability assessments. Findings indicate positive perceptions of usability and highlight the value of granular sharing and transparent information provenance. They also expose tensions between reciprocal transparency and surveillance, and between AI-mediated assistance and potentially intrusive notifications. We discuss implications for designing accountable, context-sensitive technologies that support collective care.

0.49

Towards Multi-Method Evaluation of AI in Health: Integrating Benchmarks, Autoraters, and Expert Evaluation

cs.CY ·2026-11-03 ·Sarah Montgomery-Taylor, Hamsa Subramaniam, Rob Brisk, Liam Foster, Jack Ching +28 more
similarity 0.61
Abstract · 16 pages, 1 figure, 2 tables

General-purpose generative AI (GenAI) models are being rapidly adopted in healthcare, but they are challenging to assess with traditional evaluation frameworks. These conceptual, methodological, and practical challenges parallel decades of work in evaluating clinician and health system quality and safety. We describe the adaptation of methods from several fields, combined with emerging technical approaches, into a structured, multi-method evaluation framework designed to give well-rounded assessments of GenAI in health at scale. The approach first anchors evaluation to Critical User Journeys (CUJs) to help prioritize out of a large choice space, and maps these to a phased AI system lifecycle (base model development, product development, and post-deployment monitoring), since evaluation at each stage requires different methodological approaches. In this context, the approach then integrates three distinct evaluation methods (closed-question benchmarks, LLM-as-a-judge autoraters, and expert human review) across three core domains: accuracy, safety, and usefulness. Taken together, this approach provides a strategic framework for bridging the gap between benchmarking results and performance in the real world with the goal of supporting decision-making and accountability across development, procurement, and deployment for product teams, health systems, and regulators.

0.14

READY or Not: Reliable Enterprise Agent Deployment

cs.AI ·2026-09-02 ·Veronica Chatrath, Bryan Zhu, Jingxuan Fan, George Pu, Soham Dinesh Tiwari +13 more
similarity 0.61
Abstract

An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.

0.41

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

cs.AI ·2026-10-11 ·Rui Tang, Hongqiu Wang
benchmarkcode/data similarity 0.60
Abstract

Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions under which model-centric systems may lose context, misuse tools, fail to verify consequential transitions, or cross authorization boundaries. Existing medical-agent benchmarks primarily emphasize terminal task outcomes, making it difficult to determine whether execution failures arise from the agent model, the runtime harness, or their interaction. We present the Medical Harness Protocol, an outcome-preserving framework that independently evaluates seven healthcare-specific process dimensions: Care Action Fulfillment, Clinical Resource Competence, Patient-Evidence Alignment, Clinical Workflow Continuity, Clinical Auditability, Closed-Loop Verification, and Clinical Conduct Boundaries. A normalized intervention trace further tracks whether runtime support is available, activated, delivered, adopted by the agent, and propagated into subsequent behavior. We instantiate the protocol as MedHarnessBench, comprising 342 executable tasks across FHIR/EHR operations, healthcare-administration interfaces, and multimodal clinical tools. Across more than 16,000 controlled task executions, MedHarness yields broad improvements in terminal task success, intermediate subtask completion, and healthcare-specific process quality. We further evaluate MedHarness against multiple general-purpose harnesses and use controlled single-fault perturbations to verify the sensitivity and discrimination of the seven-dimensional evaluation. These results demonstrate that healthcare-specific runtime support can substantially improve both task completion and execution quality, while revealing failures that terminal outcomes alone cannot capture. Code and resources are available at https://github.com/Tomry1114/Awesome-Agent-Harness-for-Healthcare.

0.20

Reliability Engineering for AI Systems: Challenges, Methods, and Directions

cs.AI stat.AP ·2026-09-28 ·Rong Pan, Yili Hong, Min Xie
benchmark similarity 0.60
Abstract

AI reliability concerns whether an AI system performs its intended function dependably over a stated period and under stated operating conditions, with clear evidence. As these systems become more autonomous, this function should include more than only a correct output. Retrieval, memory, tool use, permissions, human oversight, and interactions among systems must operate consistently and safely, and, for a generative AI, so must the reasoning process that produces the output. Average benchmark accuracy only measures system capability; it does not quantify this broader reliability claim. This paper adapts established reliability engineering methods, from failure definitions and operational envelopes to FMEA, accelerated testing, field monitoring, and reliability growth, to AI systems. A four-level diagnostic framework classifies failures as component, operational-loop, agentic-conduct, or network and governance failures. Test, evaluation, verification, and validation (TEVV), sequential monitoring, and FRACAS create and refresh evidence. The SMART framework provides statistical guidance for measurement, analysis, assessment, and test planning, and the NIST's AI risk management framework provides organizational guidance for governance, evaluation, monitoring, and mitigation. Three cases are discussed to illustrate these programs. It is shown that while the established reliability engineering principles can still provide a usable foundation, new measurements and safety guardrails are urgently needed as these systems are self-evolving at a fast pace.

0.14