英伟达开源 AI 工作负载故障归因系统
英伟达宣布，用于诊断分布式 AI 基础设施工作负载故障的 Attribution 系统现已开源。该系统关联工作范围内的日志、指标、遥测数据和调度器状态，输出结果、根因、受影响节点、相关症状及证据链；英伟达称，确定性规则可让诊断结果可复现、可定制。此次发布包括可自托管的 NVIDIA Attribution Service、Python 参考编排器 NVIDIA Attribution Blueprint、可部署规则和集成指南。
此次开源为 AI 基础设施团队提供了可用于排查分布式工作负载故障的部署方案，并可将结构化诊断接入现有的健康监控、恢复和工作负载重启系统。
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began.
In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence, inspect the diagnosis, and explore how attribution rules identify failure patterns.
NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state and returns a structured diagnosis: outcome, root cause, affected nodes, supporting symptoms, and evidence trail. Workload-centric means analyzing failures in the context of a workload rather than as isolated node events. Deterministic rules make the results reproducible and customizable, while structured output enables integration with existing health and recovery systems.
NVIDIA health and resilience software helps teams keep AI workloads running. NVIDIA Attribution explains why a workload failed and provides evidence to guide the next action. It complements readiness checks, continuous health monitoring, and the systems that repair infrastructure and restart workloads.
NVIDIA Attribution, the workload-centric failure-attribution system used across NVIDIA AI infrastructure, is open source.
#NVIDIA Attribution#NVIDIA Attribution Service#NVIDIA Attribution Blueprint