NVIDIA open-sources Attribution for AI workload failure diagnosis
NVIDIA announced that Attribution, its workload-centric failure-attribution system for distributed AI infrastructure, is now open source. The system correlates workload-scoped logs, metrics, telemetry, and scheduler state to produce structured diagnoses with outcomes, attributed root causes, affected nodes, symptoms, and evidence trails; NVIDIA says deterministic rules make the results reproducible and customizable. The release includes the self-hosted NVIDIA Attribution Service, a Python reference orchestrator called NVIDIA Attribution Blueprint, and deployable rules with integration guidance.
The open-source release gives AI infrastructure teams a deployable way to investigate distributed workload failures and integrate structured diagnoses with existing health, recovery, and workload-restart systems.
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began.
In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence, inspect the diagnosis, and explore how attribution rules identify failure patterns.
NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state and returns a structured diagnosis: outcome, root cause, affected nodes, supporting symptoms, and evidence trail. Workload-centric means analyzing failures in the context of a workload rather than as isolated node events. Deterministic rules make the results reproducible and customizable, while structured output enables integration with existing health and recovery systems.
NVIDIA health and resilience software helps teams keep AI workloads running. NVIDIA Attribution explains why a workload failed and provides evidence to guide the next action. It complements readiness checks, continuous health monitoring, and the systems that repair infrastructure and restart workloads.
NVIDIA Attribution, the workload-centric failure-attribution system used across NVIDIA AI infrastructure, is open source.
#NVIDIA Attribution#NVIDIA Attribution Service#NVIDIA Attribution Blueprint