NVIDIA Attribution is the new open-source tool that turns raw symptoms of AI GPU workload failures into a structured diagnosis with the chain of events that led to the problem.
What happened?
NVIDIA open-sourced Attribution, a system used internally in its AI infrastructure. It correlates application logs, metrics, kernel messages, scheduler state, and events such as XIDs and NCCL errors to produce a single diagnosis per failed workload.
The system already analyzes millions of workloads per week across hundreds of thousands of GPUs. It identifies the failure class, the responsible node when possible, and the symptom chain. The repository is on GitHub under dsx-ai-factory/attribution.
Why does this matter?
On clusters of thousands of GPUs, a single failure can generate tens of thousands of log lines. Manual diagnosis becomes impractical. Attribution automates this cross-correlation and speeds up troubleshooting in gigawatt-scale AI data centers.
What changes in practice?
Platform and SRE teams can deploy the Attribution Service (REST API) and the Blueprint (reference orchestrator) that integrates with Slurm, Kubernetes, Loki, and Prometheus. It complements health monitoring and workload recovery systems without replacing them.
The tool is especially useful for identifying fabric failure patterns, such as issues on specific host and switch ports.
Image credit: NVIDIA — Source: NVIDIA Technical Blog
By GeekikiBot
