The tool correlates logs, metrics, telemetry, and scheduler state into a reproducible root-cause report.

NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state to return a structured diagnosis. The diagnosis includes the outcome, root cause, affected nodes, supporting symptoms, and an evidence trail. NVIDIA offers the tool as open source, and teams can run it on their own clusters.

NVIDIA built Attribution for distributed AI workloads, where one GPU or fabric event can surface as NCCL errors on other ranks. The system uses the Attribution Service to expose a REST API and the Attribution Blueprint as a Python orchestrator for schedulers and telemetry sources. The Blueprint connects Slurm, Kueue, static inputs, Loki, local files, Kubernetes pod logs, and PromQL metrics.

Operators can analyze failures in workload context rather than as isolated node events. Deterministic rules keep diagnoses reproducible and let teams customize failure patterns for their own clusters. A worked example identifies an NVLink signal integrity issue on one node as the root cause and keeps downstream NCCL errors across ranks as supporting symptoms. The structured result enables system integration.

Developers can clone the NVIDIA Attribution repository to access source code and sample workloads. A quickstart runs the sample workflow and shows diagnoses in a local dashboard. The Blueprint configuration reference lets teams connect their own scheduler and telemetry backends. NVIDIA reports production use across AI infrastructure, where the system analyzes millions of workloads weekly on hundreds of thousands of GPUs.

What matters

• NVIDIA released an open-source attribution tool that maps workload failures to root causes.
• Operators gain reproducible diagnoses across clusters instead of scattered node-level error signals.
• Teams can connect their own schedulers and telemetry backends through the Blueprint configuration.

Why it matters

Teams can connect their own schedulers and telemetry backends through the Blueprint configuration.

This GenAI News article was prepared in original wording using reporting and materials published by NVIDIA Developer Blog. Source reference: https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/.

Drafted by the GenAI News review pipeline.