analysisOfficial
Official Provider SkillView repo
Closed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs /log-analysis and /fr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close
Files1 files
SKILL.md418 lines
Loading editor…
Install
RecommendedOne command — your agent picks it up automatically.
Select an AI agent above to see the install command.
or
Manual Install
More stepsDownload the file and paste it into your agent's system prompt.
Skill details
Versionv1.0.0
AuthorNVIDIA
Categoryanalysis
Skill IDNVIDIA/nvidia-resiliency-ext/src/nvidia_resiliency_ext/skills/nvrx-attr/fault-injection-loop
Related skills
Fr AnalysisAnalyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.Log AnalysisAnalyze a SLURM job log file for failure root-cause attribution and restart decisions using the packaged NVRx restart-agent. Use when you have a SLURM training job log and need to determine why the job failed and whether it should be restarted. Performs deterministic evidence extraction plus optional LLM enrichment.Nvrx AttrOrchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.Nemoclaw Maintainer Analyze Pr Value StreamAnalyze one NemoClaw PR lifetime to identify contributor, review, and automation delays or compare it with a latency target.Nemoclaw Maintainer Pr ComparatorCompare competing NemoClaw PRs for one issue and recommend a merge or salvage candidate from review evidence.Perf AnalysisPerformance analysis coordination workflow. Guides profiling delegation, bottleneck classification (compute/memory/launch/communication/sync), and structured report generation. Use when the user asks to analyze performance, profile a workload, check MFU/SOL, or diagnose bottlenecks.