analysisResmi
Resmi Sağlayıcı Skill'iView repo
Closed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs /log-analysis and /fr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close
Dosyalar1 dosya
SKILL.md414 satır
Loading editor…
Kurulum
ÖnerilenTek komut — ajanınız otomatik olarak devreye alır.
Kurulum komutunu görmek için yukarıdan bir AI aracı seçin.
veya
Manuel Kurulum
Daha fazla adımDosyayı indirin ve ajanınızın sistem istemine yapıştırın.
Skill detayları
Versiyonv1.0.0
YazarNVIDIA
Kategorianalysis
Skill IDNVIDIA/nvidia-resiliency-ext/src/nvidia_resiliency_ext/skills/nvrx-attr/fault-injection-loop
İlgili skill'ler
Fr AnalysisAnalyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.Log AnalysisAnalyze a SLURM job log file for failure root-cause attribution and restart decisions using NVRxLogAnalyzer. Use when you have a SLURM training job log and need to determine why the job failed and whether it should be restarted. Performs per-cycle chunking, fast-path pattern matching, and LLM-based classification.Nvrx AttrOrchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.Perf AnalysisPerformance analysis coordination workflow. Guides profiling delegation, bottleneck classification (compute/memory/launch/communication/sync), and structured report generation. Use when the user asks to analyze performance, profile a workload, check MFU/SOL, or diagnose bottlenecks.Perf Nsight Compute AnalysisAnalyze ncu (NVIDIA Nsight Compute) profiling output: SOL% bottleneck classification, roofline analysis, occupancy diagnosis, memory hierarchy analysis, warp stall analysis, metric interpretation, and programmatic .ncu-rep report analysis. NOT for kernel writing or code generation, Nsight Systems (nsys), host-side profiling, or system-level profiling.Perf Nsight SystemsNsight Systems (nsys) CLI for system-level timeline profiling. Use when the user wants to run nsys profile, analyze .nsys-rep reports, use nsys stats/analyze/recipe commands, diagnose GPU idle time from timeline traces, or profile distributed training with NCCL overlap analysis. NOT for kernel-level metrics like SOL%, occupancy, or roofline (use perf-nsight-compute-analysis for ncu). NOT for writing or generating kernels. NOT for applying optimizations like CUDA Graphs.