Çoklu dosya
ai-mlResmi
Resmi Sağlayıcı Skill'iView repo
Analyze a SLURM job log file for failure root-cause attribution and restart decisions using NVRxLogAnalyzer. Use when you have a SLURM training job log and need to determine why the job failed and whether it should be restarted. Performs per-cycle chunking, fast-path pattern matching, and LLM-based classification.
Dosyalar2 dosya
SKILL.md117 satır
Loading editor…
Kurulum
ÖnerilenTek komut — ajanınız otomatik olarak devreye alır.
Kurulum komutunu görmek için yukarıdan bir AI aracı seçin.
veya
Manuel Kurulum
Daha fazla adımArşivi indirin ve dosyaları projenize manuel olarak ekleyin.
Skill detayları
Versiyonv1.0.0
YazarNVIDIA
Kategoriai-ml
Skill IDNVIDIA/nvidia-resiliency-ext/src/nvidia_resiliency_ext/skills/nvrx-attr/log-analysis
Dosyalar2 dosya
İlgili skill'ler
Fault Injection LoopClosed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs /log-analysis and /fr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close Fr AnalysisAnalyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.Nvrx AttrOrchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.Nemoclaw Maintainer Cross Issue SweepScans other open issues to find ones a given PR may also fix or accidentally break. Outputs adjacent-fix opportunities and contradiction risks with file:line evidence. Use when reviewing a PR to discover bundling opportunities or downstream impact across the issue queue.Nemoclaw Maintainer EveningRuns the end-of-day maintainer handoff for NemoClaw. Checks version target progress, records stragglers for an automatic post-tag bump, generates a QA handoff summary, and cuts the release tag. Use at the end of the workday. Trigger keywords - evening, end of day, EOD, wrap up, ship it, cut tag, handoff, done for the day.Nemoclaw Maintainer MorningRuns the morning maintainer standup for NemoClaw. Triages the backlog, determines the day's target version, labels selected items, surfaces stragglers from previous versions, and outputs the daily plan. Use at the start of the workday. Trigger keywords - morning, standup, start of day, daily plan, what are we shipping today.