Official Provider SkillView repo
Run HolmesGPT evals (tests/llm/fixtures/test_ask_holmes) against the current agent itself (agent-native). The agent must answer the prompts itself (no external wrapper) using its own tools, run before_test/after_test around each case, compare the answer to expected_output using the scoring rules here, and generate JSON + Markdown reports. This skill focuses exclusively on the agent-native flow. Supports bulk run (all tests) and individual run (specific test case). Supports filtering tests by tag
Files4 files
SKILL.md116 lines
Loading editor…
Install
RecommendedOne command — your agent picks it up automatically.
Select an AI agent above to see the install command.
or
Manual Install
More stepsDownload the archive and add the files to your project manually.
Skill details
Versionv1.0.0
AuthorAzure
Categoryai-ml
Skill IDAzure/AKS-Skills/evals/holmesgpt-eval
Files4 files
Related skills
Aks Automatic ReadinessAssess Kubernetes workloads and cluster configuration for AKS Automatic compatibility. Identifies incompatibilities, generates fixes, and guides migration from AKS Standard to AKS Automatic. WHEN: migrate to AKS Automatic, check AKS Automatic readiness, validate manifests for Automatic, assess cluster for Automatic compatibility, fix deployment for Automatic compatibility, identify AKS Automatic migration blockers, is my cluster ready for AKS Automatic. DO NOT USE FOR: creating a brand-new clustAks Cluster SetupMake the AKS-specific design decisions for a new production Azure Kubernetes Service (AKS) cluster — SKU (Automatic vs Standard), pod IP model (Azure CNI Overlay vs kubenet), API-server access, egress, identity, upgrades, node pools, and reliability — then delegate the actual provisioning to the Azure Skills deployment engine. WHEN: create AKS cluster, provision AKS environment, design AKS networking, choose AKS SKU, Day-0 AKS checklist, plan a production AKS cluster. DO NOT USE FOR: debugging aAks Cost OptimizationReduce Azure Kubernetes Service (AKS) spending: pod rightsizing, VPA-driven recommendations, cluster-autoscaler tuning, spot node pools, namespace-level cost visibility, and cost-anomaly detection. WHEN: rightsize pods, VPA recommendations, idle nodes, scale-down, autoscaler profile, spot nodes, cheaper compute, cost add-on, namespace cost breakdown, spending anomaly, 'my AKS bill is too high', 'is my app consuming what it requests', resource requests vs actual usage, over-provisioned workloads,Aks Gpu InferenceDay-2 operations for GPU and model-inference workloads on Azure Kubernetes Service (AKS): diagnose GPU pods stuck Pending, missing nvidia.com/gpu, CUDA/driver mismatches, model OOM on weight load, GPU vCPU-quota failures, KAITO (AI toolchain operator) Workspaces stuck not-ready, and GPU cost / scale-to-zero / spot eviction. WHEN: GPU pod Pending 'Insufficient nvidia.com/gpu', no nvidia.com/gpu on node, CUDA driver version insufficient, model OOMKilled loading weights, GPU node pool quota exceedeAks Known IssuesMatch exact AKS operation-failure signatures to documented causes and fixes. WHEN: an AKS create, scale, upgrade, node-image, or image-pull failure names VMCannotFitEphemeralOSDisk, LinkedAuthorizationFailed, NodePoolMcVersionIncompatible, 'NodeImageVersion is not accepted', SkuNotAvailable, ZonalAllocationFailed, OverconstrainedAllocationRequest, or an AKS vmssCSE/CSE nested signature: VMExtensionError_OutboundConnFail (exit 50), VMExtensionError_K8SAPIServerConnFail (exit 51), or VMExtensionErAks Network CapturePacket-level network evidence for AKS: run a bounded, distributed packet capture across nodes (filtered by IP, port, or tcpdump/BPF expression), and collect Azure network resources (NSG rules, route tables, firewall, VNET peering) when you need pcap-level proof of where traffic drops. Escalation tool for when logs and read-only checks are inconclusive. WHEN: capture packets on a node, take a pcap, tcpdump on AKS, prove where a packet is dropped, verify an NSG or route is blocking traffic at the