Agent Observability Experiment Bootstrap
datadog-labs/agent-skills/agent-observability/agent-observability-experiment-bootstrap
ai-mlResmi
Resmi Sağlayıcı Skill'iView repo
Bootstrap a reproducible LLM Observability experiment through the Python ddtrace SDK or the Node dd-trace SDK. Use for experiment, dataset, evaluator, benchmark, regression, or LLM-as-a-judge scaffolding. The legacy Python invocation remains supported.
Dosyalar14 dosya
SKILL.md176 satır
Loading editor…
Kurulum
ÖnerilenTek komut — ajanınız otomatik olarak devreye alır.
Kurulum komutunu görmek için yukarıdan bir AI aracı seçin.
veya
Manuel Kurulum
Daha fazla adımArşivi indirin ve dosyaları projenize manuel olarak ekleyin.
Skill detayları
Versiyonv1.0.0
YazarDatadog Labs
Kategoriai-ml
Skill IDdatadog-labs/agent-skills/agent-observability/agent-observability-experiment-bootstrap
Dosyalar14 dosya
İlgili skill'ler
Agent InstallInstall the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.Agent InstallInstall the Datadog Agent on Linux hosts via SSH with Single Step Instrumentation (SSI) enabled — SSI automatically instruments applications for APM without code changes. Only use if no agent is installed yet.Agent Observability Auto ExperimentRun an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says "run an auto experiment", "hill-climb this code", "iteratively improve X and measure the delta", "optimize this prompt/file against my traces", "autoAgent Observability Build Eval From AnnotationsFit a Datadog LLM-Obs evaluator to human labels. Takes an annotation queue, works out where in the trace the labelled property actually lives, drafts an LLM-judge that predicts the human label, scores that judge against the already-labelled rows with a metric agreed with the user, then hill-climbs it — inspect the errors, make one focused change, re-score, keep it only if it beats the best — for a bounded number of iterations, and finally publishes the winner to Datadog as a DISABLED evaluator (Agent Observability Eval BootstrapBootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge confiAgent Observability Eval PipelineEnd-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `ag