Agent Observability Replay Trace
datadog-labs/agent-skills/agent-observability/agent-observability-replay-trace
ai-mlOfficial
Official Provider SkillView repo
Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace <trace-id> [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace <id> with <change>"; pasting a
Files4 files
SKILL.md197 lines
Loading editor…
Install
RecommendedOne command — your agent picks it up automatically.
Select an AI agent above to see the install command.
or
Manual Install
More stepsDownload the archive and add the files to your project manually.
Skill details
Versionv1.0.0
AuthorDatadog Labs
Categoryai-ml
Skill IDdatadog-labs/agent-skills/agent-observability/agent-observability-replay-trace
Files4 files
Related skills
Agent InstallInstall the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.Agent InstallInstall the Datadog Agent on Linux hosts via SSH with Single Step Instrumentation (SSI) enabled — SSI automatically instruments applications for APM without code changes. Only use if no agent is installed yet.Agent Observability Auto ExperimentRun an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says "run an auto experiment", "hill-climb this code", "iteratively improve X and measure the delta", "optimize this prompt/file against my traces", "autoAgent Observability Build Eval From AnnotationsFit a Datadog LLM-Obs evaluator to human labels. Takes an annotation queue, works out where in the trace the labelled property actually lives, drafts an LLM-judge that predicts the human label, scores that judge against the already-labelled rows with a metric agreed with the user, then hill-climbs it — inspect the errors, make one focused change, re-score, keep it only if it beats the best — for a bounded number of iterations, and finally publishes the winner to Datadog as a DISABLED evaluator (Agent Observability Eval BootstrapBootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge confiAgent Observability Eval PipelineEnd-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `ag