Aiconfig Online Evals
launchdarkly/launchdarkly-github-agent/skills/aiconfig-online-evals
ai-mlOfficial
Official Provider SkillView repo
Attach judges to AI Config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.
Files1 files
SKILL.md455 lines
Loading editor…
Install
RecommendedOne command — your agent picks it up automatically.
Select an AI agent above to see the install command.
or
Manual Install
More stepsDownload the file and paste it into your agent's system prompt.
Skill details
Versionv1.0.0
AuthorLaunchDarkly
Categoryai-ml
Skill IDlaunchdarkly/launchdarkly-github-agent/skills/aiconfig-online-evals
Related skills
Aiconfig Agent GraphsCreate and manage agent graphs — directed graphs of AI Configs connected by edges with handoff logic. Use when building multi-agent workflows where configs route to each other.Aiconfig Ai MetricsInstrument an existing codebase with LaunchDarkly AI Config tracking. Walks the four-tier ladder (managed runner → provider package → custom extractor + trackMetricsOf → raw manual) and picks the lowest-ceremony option that still captures duration, tokens, and success/error.Aiconfig CreateCreate and configure AI Configs in LaunchDarkly. Helps you choose between agent vs completion mode, create the config, add variations with models and prompts, and verify the setup.Aiconfig Custom MetricsCreate, track, retrieve, update, and delete custom business metrics for AI Configs. Covers full lifecycle: define metric kinds via API, emit events via SDK, and query results.Aiconfig MigrateMigrate an application with hardcoded LLM prompts to a full LaunchDarkly AI Configs implementation in five stages: audit the code, wrap the call, move the tools, add tracking, attach evaluators. Use when the user wants to externalize model/prompt configuration, move from direct provider calls (OpenAI, Anthropic, Bedrock, Gemini, Strands) to a managed AI Config, or stage a full hardcoded-to-LaunchDarkly migration.Aiconfig ProjectsGuide for setting up LaunchDarkly projects in your codebase. Helps you assess your stack, choose the right approach, and integrate project management that makes sense for your architecture.