--- title: "launchdarkly/ai-tooling" description: "LaunchDarkly's official AI tooling" source: https://github.com/launchdarkly/ai-tooling ref: main license: null licenseName: "Other" canonical: https://skillsdocs.com/launchdarkly/ai-tooling base: https://github.com/launchdarkly/ai-tooling/blob/main/ chapters: 50 inlined: 50 withheld: 0 words: 48391 updated: 2026-08-09T11:44:34Z generator: "Skills Docs" --- > **launchdarkly/ai-tooling** — every Agent Skill in this repository, inlined verbatim. > > Canonical HTML: https://skillsdocs.com/launchdarkly/ai-tooling > Per-chapter Markdown: https://skillsdocs.com/launchdarkly/ai-tooling/.md > Machine manifest: https://skillsdocs.com/launchdarkly/ai-tooling/.well-known/agent-skills/index.json > JSON: https://skillsdocs.com/api/v1/books/launchdarkly/ai-tooling > Install: `npx skills add launchdarkly/ai-tooling` > Upstream: https://github.com/launchdarkly/ai-tooling @ `main` > Licence: Other > > Content is mirrored from GitHub and © its authors, served unmodified. Takedown: https://github.com/kyleledbetter/skillsdocs/issues/new?labels=takedown&title=Takedown+request # launchdarkly/ai-tooling LaunchDarkly's official AI tooling - **Chapters:** 50 - **Inlined:** 50 (licence detected) - **Words:** 48,391 - **Reading time:** 232 min - **Stars:** 25 ## Table of contents 1. [agent-graphs](https://skillsdocs.com/launchdarkly/ai-tooling/agent-graphs.md) — Create and manage agent graphs — directed graphs of configs connected by edges with handoff logic. Use when building multi-agent workflows where configs route… 2. [aiconfig-agent-graphs](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-agent-graphs.md) — DEPRECATED redirect — this skill was renamed to agent-graphs. Do not use this skill; invoke agent-graphs instead. Kept only so old references to aiconfig-agent… 3. [aiconfig-ai-metrics](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-ai-metrics.md) — DEPRECATED redirect — this skill was renamed to built-in-metrics. Do not use this skill; invoke built-in-metrics instead. Kept only so old references to aiconf… 4. [aiconfig-create](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-create.md) — DEPRECATED redirect — this skill was renamed to configs-create. Do not use this skill; invoke configs-create instead. Kept only so old references to aiconfig-c… 5. [aiconfig-custom-metrics](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-custom-metrics.md) — DEPRECATED redirect — this skill was renamed to custom-metrics. Do not use this skill; invoke custom-metrics instead. Kept only so old references to aiconfig-c… 6. [aiconfig-migrate](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-migrate.md) — DEPRECATED redirect — this skill was renamed to migrate. Do not use this skill; invoke migrate instead. Kept only so old references to aiconfig-migrate still p… 7. [aiconfig-online-evals](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-online-evals.md) — DEPRECATED redirect — this skill was renamed to online-evals. Do not use this skill; invoke online-evals instead. Kept only so old references to aiconfig-onlin… 8. [aiconfig-projects](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-projects.md) — DEPRECATED redirect — this skill was renamed to projects. Do not use this skill; invoke projects instead. Kept only so old references to aiconfig-projects stil… 9. [aiconfig-snippets](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-snippets.md) — DEPRECATED redirect — this skill was renamed to snippets. Do not use this skill; invoke snippets instead. Kept only so old references to aiconfig-snippets stil… 10. [aiconfig-targeting](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-targeting.md) — DEPRECATED redirect — this skill was renamed to configs-targeting. Do not use this skill; invoke configs-targeting instead. Kept only so old references to aico… 11. [aiconfig-tools](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-tools.md) — DEPRECATED redirect — this skill was renamed to tools. Do not use this skill; invoke tools instead. Kept only so old references to aiconfig-tools still point u… 12. [aiconfig-update](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-update.md) — DEPRECATED redirect — this skill was renamed to configs-update. Do not use this skill; invoke configs-update instead. Kept only so old references to aiconfig-u… 13. [aiconfig-variations](https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-variations.md) — DEPRECATED redirect — this skill was renamed to configs-variations. Do not use this skill; invoke configs-variations instead. Kept only so old references to ai… 14. [built-in-metrics](https://skillsdocs.com/launchdarkly/ai-tooling/built-in-metrics.md) — Instrument an existing codebase with LaunchDarkly config tracking. Walks the four-tier ladder (managed runner → provider package → custom extractor + trackMetr… 15. [configs-create](https://skillsdocs.com/launchdarkly/ai-tooling/configs-create.md) — Create and configure configs in LaunchDarkly. Helps you choose between agent vs completion mode, create the config, add variations with models and prompts, and… 16. [configs-targeting](https://skillsdocs.com/launchdarkly/ai-tooling/configs-targeting.md) — Configure config targeting rules to control which variations serve to different users. Enable percentage rollouts, attribute-based rules, segment targeting, an… 17. [configs-update](https://skillsdocs.com/launchdarkly/ai-tooling/configs-update.md) — Update, archive, and delete LaunchDarkly configs and their variations. Use when you need to modify config properties, change model parameters, update instructi… 18. [configs-variations](https://skillsdocs.com/launchdarkly/ai-tooling/configs-variations.md) — Experiment with configs by creating and managing variations. Helps you test different models, prompts, and parameters to find what works best through systemati… 19. [custom-metrics](https://skillsdocs.com/launchdarkly/ai-tooling/custom-metrics.md) — Create, track, retrieve, update, and delete custom business metrics for configs. Covers full lifecycle: define metric kinds via API, emit events via SDK, and q… 20. [migrate](https://skillsdocs.com/launchdarkly/ai-tooling/migrate.md) — Migrate an application with hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation in five stages: audit the code, wrap the call, move the to… 21. [online-evals](https://skillsdocs.com/launchdarkly/ai-tooling/online-evals.md) — Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores. 22. [projects](https://skillsdocs.com/launchdarkly/ai-tooling/projects.md) — Guide for setting up LaunchDarkly projects in your codebase. Helps you assess your stack, choose the right approach, and integrate project management that make… 23. [snippets](https://skillsdocs.com/launchdarkly/ai-tooling/snippets.md) — Create and manage prompt snippets — reusable text blocks referenced inside config variation prompts. Keeps common instructions, personas, and guardrails consis… 24. [tools](https://skillsdocs.com/launchdarkly/ai-tooling/tools.md) — Give your agents capabilities through tools (function calling). Helps you identify what your agent needs to do, create tool definitions, and attach them to con… 25. [launchdarkly-experiment-setup](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-experiment-setup.md) — Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between… 26. [flag-and-release-change](https://skillsdocs.com/launchdarkly/ai-tooling/flag-and-release-change.md) — Drive a pull request's change end to end: decide it's flag-worthy, create the guarding flag, wire the new code path behind it on the PR branch, and record an a… 27. [flag-release](https://skillsdocs.com/launchdarkly/ai-tooling/flag-release.md) — Record an automated rollout for an existing LaunchDarkly flag that guards a pull request's change, so the change releases safely when the PR merges. Honors a s… 28. [launchdarkly-flag-cleanup](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-cleanup.md) — Safely remove a feature flag from code while preserving production behavior. Use when the user wants to remove a flag from code, delete flag references, or cre… 29. [launchdarkly-flag-command](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-command.md) — Resolve `/flag` style requests into the right LaunchDarkly flag lookup flow. Use when the user types `/flag`, asks to quickly find a flag by name/key, wants a… 30. [launchdarkly-flag-create](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-create.md) — Create and configure LaunchDarkly feature flags in a way that fits the existing codebase. Use when the user wants to create a new flag, wrap code in a flag, ad… 31. [launchdarkly-flag-discovery](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-discovery.md) — Audit your LaunchDarkly feature flags to understand the landscape, find stale or launched flags, and assess removal readiness. Use when the user asks about fla… 32. [launchdarkly-flag-drift](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-drift.md) — Detect and reconcile drift between a feature flag's in-code SDK fallback default and its LaunchDarkly default rule (fallthrough). Use when a flag's default rul… 33. [launchdarkly-flag-qualitative-feedback-setup](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-qualitative-feedback-setup.md) — Integrate LaunchDarkly qualitative user feedback into a JavaScript/TypeScript codebase. Guides framework and design system detection, builds the sendFeedback u… 34. [launchdarkly-flag-targeting](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-targeting.md) — Control LaunchDarkly feature flag targeting including toggling flags on/off, percentage rollouts, targeting rules, individual targets, and copying flag configu… 35. [launchdarkly-guarded-rollout](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-guarded-rollout.md) — Configure guarded rollouts with progressive traffic increases, metric monitoring, and automatic rollback. Use when releasing features gradually with safety thr… 36. [should-flag-change](https://skillsdocs.com/launchdarkly/ai-tooling/should-flag-change.md) — Decide whether a given code change should be placed behind a LaunchDarkly feature flag. Use when a developer asks whether a change should be behind a flag, whe… 37. [launchdarkly-metric-choose](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-metric-choose.md) — Choose the right metrics for a LaunchDarkly experiment, guarded rollout, or release policy. Use when the user wants to know which metrics to use, which is the… 38. [launchdarkly-metric-create](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-metric-create.md) — Create a LaunchDarkly metric that measures what matters for an experiment or rollout. Use when the user wants to create a metric, track an event, measure page… 39. [launchdarkly-metric-instrument](https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-metric-instrument.md) — Instrument a LaunchDarkly metric event in a codebase by adding a track() call. Use when the user wants to wire up an event, instrument an action for a metric,… 40. [alert-investigation](https://skillsdocs.com/launchdarkly/ai-tooling/alert-investigation.md) — Investigates a triggered observability alert and returns a structured diagnosis with likely cause, scope, and next steps. 41. [create-fix-pr](https://skillsdocs.com/launchdarkly/ai-tooling/create-fix-pr.md) — Investigates a root cause and files a minimal fix PR for a reported bug or observability finding. 42. [create-graph](https://skillsdocs.com/launchdarkly/ai-tooling/create-graph.md) — Creates observability dashboards and graphs from logs, traces, errors, sessions, metrics, and events data by previewing charts inline and saving them to a dash… 43. [investigate](https://skillsdocs.com/launchdarkly/ai-tooling/investigate.md) — Analyzes observability data — logs, traces, errors, sessions, and metrics — to find root cause and actionable evidence. Use when the user reports a bug, an une… 44. [first-flag](https://skillsdocs.com/launchdarkly/ai-tooling/first-flag.md) — Create a boolean first flag, add evaluation, toggle on/off for end-to-end proof. Parent onboarding Step 6; uses MCP, API, or ldcli; optional flag-create skill. 45. [mcp-configure](https://skillsdocs.com/launchdarkly/ai-tooling/mcp-configure.md) — Configure the LaunchDarkly hosted MCP server during onboarding. Use when the parent LaunchDarkly onboarding skill reaches Step 4 (MCP). Supports Cursor, Claude… 46. [apply](https://skillsdocs.com/launchdarkly/ai-tooling/apply.md) — Apply LaunchDarkly SDK onboarding: install dependency (or dual-SDK pair), configure env and secrets with consent, add init at entrypoint(s), verify compile. Ne… 47. [detect](https://skillsdocs.com/launchdarkly/ai-tooling/detect.md) — Detect repository stack for LaunchDarkly SDK onboarding: languages, frameworks, package managers, monorepo targets, entrypoints, existing LD usage. Nested unde… 48. [plan](https://skillsdocs.com/launchdarkly/ai-tooling/plan.md) — Generate a minimal LaunchDarkly SDK integration plan from detected stack: choose SDK type(s), dual-SDK server+client when required, files to change, env conven… 49. [sdk-install](https://skillsdocs.com/launchdarkly/ai-tooling/sdk-install.md) — Install and initialize the correct LaunchDarkly SDK during onboarding by running nested skills in order: detect, plan, apply. Parent onboarding Step 6 is first… 50. [onboarding](https://skillsdocs.com/launchdarkly/ai-tooling/onboarding.md) — Onboard a project to LaunchDarkly: kickoff roadmap, resumable log, explore repo, MCP, companion flag skills, nested SDK install (detect/plan/apply), first flag… ## Front matter _The repository README, verbatim except that relative links are resolved against https://github.com/launchdarkly/ai-tooling/blob/main/._ # LaunchDarkly Agent Skills LaunchDarkly's public collection of agent skills and playbooks. These skills encode repeatable workflows for working with LaunchDarkly, so coding agents can execute common tasks safely and consistently. ## What Is This Repo? Agent Skills are modular, text-based playbooks that teach an agent how to perform a workflow. This repo is designed to be a public, open-source home for LaunchDarkly skills and to align with the emerging Agent Skills Open Standard. ## Available Skills ### Feature Flags | Skill | Description | |-------|-------------| | `feature-flags/launchdarkly-flag-command` | Resolve `/flag` style requests into fast flag lookup, detail, and disambiguation | | `feature-flags/should-flag-change` | Advisory, read-only call on whether a code change (diff/PR) should ship behind a feature flag | | `feature-flags/launchdarkly-flag-discovery` | Audit flags, find stale/launched flags, and assess removal readiness | | `feature-flags/launchdarkly-flag-create` | Create new feature flags in a way that fits existing codebase patterns | | `feature-flags/launchdarkly-flag-targeting` | Control targeting, rollouts, rules, and cross-environment config | | `feature-flags/launchdarkly-flag-cleanup` | Safely remove flags from code using LaunchDarkly as the source of truth | | `feature-flags/launchdarkly-flag-drift` | Detect and reconcile drift between an in-code SDK fallback default and the LaunchDarkly default rule | | `feature-flags/launchdarkly-guarded-rollout` | Configure guarded rollouts with progressive traffic, metric monitoring, and rollback | | `feature-flags/flag-release` | Record a flag's automated release for a PR, honoring release intent and per-environment release policies | | `feature-flags/flag-and-release-change` | End-to-end PR orchestrator: decide → create + wire the flag → record its release (composes the skills above) | | `feature-flags/launchdarkly-flag-qualitative-feedback-setup` | Add a qualitative user feedback widget tied to a flag, adapting to the project's framework and design system | ### AgentControl | Skill | Description | |-------|-------------| | `agentcontrol/configs-create` | Create configs with variations for agent or completion mode | | `agentcontrol/migrate` | Migrate an app with hardcoded LLM prompts to AgentControl in five stages (extract, wrap, tools, tracking, evals) | | `agentcontrol/configs-update` | Update and delete configs, manage lifecycle | | `agentcontrol/configs-variations` | Manage config variations for A/B testing | | `agentcontrol/tools` | Create and attach tools for function calling | | `agentcontrol/projects` | Create and manage projects to organize configs | | `agentcontrol/online-evals` | Attach LLM-as-a-judge evaluators to configs | | `agentcontrol/configs-targeting` | Configure targeting rules for config rollouts | | `agentcontrol/snippets` | Create and manage reusable prompt snippets across configs | | `agentcontrol/agent-graphs` | Create and manage multi-agent graphs with routing and handoffs | ### Experiments | Skill | Description | |-------|-------------| | `experiments/launchdarkly-experiment-setup` | Set up experiments with metrics, treatments, and data collection | ### Metrics | Skill | Description | |-------|-------------| | `metrics/launchdarkly-metric-choose` | Select the right metric type for an experiment | | `metrics/launchdarkly-metric-create` | Create metrics and instrument tracking events | | `metrics/launchdarkly-metric-instrument` | Add tracking calls to code for existing metrics | ## Install as a Claude Code Plugin This repo is a [Claude Code plugin](https://code.claude.com/docs/en/create-plugins). Installing it gives you all the skills above plus the LaunchDarkly MCP server. 1. Add this repo as a plugin marketplace in Claude Code: ``` /plugin marketplace add launchdarkly/ai-tooling ``` 2. Install the plugin: ``` /plugin install launchdarkly@launchdarkly-ai-tooling ``` 3. Authenticate the LaunchDarkly MCP server when prompted with your [API access token](https://docs.launchdarkly.com/home/account/api). Once installed, skills are available as `/launchdarkly:` across all your projects, and the MCP server can read and modify your flags directly. ### Onboarding | Skill | Description | |-------|-------------| | `onboarding` | End-to-end LaunchDarkly setup: kickoff roadmap, MCP, SDK install, first flag | | `onboarding/mcp-configure` | Configure the LaunchDarkly hosted MCP server (OAuth, no API keys needed) | | `onboarding/sdk-install` | Install and initialize the correct SDK via detect, plan, and apply sub-steps | | `onboarding/first-flag` | Create a boolean flag, evaluate it, toggle on/off for end-to-end proof | ## Install as a Cursor Plugin This repo is a [Cursor plugin](https://cursor.com/docs/plugins/building). Installing it gives you all the skills above plus the LaunchDarkly MCP server, so the agent can read and modify your flags directly. 1. Open Cursor and go to **Settings > Plugins**. 2. Search for **LaunchDarkly** in the marketplace, or install from the repo URL: ``` https://github.com/launchdarkly/ai-tooling ``` Once installed, the skills and MCP server are available across all your projects. ## Quick Start (Local) ```bash # Clone the repo git clone https://github.com/launchdarkly/ai-tooling.git cd ai-tooling # If your agent supports skills.sh installs: npx skills add launchdarkly/ai-tooling # Or manually copy a skill into your agent's skills path: cp -r skills/feature-flags/launchdarkly-flag-cleanup / ``` Then ask your agent something like: ``` Which feature flags are stale and should be cleaned up? ``` ``` Create a feature flag for the new checkout flow ``` ``` Roll out dark-mode to 25% of users in production ``` ``` Remove the `new-checkout-flow` feature flag from this codebase ``` ## Install via skills.sh CLI ```bash npx skills add ``` ## Contributing See `CONTRIBUTING.md` for how to add new skills and the conventions we follow. ## License Apache-2.0 --- ## 1. agent-graphs - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/agent-graphs/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/agent-graphs/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/agent-graphs.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: agent-graphs description: "Create and manage agent graphs — directed graphs of configs connected by edges with handoff logic. Use when building multi-agent workflows where configs route to each other." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "0.1.0" --- # Config Agent Graphs You're using a skill that will guide you through creating and managing agent graphs in LaunchDarkly. Your job is to design the graph topology, create it with the right edges and handoffs, and verify the routing between config nodes. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `create-agent-graph` -- create a new graph with nodes and edges - `get-agent-graph` -- inspect a graph's structure and edges - `list-agent-graphs` -- browse existing graphs in the project **Optional MCP tools:** - `update-agent-graph` -- modify edges, root config, or description - `delete-agent-graph` -- permanently remove a graph - `get-ai-config` -- inspect individual configs that serve as nodes - `create-ai-config` -- create new configs to use as graph nodes ## Core Concepts ### What Are Agent Graphs? An agent graph is a directed graph where: - **Nodes** are configs (each config is an agent with its own model, prompt, and tools) - **Edges** define routing between configs (source -> target) - **Handoff data** on edges controls how context is passed between agents - **Root config** is the entry point — the first agent that receives user input ### When to Use Agent Graphs | Scenario | Example | |----------|---------| | **Multi-step workflows** | Triage agent -> Specialist agent -> Summary agent | | **Routing by intent** | Router agent decides which specialist handles the request | | **Escalation chains** | L1 support -> L2 support -> Human handoff | | **Pipeline processing** | Extract -> Transform -> Validate -> Store | ### Graph Structure ``` [Root Config] --edge--> [Config A] --edge--> [Config C] \--edge--> [Config B] ``` Each edge has: - `key` -- unique identifier for the edge - `sourceConfig` -- the config key that routes FROM - `targetConfig` -- the config key that routes TO - `handoff` (optional) -- data/instructions passed during the transition ## Core Principles 1. **Design Before Building**: Map out nodes and edges on paper/whiteboard first 2. **One Agent, One Job**: Each node should have a clear, focused responsibility 3. **Root Config Is the Router**: The entry point should understand how to dispatch 4. **Handoff Data Matters**: Define what context flows between agents 5. **Verify the Full Path**: Test that routing works end-to-end ## Workflow ### Step 1: Design the Graph Before creating anything: 1. Identify the agents (configs) needed — each is a graph node 2. Map the routing: which agent hands off to which? 3. Define handoff data: what context does each edge carry? 4. Identify the root config: which agent receives initial input? 5. Check existing graphs with `list-agent-graphs` to avoid duplicates 6. Check existing configs with `get-ai-config` to see what nodes already exist ### Step 2: Ensure Nodes Exist Each node in the graph must be an existing config. If configs don't exist yet: 1. Use `create-ai-config` to create each agent config 2. Set up variations with appropriate models and prompts for each agent's role 3. Verify each config exists with `get-ai-config` ### Step 3: Create the Graph Use `create-agent-graph` with: - `projectKey` -- the project containing the configs - `key` -- unique identifier for the graph - `name` -- human-readable display name - `description` (optional) -- explain the graph's purpose - `rootConfigKey` -- the entry-point config key - `edges` -- array of connections between configs ```json { "projectKey": "my-project", "key": "support-triage-graph", "name": "Customer Support Triage", "description": "Routes customer queries to the appropriate specialist agent", "rootConfigKey": "triage-agent", "edges": [ { "key": "triage-to-billing", "sourceConfig": "triage-agent", "targetConfig": "billing-specialist", "handoff": {"category": "billing", "priority": "normal"} }, { "key": "triage-to-technical", "sourceConfig": "triage-agent", "targetConfig": "technical-specialist", "handoff": {"category": "technical", "priority": "normal"} } ] } ``` ### Step 4: Verify 1. Use `get-agent-graph` to confirm the graph was created with the correct structure 2. Verify edges connect the right source and target configs 3. Check that the root config key matches the intended entry point 4. Confirm handoff data is present on edges that need it **Report results:** - Graph created with N nodes and M edges - Root config set correctly - All edges verified ## Edge Cases | Situation | Action | |-----------|--------| | Config doesn't exist yet | Create it first with `create-ai-config` before referencing in a graph | | Circular routing | Allowed but warn user — ensure there's a termination condition in the agent logic | | Single-node graph | Valid but unusual — consider if a graph is actually needed | | Updating edges | Use `update-agent-graph` — provide the complete new edge list | ## What NOT to Do - Don't create a graph before the config nodes exist - Don't forget handoff data when agents need context from predecessors - Don't create overly complex graphs — start simple and add nodes as needed - Don't delete a graph without understanding if it's actively used in agent workflows ## Other Resources To learn more, read [Agent graphs](https://launchdarkly.com/docs/home/agentcontrol/agent-graphs.md). --- ## 2. aiconfig-agent-graphs - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-agent-graphs/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-agent-graphs/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-agent-graphs.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-agent-graphs description: "DEPRECATED redirect — this skill was renamed to agent-graphs. Do not use this skill; invoke agent-graphs instead. Kept only so old references to aiconfig-agent-graphs still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `agent-graphs` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`agent-graphs`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-agent-graphs` to `agent-graphs`. --- ## 3. aiconfig-ai-metrics - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-ai-metrics/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-ai-metrics/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-ai-metrics.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-ai-metrics description: "DEPRECATED redirect — this skill was renamed to built-in-metrics. Do not use this skill; invoke built-in-metrics instead. Kept only so old references to aiconfig-ai-metrics still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `built-in-metrics` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`built-in-metrics`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-ai-metrics` to `built-in-metrics`. --- ## 4. aiconfig-create - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-create/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-create/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-create.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-create description: "DEPRECATED redirect — this skill was renamed to configs-create. Do not use this skill; invoke configs-create instead. Kept only so old references to aiconfig-create still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `configs-create` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`configs-create`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-create` to `configs-create`. --- ## 5. aiconfig-custom-metrics - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-custom-metrics/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-custom-metrics/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-custom-metrics.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-custom-metrics description: "DEPRECATED redirect — this skill was renamed to custom-metrics. Do not use this skill; invoke custom-metrics instead. Kept only so old references to aiconfig-custom-metrics still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `custom-metrics` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`custom-metrics`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-custom-metrics` to `custom-metrics`. --- ## 6. aiconfig-migrate - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-migrate/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-migrate/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-migrate.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-migrate description: "DEPRECATED redirect — this skill was renamed to migrate. Do not use this skill; invoke migrate instead. Kept only so old references to aiconfig-migrate still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `migrate` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`migrate`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-migrate` to `migrate`. --- ## 7. aiconfig-online-evals - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-online-evals/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-online-evals/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-online-evals.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-online-evals description: "DEPRECATED redirect — this skill was renamed to online-evals. Do not use this skill; invoke online-evals instead. Kept only so old references to aiconfig-online-evals still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `online-evals` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`online-evals`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-online-evals` to `online-evals`. --- ## 8. aiconfig-projects - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-projects/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-projects/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-projects.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-projects description: "DEPRECATED redirect — this skill was renamed to projects. Do not use this skill; invoke projects instead. Kept only so old references to aiconfig-projects still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `projects` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`projects`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-projects` to `projects`. --- ## 9. aiconfig-snippets - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-snippets/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-snippets/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-snippets.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-snippets description: "DEPRECATED redirect — this skill was renamed to snippets. Do not use this skill; invoke snippets instead. Kept only so old references to aiconfig-snippets still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `snippets` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`snippets`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-snippets` to `snippets`. --- ## 10. aiconfig-targeting - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-targeting/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-targeting/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-targeting.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-targeting description: "DEPRECATED redirect — this skill was renamed to configs-targeting. Do not use this skill; invoke configs-targeting instead. Kept only so old references to aiconfig-targeting still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `configs-targeting` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`configs-targeting`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-targeting` to `configs-targeting`. --- ## 11. aiconfig-tools - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-tools/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-tools/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-tools.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-tools description: "DEPRECATED redirect — this skill was renamed to tools. Do not use this skill; invoke tools instead. Kept only so old references to aiconfig-tools still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `tools` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`tools`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-tools` to `tools`. --- ## 12. aiconfig-update - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-update/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-update/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-update.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-update description: "DEPRECATED redirect — this skill was renamed to configs-update. Do not use this skill; invoke configs-update instead. Kept only so old references to aiconfig-update still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `configs-update` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`configs-update`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-update` to `configs-update`. --- ## 13. aiconfig-variations - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/aiconfig-variations/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/aiconfig-variations/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/aiconfig-variations.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: aiconfig-variations description: "DEPRECATED redirect — this skill was renamed to configs-variations. Do not use this skill; invoke configs-variations instead. Kept only so old references to aiconfig-variations still point users to the new name." license: Apache-2.0 metadata: version: 0.1.0 --- # Deprecated — renamed to `configs-variations` This skill has been renamed. **Do not follow any instructions here and do not perform the task from this skill.** Invoke the **`configs-variations`** skill instead — it contains the current, complete instructions. If you arrived here from an old reference (documentation, a saved prompt, or automation), update it from `aiconfig-variations` to `configs-variations`. --- ## 14. built-in-metrics - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/built-in-metrics/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/built-in-metrics.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (8), referenced from this skill's directory: - `references/anthropic-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/anthropic-tracking.md - `references/bedrock-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/bedrock-tracking.md - `references/gemini-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/gemini-tracking.md - `references/langchain-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/langchain-tracking.md - `references/metrics-api.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/metrics-api.md - `references/openai-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/openai-tracking.md - `references/strands-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/strands-tracking.md - `references/streaming-tracking.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/built-in-metrics/references/streaming-tracking.md --- name: built-in-metrics description: "Instrument an existing codebase with LaunchDarkly config tracking. Walks the four-tier ladder (managed runner → provider package → custom extractor + trackMetricsOf → raw manual) and picks the lowest-ceremony option that still captures duration, tokens, and success/error." license: Apache-2.0 compatibility: Requires the LaunchDarkly server-side AI SDK (`launchdarkly-server-sdk-ai>=0.20.0` for Python or `@launchdarkly/server-sdk-ai>=0.20.0` for Node) and an existing config. metadata: author: launchdarkly version: "1.0.0-experimental" --- # Agent Metrics Instrumentation You're using a skill that wires LaunchDarkly agent metrics around an existing provider call. Your job is to audit what's already there, pick the right tier from the ladder below, and implement it with the **least ceremony that still captures the metrics the Monitoring tab needs** (duration, input/output tokens, success/error, plus TTFT when streaming). The single most important thing to get right: **default to the highest tier that fits the shape of the call**. Going lower ("just write the manual tracker calls") looks flexible but costs you drift, missed metrics, and legacy patterns the SDKs have moved past. ## The four-tier ladder This is the order the official SDK READMEs (Python core, Node core, and every provider package) recommend. Walk from the top and stop at the first tier that fits: | Tier | Pattern | Use when | Tracks automatically | |------|---------|----------|----------------------| | **1 — Managed runner** | Python: `ai_client.create_model(...)` returning a `ManagedModel`, then `await model.run(...)`.
Node: `aiClient.createModel(...)` returning a `ManagedModel`, then `await model.run(...)`. | The call is conversational (chat history, turn-based). This is what the provider READMEs lead with. | Duration, tokens, success/error — **all of it, zero tracker calls**. | | **2 — Provider package + `trackMetricsOf`** | `tracker.trackMetricsOf(Provider.getAIMetricsFromResponse, () => providerCall())`. Provider packages today: `@launchdarkly/server-sdk-ai-openai`, `-langchain`, `-vercel` (Node) and `launchdarkly-server-sdk-ai-openai`, `-langchain` (Python). | The shape isn't a chat loop (one-shot completion, structured output, agent step) but the framework or provider has a package. | Duration + success/error from the wrapper; tokens from the package's built-in `getAIMetricsFromResponse` extractor. | | **3 — Custom extractor + `trackMetricsOf`** | Same `trackMetricsOf` wrapper, but you write a small function that maps the provider response to `LDAIMetrics` (tokens + success). | No provider package exists (Anthropic direct, Gemini, Cohere, custom HTTP). | Duration + success/error from the wrapper; tokens from your extractor. | | **4 — Raw manual** | Separate calls to `trackDuration`, `trackTokens`, `trackSuccess` / `trackError`, plus `trackTimeToFirstToken` for streams. | Streaming with TTFT, unusual response shapes, partial tracking, anything Tier 2–3 can't cleanly wrap. | Only what you explicitly call — it's on you to not miss one. | Every provider — OpenAI, LangChain, Vercel, Bedrock, Anthropic, Gemini, custom HTTP — uses the same generic shape: `tracker.trackMetricsOf(getAIMetricsFromResponse, () => providerCall())` in Node, `tracker.track_metrics_of(get_ai_metrics_from_response, provider_call)` in Python. The extractor is the only thing that changes per provider: import `getAIMetricsFromResponse` from the matching `@launchdarkly/server-sdk-ai-` (or `ldai_`) package, or write a small custom function that returns `LDAIMetrics`. There are no provider-specific tracker methods. ## Workflow ### 1. Explore the existing call site Before picking a tier, find the provider call and answer these questions: - [ ] **Shape?** Is it a chat loop (history + turn-based), a one-shot completion, an agent step, or something else? → drives Tier 1 vs 2. - [ ] **Framework?** Raw provider SDK? LangChain / LangGraph? Vercel AI SDK? CrewAI? Strands? → drives which Tier-2 provider package (if any) applies. - [ ] **Provider?** OpenAI, Anthropic, Bedrock, Gemini, Azure, custom HTTP? → cross-reference with the package availability matrix below. - [ ] **Streaming?** If yes, you'll need TTFT tracking, which means Tier 4 for the TTFT part even if the rest is Tier 2. - [ ] **Language?** Python or Node? Provider-package coverage differs between them. - [ ] **Already using a config?** If not, route to `configs-create` first — tracking requires a tracker, which is obtained by calling `create_tracker()` / `createTracker()` on the config object returned by `completion_config()` / `completionConfig()` / `createModel()`. - [ ] **On the current SDK API?** If the call site uses `aiclient.config(...)` / `aiClient.config(...)` or constructs an `AIConfig(...)` / `LDAIConfig` default, it's on the pre-0.20 surface. Migrate it as part of this work before adding tracking: - `aiclient.config(...)` → `aiclient.completion_config(...)` for one-shot/chat or `aiclient.agent_config(...)` for agent mode (mirror the call signature). Node is the same with camelCase. - `AIConfig(...)` default → `AICompletionConfigDefault(...)` or `AIAgentConfigDefault(...)` (Node: `LDAICompletionConfigDefault` / `LDAIAgentConfigDefault`). `AIConfig` is the base class the SDK returns; it isn't a valid default-value constructor — the typed `*Default` variants are. - If the result was being tuple-unpacked (`config, tracker = aiclient.config(...)`), drop the unpack — the new methods return a single config object. Obtain the tracker via `config.create_tracker()` / `aiConfig.createTracker()`. - For deeper rewrites (call sites with hardcoded model/prompt as well), hand off to `migrate` instead of doing the full migration here. ### 2. Look up your Tier-2 option Use this matrix to decide whether Tier 2 (provider package) is available for your situation. If it's not, drop to Tier 3 (custom extractor). If the shape is chat-loop, go to Tier 1 first regardless of what's in this matrix. | Framework / provider | Python provider package | Node provider package | Reference | |---|---|---|---| | OpenAI (direct SDK) | `launchdarkly-server-sdk-ai-openai` | `@launchdarkly/server-sdk-ai-openai` | [openai-tracking.md](references/openai-tracking.md) | | LangChain / LangGraph | `launchdarkly-server-sdk-ai-langchain` | `@launchdarkly/server-sdk-ai-langchain` | [langchain-tracking.md](references/langchain-tracking.md) | | Vercel AI SDK | — | `@launchdarkly/server-sdk-ai-vercel` | (use the Vercel provider docs) | | AWS Bedrock (Converse or InvokeModel) | — (use LangChain-aws or custom extractor) | — (use LangChain-aws or custom extractor) | [bedrock-tracking.md](references/bedrock-tracking.md) | | Anthropic direct SDK | — | — | [anthropic-tracking.md](references/anthropic-tracking.md) | | Gemini / Google GenAI | — | — | [gemini-tracking.md](references/gemini-tracking.md) | | Strands Agents | — (Tier 3 custom extractor) | — (Tier 3 custom extractor) | [strands-tracking.md](references/strands-tracking.md) | | Cohere, Mistral, custom HTTP | — | — | Tier 3 custom extractor | | **Any provider, streaming + TTFT** | — (Tier 4 only) | `trackStreamMetricsOf` (no TTFT) + manual TTFT | [streaming-tracking.md](references/streaming-tracking.md) | ### 3. Implement from the matching reference Once you know the tier and the provider, open the reference file and follow the pattern. The references are written so Tier 1 is always the first example, Tier 2/3 next, and Tier 4 last. Stop at the first tier that matches the app's shape. Guardrails that apply to every tier: 1. **Always check `config.enabled`** before making the tracked call. A disabled config means the user has flagged the feature off — you should short-circuit to whatever fallback the app uses (cached response, error, degraded path) rather than making the provider call at all. 2. **Wrap the existing call, don't rewrite it.** Tier 2 and Tier 3 are designed to slot around an unmodified provider call. If you find yourself rewriting the call to fit the tracker, you're at the wrong tier — drop down one. 3. **Errors are handled inside `trackMetricsOf`.** The wrapper catches exceptions, records `trackError()` internally, and re-raises — do **not** add `except: tracker.trackError()` on top, it's a noop that also trips the at-most-once guard. Tier 1 handles both paths automatically. At Tier 4 (manual, streaming, `track_duration_of`) the caller does own the error-tracking call. 4. **Always flush before close.** Call `ldClient.flush()` (Python: `ldclient.get().flush()`; Node: `await ldClient.flush()`) before closing the client. Trailing events are at risk of being lost otherwise — in short-lived scripts and long-running services alike. In Node, `ldClient.close()` returns a Promise; await it. ### 4. Verify Confirm the Monitoring tab fills in: - [ ] Run one real request through the instrumented path. - [ ] Open the config in LaunchDarkly → **Monitoring** tab. Duration, token counts, and generation counts should appear within 1–2 minutes. - [ ] Force an error (bad API key, zero `max_tokens`, whatever) and confirm the error count increments. - [ ] If streaming: verify TTFT appears. If it doesn't, you probably wrapped the stream creation with `trackMetricsOf` but didn't add the manual `trackTimeToFirstToken` call — see [streaming-tracking.md](references/streaming-tracking.md). ## Quick reference: tracker methods Obtain a tracker via the factory on the config object: `tracker = config.create_tracker()` (Python) or `const tracker = aiConfig.createTracker()` (Node). Call the factory once per execution and reuse the returned `tracker` for every call — each factory invocation mints a new `runId` that tags every tracking event emitted by that tracker so events from a single execution can be correlated together (via exported events / downstream systems). The Monitoring tab aggregates events rather than grouping them by run today — the `runId` is useful when events are exported or queried outside the UI, and is the identifier the SDK's at-most-once guards are keyed on. The methods below are the raw API surface — most of the time you should not call them individually; use `trackMetricsOf` or a Tier-1 managed runner. The list is here so you can recognize the methods in existing code and reach for the right one when you genuinely need Tier 4. | Method (Python ↔ Node) | Tier | What it does | |---|---|---| | `track_metrics_of(extractor, fn)` / `trackMetricsOf(extractor, fn)` | **2 / 3** | Wraps a provider call, captures duration + success/error, calls your extractor for tokens. **This is the default generic tracker.** | | `track_metrics_of_async(extractor, fn)` (Python) | 2 / 3 | Async variant of the above. | | `trackStreamMetricsOf(extractor, streamFn)` (Node only) | 2 / 3 | Streaming variant. Captures per-chunk usage when the extractor handles chunks. Does **not** auto-capture TTFT. | | `track_duration(ms)` / `trackDuration(ms)` | 4 | Record latency in milliseconds. | | `track_duration_of(fn)` / `trackDurationOf(fn)` | 4 | Wraps a callable and records duration automatically. Does not capture tokens or success — pair with explicit calls. | | `track_tokens(TokenUsage)` / `trackTokens({input, output, total})` | 4 | Record token usage. | | `track_time_to_first_token(ms)` / `trackTimeToFirstToken(ms)` | 4 | Record TTFT for streaming responses. | | `track_success()` / `trackSuccess()` | 4 | Mark the generation as successful. Required for the Monitoring tab to count it. | | `track_error()` / `trackError()` | 4 | Mark the generation as failed. Do not also call `trackSuccess()` in the same request. | | `track_feedback({kind})` / `trackFeedback({kind})` | any | Record thumbs-up / thumbs-down from a feedback UI. Independent of the success/error path. | | `track_tool_call(name)` / `trackToolCall(name)` | any | Record a single tool invocation by name. Available on both SDKs. | | `track_tool_calls([names])` / `trackToolCalls([names])` | any | Batch variant — record a list of tool invocations in one call. | | `track_judge_result(result)` / `trackJudgeResult(result)` | any | Record a programmatic judge evaluation. `result.sampled` indicates whether evaluation ran. | ## Related skills - `configs-create` — prerequisite if the app doesn't have a config yet - `custom-metrics` — business metrics (conversion, resolution, retention) layered on top of the agent metrics this skill captures - `online-evals` — automatic quality scoring (LLM-as-judge) on sampled live requests; complementary to the metrics here - `migrate` — Stage 4 of the hardcoded-to-AgentControl migration delegates to this skill --- ## 15. configs-create - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/configs-create/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-create/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/configs-create.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-create/README.md --- name: configs-create description: "Create and configure configs in LaunchDarkly. Helps you choose between agent vs completion mode, create the config, add variations with models and prompts, and verify the setup." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # Create Config You're using a skill that will guide you through creating a config in LaunchDarkly. Your job is to understand the use case, choose the right mode, create the config and its variations, and verify everything is set up correctly. > **⚠️ This skill creates a config — it does not make it servable.** A freshly-created config has its **fallthrough pointing at an auto-generated disabled variation**, not at the variation you just created. The SDK will return `ai_config.enabled=False` on every evaluation until you flip targeting on and point the fallthrough at your new variation. This is not a bug — it's the default state. **You must run `/configs-targeting` (or the equivalent REST / CLI call shown in Step 5) before verifying against the SDK**, or verification will look like the LD-served path is broken when it isn't. The single most common failure mode users hit with this skill is skipping the targeting step and spending time debugging `enabled=False` in their application code. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Primary MCP tool:** - `setup-ai-config` -- create a config with its first variation in one step (recommended) **Alternative MCP tools (for more control):** - `create-ai-config` -- create just the config shell (key, name, mode) - `create-ai-config-variation` -- add a variation with model, prompts, and parameters - `get-ai-config` -- verify the config was created correctly **Optional MCP tools (enhance workflow):** - `list-ai-configs` -- browse existing configs to understand naming conventions - `create-project` -- create a project if one doesn't exist yet ## Important: Bias Towards Action When the user provides enough context (use case, model, mode), proceed through the entire workflow without stopping to ask for details you can infer. Use reasonable defaults for unspecified fields: `default` for variation key, the use case as the basis for instructions/messages, kebab-case for config keys. Complete all steps (create + verify) in one pass. ## Workflow ### Step 1: Understand the Use Case Before creating, identify what you're building: - **What framework?** LangGraph, LangChain, CrewAI, Strands, OpenAI SDK, Anthropic SDK, custom - **What does the agent need?** Just text generation, or tools/function calling? - **Agent or completion?** See the decision matrix below ### Step 2: Choose Agent vs Completion Mode This choice is about **input schema and framework compatibility**, not execution behavior. Agent mode returns an `instructions` string; completion mode returns a `messages` array. Both provide provider abstraction, A/B testing, and metrics tracking. | Your Need | Mode | Why | |-----------|------|-----| | LangGraph, CrewAI, Strands, AutoGen frameworks | **Agent** | Frameworks expect goal/instruction input | | Persistent instructions across interactions | **Agent** | Single instructions string, SDK method: `agent_config()` (Python) / `agentConfig()` (Node) | | Direct OpenAI/Anthropic API calls | **Completion** | Messages array maps directly to provider APIs | | Full control of message structure | **Completion** | System/user/assistant role-based messages | | One-off text generation | **Completion** | Standard chat format | | Need online evaluations (LLM-as-judge) | **Completion** | Online evals are only available in completion mode | **Both modes support tools.** Not all models support agent mode -- check model compatibility if using agent mode. If unsure, start with completion mode (it's the API default and more flexible). ### Step 3: Create the Config (Recommended: One Step) Use `setup-ai-config` to create the config and its first variation in one call. This is the recommended approach: it handles creation, variation setup, and verification automatically. **Config fields:** - `key` -- unique identifier (lowercase, hyphens) - `name` -- human-readable name - `mode` -- `"agent"` or `"completion"` - Optional: `description`, `tags` **Variation fields:** - `variationKey`, `variationName` -- identifiers for the first variation - `modelConfigKey` -- must be `Provider.model-id` format (e.g., `OpenAI.gpt-4o`, `Anthropic.claude-sonnet-4-5`) - `modelName` -- the model identifier (e.g., `gpt-4o`). **Always pass this in the initial call** — leaving it off produces a variation that displays "NO MODEL" and forces a second PATCH to set it. The field is `modelName`; it is **not** `name` or `model.name` on this endpoint. **For agent mode**, provide: - `instructions` -- a string with the agent's system instructions Example agent-mode call: ```json { "projectKey": "my-project", "key": "support-agent", "name": "Support Agent", "mode": "agent", "variationKey": "default", "variationName": "Default", "modelConfigKey": "OpenAI.gpt-4o", "modelName": "gpt-4o", "instructions": "You are a customer support agent. Help users resolve their issues." } ``` **For completion mode**, provide: - `messages` -- an array of `{role, content}` objects (system, user, assistant) Example completion-mode call: ```json { "projectKey": "my-project", "key": "product-descriptions", "name": "Product Descriptions", "mode": "completion", "variationKey": "default", "variationName": "Default", "modelConfigKey": "Anthropic.claude-sonnet-4-5", "modelName": "claude-sonnet-4-5", "messages": [ {"role": "system", "content": "You are a product copywriter. Write compelling descriptions."}, {"role": "user", "content": "Write a description for: {{product_name}}"} ] } ``` **Optional:** - `parameters` -- model parameters like `{temperature: 0.7, max_tokens: 2000}` (match the UI's snake_case keys) The tool returns the full verified config detail with the variation attached. ### Step 3 (Alternative): Two-Step Creation If the user asks for more control or a step-by-step approach, use the individual tools: 1. `create-ai-config` -- create the config shell 2. `create-ai-config-variation` -- add the variation with model, prompts, and parameters 3. `get-ai-config` -- verify the result **Execute all three steps without stopping to ask for details.** Infer the variation key (`default`), name (`Default`), instructions/messages, and model from the user's request context. If the user asked for GPT-4o agent mode, you have enough to complete the entire flow. Only ask clarifying questions if the mode or model is truly ambiguous. ### Step 4: Verify If you used `setup-ai-config`, verification is automatic: the response includes the full config with variations. Check: 1. Config exists with the correct mode 2. Variation has a model assigned (not "NO MODEL") 3. Instructions or messages are present 4. Parameters are set **Use `get-ai-config` for the verification call — do not drop to raw `curl` + `jq`.** The MCP tool returns a typed object you can inspect directly. Hand-rolled `jq` filters against the REST response routinely break: the configs detail endpoint returns the variation list under different keys depending on `expand`, and a filter like `.variations.items[]` will fail with `Cannot index array with string "items"` when the response shape is a bare array. If you must call the REST API, use `jq -e .` first to inspect the actual shape before drilling in. **Report results:** - Config created with correct structure - Variation has model assigned - Flag any missing model or parameters - Provide config URL: `https://app.launchdarkly.com/projects/{projectKey}/ai-configs/{configKey}` ### Step 5: Make the variation servable `setup-ai-config` and `create-ai-config-variation` create the variation but **do not promote it to fallthrough**. The new config will return `enabled=False` to every consumer until targeting is updated. This is the single most common "I created a config but my SDK still gets the fallback" failure. **The workflow is not complete until this step is done.** #### What to tell the user Print this checklist verbatim to the user after Step 4, then wait for confirmation. Do not claim the skill succeeded until the user confirms the fallthrough was flipped. > ✅ Config and variation are created. > > 🔴 **The SDK will return `enabled=False` until you flip targeting on.** The fallthrough is currently pointing at an auto-generated disabled variation, not at the `{variationKey}` you just created. > > **Next step — run `/configs-targeting`** with these inputs: > - Project key: `{projectKey}` > - Config key: `{configKey}` > - Environment key: the env whose SDK key is in your `.env` (usually `test` or `production`) > - Fallthrough variation: `{variationKey}` (the one this skill just created) > > Verify after targeting is flipped by: > 1. Opening the config in the LD UI, switching to the correct environment, and confirming "Default rule serves: `{variationName}`" is shown with targeting **On**. > 2. Running a quick test: `ai_config = ai_client.{completion|agent}_config(...)` and asserting `ai_config.enabled is True`. #### Direct shortcut if the user wants to flip targeting without invoking the sibling skill `configs-targeting` is the canonical path — it handles percentage rollouts, targeted rules, and variation-ID lookups. But for the simplest case ("promote the new variation to fallthrough in one environment"), you can run the underlying semantic PATCH yourself once you know the new variation's `_id`. Get the variation ID (use `get-ai-config` MCP, or): ```bash curl -s "https://app.launchdarkly.com/api/v2/projects/$PROJECT/ai-configs/$CONFIG_KEY/targeting?env=$ENV" \ -H "Authorization: $LD_API_KEY" -H "LD-API-Version: beta" \ | jq '.variations[] | {key, _id}' ``` Flip the fallthrough to point at it: ```bash curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/$PROJECT/ai-configs/$CONFIG_KEY/targeting?env=$ENV" \ -H "Authorization: $LD_API_KEY" \ -H "Content-Type: application/json; domain-model=launchdarkly.semanticpatch" \ -H "LD-API-Version: beta" \ -d '{"instructions":[{"kind":"updateFallthroughVariationOrRollout","variationId":""}]}' ``` Or the same thing via the LD CLI if it's installed locally: ```bash ldcli resources ai-configs update-ai-config-targeting \ --projectKey $PROJECT --configKey $CONFIG_KEY --envKey $ENV \ --data '{"instructions":[{"kind":"updateFallthroughVariationOrRollout","variationId":""}]}' ``` Do not use `turnTargetingOn` — that semantic-patch instruction does **not** work for configs. `updateFallthroughVariationOrRollout` is the only instruction that actually flips the fallthrough. ## modelConfigKey Format Required for models to display in the UI. Format: `{Provider}.{model-id}` - `OpenAI.gpt-4o` - `OpenAI.gpt-4o-mini` - `Anthropic.claude-sonnet-4-5` - `Anthropic.claude-3-5-sonnet` The `create-ai-config-variation` tool validates this format and rejects invalid values. ## Edge Cases | Situation | Action | |-----------|--------| | Config already exists | Ask if user wants to update instead | | Variation shows "NO MODEL" | Use `update-ai-config-variation` to set modelConfigKey | | Need to attach tools | Create tools first (`tools` skill), then update the variation | ## What NOT to Do - Don't create configs without understanding the use case - Don't skip the two-step process (config then variation) - Don't try to attach tools during initial creation -- update the variation afterward - Don't forget modelConfigKey (models won't show in the UI) - Don't omit `modelName` from the initial variation call. It is required at create time; setting it via a follow-up PATCH is a workaround for a bug, not the intended flow. The PATCH field is also `modelName`, not `name`. - Don't drop to raw `curl` + `jq` for verification. Use `get-ai-config` (MCP) — it returns a typed object and avoids brittle `jq` filters that break on response-shape variation. - Don't consider the workflow complete until the user has been told to run `configs-targeting`. A created variation that isn't promoted to fallthrough returns `enabled=False` to every consumer. ## More resources To learn more about creating a config in the LaunchDarkly UI, read [Create configs](https://launchdarkly.com/docs/home/agentcontrol/create.md) To learn more about configuring the SDK, read: * [.NET AI SDK reference](https://launchdarkly.com/docs/sdk/ai/dotnet.md) * [Go AI SDK reference](https://launchdarkly.com/docs/sdk/ai/go.md) * [Node.js (server-side) SDK AI reference](https://launchdarkly.com/docs/sdk/ai/node-js.md) * [Python AI SDK reference](https://launchdarkly.com/docs/sdk/ai/python.md) * [Ruby AI SDK reference](https://launchdarkly.com/docs/sdk/ai/ruby.md) ## Related Skills - `tools` -- Create tools before attaching - `configs-variations` -- Add more variations for experimentation - `configs-update` -- Modify configs based on learnings --- ## 16. configs-targeting - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/configs-targeting/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-targeting/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/configs-targeting.md - **Licence:** Other — https://github.com/launchdarkly/ai-tooling Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-targeting/README.md --- name: configs-targeting description: Configure config targeting rules to control which variations serve to different users. Enable percentage rollouts, attribute-based rules, segment targeting, and guarded rollouts. compatibility: Requires LaunchDarkly API access token with ai-configs:write permission. metadata: author: launchdarkly version: "0.1.0" --- # Config Targeting Configure targeting rules for configs to control which variations serve to different contexts. Works the same for both completion and agent mode. ## Prerequisites - LaunchDarkly account with AgentControl enabled - API access token with write permissions - Project key and environment key - Existing config with variations (use `configs-create` skill) ## API Key Detection 1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY` 2. **Check MCP config** - Claude: `~/.claude/config.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY` 3. **Prompt user** - Only if detection fails ## Core Concepts ### Evaluation Order Targeting rules evaluate in this order (same as feature flags): 1. **Individual targets** - Specific context keys (highest priority) 2. **Segment rules** - Pre-defined segments 3. **Custom rules** - Attribute-based conditions (evaluated in order) 4. **Default rule** - Fallthrough for all others 5. **Off variation** - When targeting is disabled ### Semantic Patch API config targeting uses semantic patch instructions: ``` PATCH /api/v2/projects/{projectKey}/ai-configs/{configKey}/targeting Content-Type: application/json; domain-model=launchdarkly.semanticpatch ``` ### Key Concepts - **variationId**: UUIDs, not keys. Always fetch targeting first to get IDs. - **Weights**: Thousandths (50000 = 50%, 100000 = 100%) - **Clause logic**: Multiple clauses = AND, multiple values = OR - **Null attributes**: Rules with null/missing attributes are skipped ## Workflow ### Step 1: Get Targeting (with Variation IDs) ```bash curl -X GET "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/{configKey}/targeting" \ -H "Authorization: {api_token}" \ -H "LD-API-Version: beta" ``` Response includes `variations` array with `_id` (UUID) for each variation. ### Step 2: Edit the Default Rule Edit the default rule to serve the variation you created. > **Important:** The `turnTargetingOn` instruction does not work for configs. Use `updateFallthroughVariationOrRollout` instead. ```bash # First, get variation IDs from Step 1 response # Then set fallthrough to the enabled variation (e.g., "Default" variation) curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/{configKey}/targeting" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json; domain-model=launchdarkly.semanticpatch" \ -H "LD-API-Version: beta" \ -d '{ "environmentKey": "production", "instructions": [{ "kind": "updateFallthroughVariationOrRollout", "variationId": "your-enabled-variation-uuid" }] }' ``` ### Step 3: Add Targeting Rules **Attribute-based rule:** ```bash curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/{configKey}/targeting" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json; domain-model=launchdarkly.semanticpatch" \ -H "LD-API-Version: beta" \ -d '{ "environmentKey": "production", "instructions": [{ "kind": "addRule", "clauses": [{ "contextKind": "user", "attribute": "selectedModel", "op": "contains", "values": ["sonnet"], "negate": false }], "variation": 0 }] }' ``` **Percentage rollout:** ```bash curl -X PATCH "..." \ -d '{ "environmentKey": "production", "instructions": [{ "kind": "addRule", "clauses": [{ "contextKind": "user", "attribute": "tier", "op": "in", "values": ["premium"], "negate": false }], "percentageRolloutConfig": { "contextKind": "user", "bucketBy": "key", "variations": [ {"variation": 0, "weight": 60000}, {"variation": 1, "weight": 40000} ] } }] }' ``` **Set fallthrough (default rule):** ```bash curl -X PATCH "..." \ -d '{ "environmentKey": "production", "instructions": [{ "kind": "updateFallthroughVariationOrRollout", "variationId": "fallback-variation-uuid" }] }' ``` ## Python Implementation ```python import requests import os from typing import Dict, List, Optional class AIConfigTargeting: """Manager for config targeting rules""" def __init__(self, api_token: str, project_key: str): self.api_token = api_token self.project_key = project_key self.base_url = "https://app.launchdarkly.com/api/v2" def get_targeting(self, config_key: str) -> Optional[Dict]: """Get current targeting with variation IDs.""" url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/targeting" response = requests.get(url, headers={ "Authorization": self.api_token, "LD-API-Version": "beta" }) if response.status_code == 200: return response.json() print(f"[ERROR] {response.status_code}: {response.text}") return None def get_variation_id(self, config_key: str, variation_key: str) -> Optional[str]: """Look up variation UUID from key or name.""" targeting = self.get_targeting(config_key) if targeting: for var in targeting.get("variations", []): if var.get("key") == variation_key or var.get("name") == variation_key: return var.get("_id") return None def update_targeting(self, config_key: str, environment: str, instructions: List[Dict], comment: str = "") -> Optional[Dict]: """Send semantic patch instructions.""" url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/targeting" payload = {"environmentKey": environment, "instructions": instructions} if comment: payload["comment"] = comment response = requests.patch(url, headers={ "Authorization": self.api_token, "Content-Type": "application/json; domain-model=launchdarkly.semanticpatch", "LD-API-Version": "beta" }, json=payload) if response.status_code == 200: return response.json() print(f"[ERROR] {response.status_code}: {response.text}") return None def enable_config(self, config_key: str, environment: str, variation_key: str = "default") -> bool: """ Enable a config by setting fallthrough to an enabled variation. Note: turnTargetingOn doesn't work for configs. Instead, set the fallthrough from the disabled variation (index 0) to an enabled one. """ variation_id = self.get_variation_id(config_key, variation_key) if not variation_id: print(f"[ERROR] Variation '{variation_key}' not found") return False return self.set_fallthrough(config_key, environment, variation_id) def add_rule(self, config_key: str, environment: str, clauses: List[Dict], variation: int, description: str = "") -> bool: """Add targeting rule serving a specific variation index.""" instruction = { "kind": "addRule", "clauses": clauses, "variation": variation } if description: instruction["description"] = description result = self.update_targeting(config_key, environment, [instruction], f"Add rule: {description}") if result: print(f"[OK] Rule added") return True return False def add_rollout_rule(self, config_key: str, environment: str, clauses: List[Dict], weights: List[Dict], bucket_by: str = "key") -> bool: """ Add percentage rollout rule. weights: [{"variation": 0, "weight": 50000}, {"variation": 1, "weight": 50000}] """ result = self.update_targeting(config_key, environment, [{ "kind": "addRule", "clauses": clauses, "percentageRolloutConfig": { "contextKind": "user", "bucketBy": bucket_by, "variations": weights } }], "Add percentage rollout") if result: print(f"[OK] Rollout rule added") return True return False def set_fallthrough(self, config_key: str, environment: str, variation_id: str) -> bool: """Set default (fallthrough) variation by UUID.""" result = self.update_targeting(config_key, environment, [{ "kind": "updateFallthroughVariationOrRollout", "variationId": variation_id }], "Set fallthrough") if result: print(f"[OK] Fallthrough set") return True return False def target_individuals(self, config_key: str, environment: str, context_keys: List[str], variation: int, context_kind: str = "user") -> bool: """Target specific context keys.""" result = self.update_targeting(config_key, environment, [{ "kind": "addTargets", "variation": variation, "contextKind": context_kind, "values": context_keys }], f"Target {len(context_keys)} individuals") if result: print(f"[OK] Individual targets added") return True return False def target_segment(self, config_key: str, environment: str, segment_keys: List[str], variation: int) -> bool: """Target a segment.""" result = self.update_targeting(config_key, environment, [{ "kind": "addRule", "clauses": [{ "attribute": "segmentMatch", "contextKind": "", # Leave blank for segments "op": "segmentMatch", "values": segment_keys, "negate": False }], "variation": variation }], f"Target segments: {segment_keys}") if result: print(f"[OK] Segment targeting added") return True return False def clear_rules(self, config_key: str, environment: str) -> bool: """Remove all targeting rules.""" result = self.update_targeting(config_key, environment, [{"kind": "replaceRules", "rules": []}], "Clear all rules") if result: print(f"[OK] All rules cleared") return True return False ``` ## Instruction Reference > **Note:** `turnTargetingOn` and `turnTargetingOff` do not work for configs. Configs have targeting enabled by default. To "enable" a config, set the fallthrough to an enabled variation using `updateFallthroughVariationOrRollout`. ### Rules | Kind | Description | |------|-------------| | `addRule` | Add rule with clauses and variation/rollout | | `removeRule` | Remove by ruleId | | `replaceRules` | Replace all rules | | `reorderRules` | Change evaluation order | | `updateRuleVariationOrRollout` | Update what a rule serves | ### Fallthrough | Kind | Description | |------|-------------| | `updateFallthroughVariationOrRollout` | Set default variation or rollout | ### Individual Targets | Kind | Description | |------|-------------| | `addTargets` | Target specific context keys | | `removeTargets` | Remove specific targets | | `replaceTargets` | Replace all targets | ## Operators Reference | Operator | Description | Example | |----------|-------------|---------| | `in` | Value in list | `["premium", "enterprise"]` | | `contains` | String contains | `["sonnet"]` | | `startsWith` | String prefix | `["user-"]` | | `endsWith` | String suffix | `[".edu"]` | | `matches` | Regex match | `["^user-\\d+$"]` | | `greaterThan` / `lessThan` | Numeric comparison | `[100]` | | `before` / `after` | Date comparison | `["2024-12-31T00:00:00Z"]` | | `semVerEqual` / `semVerGreaterThan` | Version comparison | `["2.0.0"]` | | `segmentMatch` | Segment membership | `["beta-testers"]` | ## Clause Structure ```json { "contextKind": "user", "attribute": "email", "op": "endsWith", "values": [".edu"], "negate": false } ``` - Multiple clauses = AND (all must match) - Multiple values = OR (any can match) - `negate: true` inverts the operator ## Rollout Types ### Manual Percentage Rollout ```json { "percentageRolloutConfig": { "contextKind": "user", "bucketBy": "key", "variations": [ {"variation": 0, "weight": 50000}, {"variation": 1, "weight": 50000} ] } } ``` ### Progressive Rollout ```json { "progressiveRolloutConfig": { "contextKind": "user", "controlVariation": 1, "endVariation": 0, "steps": [ {"rolloutWeight": 1000, "duration": {"quantity": 4, "unit": "hour"}}, {"rolloutWeight": 5000, "duration": {"quantity": 4, "unit": "hour"}}, {"rolloutWeight": 10000, "duration": {"quantity": 4, "unit": "hour"}} ] } } ``` ### Guarded Rollout ```json { "guardedRolloutConfig": { "randomizationUnit": "user", "stages": [ {"rolloutWeight": 1000, "monitoringWindowMilliseconds": 17280000}, {"rolloutWeight": 5000, "monitoringWindowMilliseconds": 17280000} ], "metrics": [{ "metricKey": "error-rate", "onRegression": {"rollback": true}, "regressionThreshold": 0.01 }] } } ``` ## Common Patterns ### Model Routing by Attribute ```python # Route based on selectedModel context attribute targeting.add_rule( config_key="model-selector", environment="production", clauses=[{ "contextKind": "user", "attribute": "selectedModel", "op": "contains", "values": ["sonnet"], "negate": False }], variation=0, # Sonnet variation index description="Route sonnet requests" ) ``` ### Tier-Based Variation ```python targeting.add_rule( config_key="chat-assistant", environment="production", clauses=[{ "contextKind": "user", "attribute": "tier", "op": "in", "values": ["premium", "enterprise"], "negate": False }], variation=0 # Premium model variation ) ``` ### Segment Targeting ```python targeting.target_segment( config_key="chat-assistant", environment="production", segment_keys=["beta-testers"], variation=1 # Experimental variation ) ``` ## Error Handling | Status | Cause | Solution | |--------|-------|----------| | 400 | Invalid semantic patch | Check instruction format, ops must be lowercase | | 403 | Insufficient permissions | Check API token | | 404 | Config not found | Verify projectKey and configKey | | 422 | Invalid variation | Use index (0, 1, 2...) or UUID from targeting response | ## Next Steps After configuring targeting: 1. **Provide config URL:** ``` https://app.launchdarkly.com/projects/{projectKey}/ai-configs/{configKey} ``` 2. **Monitor performance** with `built-in-metrics` 3. **Attach judges** with `online-evals` 4. **Set up guarded rollouts** for automatic regression detection ## Related Skills - `configs-create` - Create configs with variations - `configs-variations` - Manage variations - `online-evals` - Attach judges - `segments` - Create segments for targeting ## Other Resources - [Target with AgentControl](https://docs.launchdarkly.com/home/ai-configs/target.md) - [Targeting Rules](https://docs.launchdarkly.com/home/flags/target-rules.md) - [JSON Targeting](https://docs.launchdarkly.com/home/flags/json-targeting.md) - [Guarded Rollouts](https://docs.launchdarkly.com/home/releases/guarded-rollouts.md) --- ## 17. configs-update - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/configs-update/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-update/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/configs-update.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-update/README.md --- name: configs-update description: "Update, archive, and delete LaunchDarkly configs and their variations. Use when you need to modify config properties, change model parameters, update instructions or messages, archive unused configs, or permanently remove them." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # Config Update & Lifecycle You're using a skill that will guide you through updating, archiving, and deleting configs and their variations. Your job is to understand the current state of the config, make the changes, and verify the result. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `get-ai-config-health` -- assess config health before making changes (detects missing models, orphaned tools, empty configs) - `get-ai-config` -- understand current state before making changes - `update-ai-config` -- update config metadata (name, description, tags, archive) - `update-ai-config-variation` -- update variation model, prompts, or parameters **Optional MCP tools:** - `delete-ai-config` -- permanently delete a config (irreversible) - `delete-ai-config-variation` -- permanently delete a variation (irreversible) ## Core Principles 1. **Fetch Before Changing**: Always check the current state before modifying 2. **Verify After Changing**: Fetch the config again to confirm updates were applied 3. **Archive Before Deleting**: Archival is reversible; deletion is not ## Workflow ### Step 1: Assess Health and Understand Current State Start with `get-ai-config-health` to get a structured health assessment. This detects: - Variations with no model (show as "NO MODEL" in the UI) - Variations with neither instructions nor messages - Orphaned tool references (tools attached that don't exist in the project) - Configs with no variations at all The health verdict (`healthy`, `warning`, `unhealthy`) helps you prioritize what to fix. Then use `get-ai-config` to review the full detail: - Current mode (agent or completion) - Existing variations and their models - Current instructions or messages - Attached tools and parameters ### Step 2: Make the Update **Update config metadata** -- Use `update-ai-config`: - Change name or description - Add or replace tags - Archive with `archived: true` (reversible) **Update a variation** -- Use `update-ai-config-variation`: - Switch model (provide new `modelConfigKey` and `modelName`) - Change instructions or messages - Tune parameters (temperature, max_tokens, etc.) - Attach or detach tools via the parameters object **Archive a config** -- Use `update-ai-config` with `archived: true`. Archiving is the **preferred** way to retire a config: - It is reversible (unarchive with `archived: false`) - The config is hidden from active lists but preserved - After calling the archive, treat a successful response as confirmation and proceed to verification - When a user says "remove", "retire", "decommission", or "no longer need", default to archiving unless they explicitly say "delete permanently" **Delete** -- Use `delete-ai-config` or `delete-ai-config-variation` (irreversible, requires `confirm: true`). **Always suggest archiving first.** Only proceed with deletion if the user explicitly confirms they want permanent, irreversible removal. ### Step 3: Verify Use `get-ai-config` to confirm the response shows your updated values. **Report results:** - Update applied successfully - Config reflects changes - Flag any issues or rollback if needed ## What NOT to Do - Don't update production configs without testing in another variation first - Don't change multiple things at once -- make incremental changes - Don't skip verification - Don't delete without explicit user confirmation -- always suggest archiving first - Don't retry an update because the API response doesn't echo back the exact values you sent -- verify with `get-ai-config` instead ## More resources To learn more about creating and managing variations, read [Create and manage config variations](https://launchdarkly.com/docs/home/agentcontrol/create-variation.md). ## Related Skills - `configs-variations` -- Create variations to test changes side-by-side - `tools` -- Update tool attachments --- ## 18. configs-variations - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/configs-variations/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-variations/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/configs-variations.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/configs-variations/README.md --- name: configs-variations description: "Experiment with configs by creating and managing variations. Helps you test different models, prompts, and parameters to find what works best through systematic experimentation." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # Config Variations You're using a skill that will guide you through testing and optimizing configs through variations. Your job is to design experiments, create variations, and systematically find what works best. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Primary MCP tool:** - `clone-ai-config-variation` -- clone a baseline variation with selective overrides (recommended for experimentation) **Alternative MCP tools (for more control):** - `get-ai-config` -- review existing variations before adding new ones - `create-ai-config-variation` -- create new variations from scratch **Optional MCP tools:** - `update-ai-config-variation` -- refine a variation after creation - `delete-ai-config-variation` -- remove variations that didn't work out ## Core Principles 1. **Test One Thing at a Time**: Change model OR prompt OR parameters, not all at once 2. **Have a Hypothesis**: Know what you're trying to improve 3. **Measure Results**: Use metrics to compare variations 4. **Verify via Tool**: The agent fetches the config to confirm variations exist ## Workflow ### Step 1: Identify What to Optimize What's the problem? Cost, quality, speed, accuracy? How will you measure success? ### Step 2: Design the Experiment | Goal | What to Vary | |------|--------------| | Reduce cost | Cheaper model (e.g., `gpt-4o-mini`) | | Improve quality | Better model or more detailed prompt | | Reduce latency | Faster model, lower `max_tokens` | | Increase accuracy | Different model family (Claude vs GPT-4) | ### Step 3: Create Variations (Recommended: Clone with Overrides) Use `clone-ai-config-variation` to duplicate the baseline and override only what you're testing. The tool reads the source variation, merges your overrides, and creates the new variation. Everything you **don't** pass is inherited from the source automatically. **Required fields:** - `sourceVariationKey` -- the baseline to clone from - `key` and `name` -- identifiers for the new variation (e.g., `gpt4o-mini-cost-test`) **Override ONLY the fields you are testing.** Leave all other fields unset -- do not pass them even if you know their current values. The clone tool inherits them from the source. This enforces the one-variable-at-a-time principle: - Testing a cheaper model? Pass only `modelConfigKey` and `modelName`. Do NOT pass `instructions`, `messages`, or `parameters`. - Testing different instructions? Pass only `instructions`. Do NOT pass `modelConfigKey` or `modelName`. - Testing a parameter? Pass only `parameters`. Do NOT pass model or prompt fields. The response returns both the source and created variation, so you can immediately verify the diff. ### Step 3 (Alternative): Create from Scratch If you need full control, use `get-ai-config` first to review the current state, then `create-ai-config-variation` with all fields specified manually. Always fetch before creating so you understand the existing config's mode, model, and parameters. ### Step 4: Verify If you used `clone-ai-config-variation`, the response includes both source and created variations for immediate comparison. Otherwise, use `get-ai-config` to confirm. **Report results:** - Variations created with correct models and parameters - Only the intended variable differs between variations - Flag any issues **Note on API responses:** After calling a creation or clone tool, treat a successful response as confirmation that the operation succeeded. The API response may not echo back every field you sent (e.g., model fields may show defaults). Do not retry or assume failure based on response field values alone -- verify with `get-ai-config` if needed. ## modelConfigKey Format Required for models to display in the UI. Format: `{Provider}.{model-id}`: - `OpenAI.gpt-4o`, `OpenAI.gpt-4o-mini` - `Anthropic.claude-sonnet-4-5`, `Anthropic.claude-3-5-sonnet` ## Safety: Protect the Baseline When the user wants to try a different model, prompt, or parameters, **always create a new variation alongside the baseline**. Never modify or delete the existing baseline variation. This applies even if the user says "replace" or "switch" -- the correct action is to create a new variation and let targeting/rollouts control traffic, not to edit the original. - Use `clone-ai-config-variation` or `create-ai-config-variation` to add the new variation - Do NOT use `update-ai-config-variation` on the baseline to change its model or instructions - Do NOT use `delete-ai-config-variation` on the baseline - Explain to the user that keeping the baseline enables comparison and safe rollback ## What NOT to Do - Don't test too many things at once -- change one variable per variation - Don't pass unchanged fields when cloning -- let the tool inherit them from the source - Don't forget modelConfigKey (variations without it show as "NO MODEL" in the UI) - Don't make decisions on small sample sizes - Don't modify or remove the baseline variation -- create new variations alongside it - Don't use `update-ai-config-variation` to "replace" a baseline -- create a new variation instead ## More resources To learn more about creating and managing variations, read [Create and manage config variations](https://launchdarkly.com/docs/home/agentcontrol/create-variation.md). ## Related Skills - `configs-create` -- Create the initial config - `configs-update` -- Refine based on learnings --- ## 19. custom-metrics - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/custom-metrics/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/custom-metrics/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/custom-metrics.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: custom-metrics description: "Create, track, retrieve, update, and delete custom business metrics for configs. Covers full lifecycle: define metric kinds via API, emit events via SDK, and query results." license: Apache-2.0 compatibility: Requires the LaunchDarkly server SDK and a LaunchDarkly API token with the `writer` role for metric management. metadata: author: launchdarkly version: "1.0.0-experimental" --- # Custom Metrics for Configs Full lifecycle management of custom business metrics: create metric definitions via API, track events via SDK, retrieve metric data, and manage metrics programmatically. ## Prerequisites - LaunchDarkly SDK initialized (see `sdk`) - LaunchDarkly API token with `writer` role for metric management - Understanding of built-in agent metrics (see `built-in-metrics`) ## API Key Detection Before prompting the user for an API key, try to detect it automatically: 1. **Check Claude MCP config** - Read `~/.claude/config.json` and look for `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY` 2. **Check environment variables** - Look for `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, or `LD_API_KEY` 3. **Prompt user** - Only if detection fails, ask the user for their API key ```python import os import json from pathlib import Path def get_launchdarkly_api_key(): """Auto-detect LaunchDarkly API key from Claude config or environment.""" # 1. Check Claude MCP config claude_config = Path.home() / ".claude" / "config.json" if claude_config.exists(): try: config = json.load(open(claude_config)) api_key = config.get("mcpServers", {}).get("launchdarkly", {}).get("env", {}).get("LAUNCHDARKLY_API_KEY") if api_key: return api_key except (json.JSONDecodeError, IOError): pass # 2. Check environment variables for var in ["LAUNCHDARKLY_API_KEY", "LAUNCHDARKLY_API_TOKEN", "LD_API_KEY"]: if os.environ.get(var): return os.environ[var] return None ``` ## Metrics Lifecycle Overview | Step | Method | Purpose | |------|--------|---------| | 1. Create | API | Define metric in LaunchDarkly | | 2. Track | SDK | Send events to the metric | | 3. Get | API | Retrieve metric definition/data | | 4. Update | API | Modify metric properties | | 5. Delete | API | Remove metric | ## 1. Create Metric (API) **Required fields for numeric custom metrics:** - `successCriteria` - Must be one of: `"HigherThanBaseline"`, `"LowerThanBaseline"` - `unit` - e.g., `"count"`, `"percent"`, `"milliseconds"` The API will return `400 Bad Request` if these are missing for numeric metrics. ```python import requests import os def create_metric( project_key: str, metric_key: str, name: str, kind: str = "custom", is_numeric: bool = True, unit: str = "count", success_criteria: str = "HigherThanBaseline", event_key: str = None, description: str = None ): """Create a new metric definition in LaunchDarkly.""" API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}" payload = { "key": metric_key, "name": name, "kind": kind, "isNumeric": is_numeric, "eventKey": event_key or metric_key } # Unit and successCriteria are required for numeric custom metrics if is_numeric and kind == "custom": payload["unit"] = unit payload["successCriteria"] = success_criteria if description: payload["description"] = description headers = { "Authorization": API_TOKEN, "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) if response.status_code == 201: print(f"[OK] Created metric: {metric_key}") return response.json() elif response.status_code == 409: print(f"[INFO] Metric already exists: {metric_key}") return None else: print(f"[ERROR] Failed to create metric: {response.status_code}") print(f" {response.text}") return None ``` **Metric Kinds:** - `custom` - Track any event (most common for agent metrics) - `pageview` - Track page views - `click` - Track click events **Success Criteria** (for numeric metrics): - `HigherThanBaseline` - Higher values are better (e.g., revenue, satisfaction) - `LowerThanBaseline` - Lower values are better (e.g., errors, latency) **Common Units:** - `count` - Generic count - `milliseconds` - Time duration - `percent` - Percentage values - `dollars` - Currency ## 2. Track Events (SDK) Once the metric is created, track events using the SDK: ```python from ldclient import Context from ldclient.config import Config import ldclient # Initialize (see sdk for details) ldclient.set_config(Config("your-sdk-key")) ld_client = ldclient.get() def track_metric(ld_client, user_id: str, metric_key: str, value: float, data: dict = None): """Track an event to a metric.""" context = Context.builder(user_id).build() ld_client.track( metric_key, context, data=data, metric_value=value ) ``` ### Common Tracking Patterns ```python def track_conversion(ld_client, user_id: str, amount: float, config_key: str): """Track a conversion event with revenue.""" context = Context.builder(user_id).build() ld_client.track( "business.conversion", context, data={"configKey": config_key, "category": "electronics"}, metric_value=amount ) def track_task_success(ld_client, user_id: str, task_type: str, success: bool): """Track task completion success/failure.""" context = Context.builder(user_id).build() ld_client.track( "task.success_rate", context, data={"taskType": task_type}, metric_value=1.0 if success else 0.0 ) def track_satisfaction(ld_client, user_id: str, score: float, feedback_type: str): """Track user satisfaction (0-100 scale).""" context = Context.builder(user_id).build() ld_client.track( "user.satisfaction", context, data={"feedbackType": feedback_type}, metric_value=score ) # Track negative feedback separately for alerts if score < 50: ld_client.track( "user.negative_feedback", context, metric_value=1.0 ) def track_revenue(ld_client, user_id: str, revenue: float, source: str): """Track revenue generated after agent interaction.""" context = Context.builder(user_id).set("tier", "premium").build() if revenue > 0: ld_client.track( "revenue.impact", context, data={"source": source}, metric_value=revenue ) ``` ## 3. Get Metrics (API) ### Get Single Metric ```python def get_metric(project_key: str, metric_key: str): """Get a single metric definition.""" API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}" headers = {"Authorization": API_TOKEN} response = requests.get(url, headers=headers) if response.status_code == 200: metric = response.json() print(f"[OK] Metric: {metric['key']}") print(f" Name: {metric.get('name', 'N/A')}") print(f" Kind: {metric.get('kind', 'N/A')}") print(f" Numeric: {metric.get('isNumeric', False)}") print(f" Event Key: {metric.get('eventKey', 'N/A')}") return metric elif response.status_code == 404: print(f"[INFO] Metric not found: {metric_key}") return None else: print(f"[ERROR] Failed to get metric: {response.status_code}") return None ``` ### List All Metrics ```python def list_metrics(project_key: str, limit: int = 20): """List all metrics in a project.""" API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}" headers = {"Authorization": API_TOKEN} params = {"limit": limit} response = requests.get(url, headers=headers, params=params) if response.status_code == 200: data = response.json() metrics = data.get("items", []) print(f"[OK] Found {len(metrics)} metrics:") for metric in metrics: numeric = "numeric" if metric.get("isNumeric") else "non-numeric" print(f" - {metric['key']} ({metric.get('kind', 'custom')}, {numeric})") return metrics else: print(f"[ERROR] Failed to list metrics: {response.status_code}") return None ``` ## 4. Update Metric (API) ```python def update_metric(project_key: str, metric_key: str, updates: list): """ Update a metric using JSON Patch operations. Args: updates: List of patch operations, e.g.: [{"op": "replace", "path": "/name", "value": "New Name"}] """ API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}" headers = { "Authorization": API_TOKEN, "Content-Type": "application/json" } response = requests.patch(url, json=updates, headers=headers) if response.status_code == 200: print(f"[OK] Updated metric: {metric_key}") return response.json() elif response.status_code == 404: print(f"[ERROR] Metric not found: {metric_key}") return None else: print(f"[ERROR] Failed to update metric: {response.status_code}") print(f" {response.text}") return None # Example: Update metric name and description def rename_metric(project_key: str, metric_key: str, new_name: str, new_description: str = None): """Rename a metric and optionally update description.""" updates = [ {"op": "replace", "path": "/name", "value": new_name} ] if new_description: updates.append({"op": "replace", "path": "/description", "value": new_description}) return update_metric(project_key, metric_key, updates) ``` ## 5. Delete Metric (API) ```python def delete_metric(project_key: str, metric_key: str): """Delete a metric from the project.""" API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}" headers = {"Authorization": API_TOKEN} response = requests.delete(url, headers=headers) if response.status_code == 204: print(f"[OK] Deleted metric: {metric_key}") return True elif response.status_code == 404: print(f"[INFO] Metric not found: {metric_key}") return False else: print(f"[ERROR] Failed to delete metric: {response.status_code}") return False ``` ## Complete Workflow Example ```python import os import requests from ldclient import Context from ldclient.config import Config import ldclient # Setup API_TOKEN = os.environ.get("LAUNCHDARKLY_API_TOKEN") SDK_KEY = os.environ.get("LAUNCHDARKLY_SDK_KEY") PROJECT_KEY = "support-ai" ldclient.set_config(Config(SDK_KEY)) ld_client = ldclient.get() # 1. Create metric create_metric( PROJECT_KEY, "ai.task.completion", name="Agent Task Completion Rate", kind="custom", is_numeric=True, description="Tracks successful agent task completions" ) # 2. Track events context = Context.builder("user-123").build() ld_client.track("ai.task.completion", context, metric_value=1.0) ld_client.track("ai.task.completion", context, metric_value=1.0) ld_client.track("ai.task.completion", context, metric_value=0.0) # failure ld_client.flush() # 3. Get metric definition metric = get_metric(PROJECT_KEY, "ai.task.completion") # 4. Update metric name rename_metric(PROJECT_KEY, "ai.task.completion", "Agent Task Success Rate") # 5. List all metrics list_metrics(PROJECT_KEY) # 6. Delete metric (when no longer needed) # delete_metric(PROJECT_KEY, "ai.task.completion") ``` ## Session Metrics Tracker ```python import time from ldclient import Context class SessionMetricsTracker: """Track metrics across an entire user session.""" def __init__(self, ld_client): self.ld_client = ld_client self.session_data = {} def start_session(self, user_id: str, session_id: str): """Initialize session tracking.""" self.session_data[session_id] = { "user_id": user_id, "start_time": time.time(), "interactions": 0, "successful_tasks": 0 } def track_interaction(self, session_id: str, success: bool): """Track individual interaction within session.""" if session_id not in self.session_data: return session = self.session_data[session_id] session["interactions"] += 1 if success: session["successful_tasks"] += 1 def end_session(self, session_id: str): """Finalize and track session metrics.""" if session_id not in self.session_data: return None session = self.session_data[session_id] duration = time.time() - session["start_time"] context = Context.builder(session["user_id"]).build() # Track session duration self.ld_client.track( "session.duration", context, data={"interactions": session["interactions"]}, metric_value=duration ) # Track session success rate if session["interactions"] > 0: success_rate = session["successful_tasks"] / session["interactions"] self.ld_client.track( "session.success_rate", context, metric_value=success_rate * 100 ) result = dict(session) result["duration"] = duration del self.session_data[session_id] return result ``` ## Naming Conventions ```python # Use dot notation for hierarchy "quality.accuracy" "quality.relevance" "user.satisfaction" "user.engagement" "revenue.conversion" "task.success_rate" "session.duration" "ai.task.completion" "ai.recommendation.conversion" ``` ## Best Practices 1. **Create Before Track** - Metric must exist before tracking events 2. **Use Numeric Metrics** - Set `isNumeric=True` for aggregation 3. **Consistent Keys** - Use same key in `create_metric()` and `ld_client.track()` 4. **Always flush before close** - Call `ld_client.flush()` (await in Node) before `close()`. Trailing events are at risk of being lost otherwise, in short-lived scripts and long-running services alike. This is not a serverless-only rule; it applies to any process that exits. 5. **Rate Limit** - Don't track on every keystroke ## Viewing Metrics Custom metrics appear in: - **Metrics** page in LaunchDarkly UI - **Monitoring tab** of your config - Via API using `get_metric()` or `list_metrics()` ## Related Skills - `sdk` - SDK setup - `built-in-metrics` - Built-in agent metrics (tokens, duration, cost) - `online-evals` - Quality metrics via judges ## References - [Metrics API Documentation](https://apidocs.launchdarkly.com/tag/Metrics) - [Custom Events Documentation](https://docs.launchdarkly.com/sdk/features/events) - [Python SDK track() Reference](https://launchdarkly-python-sdk.readthedocs.io/) --- ## 20. migrate - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/migrate/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/migrate.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/README.md - `references/agent-graph-reference.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/agent-graph-reference.md - `references/agent-mode-frameworks.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/agent-mode-frameworks.md - `references/before-after-examples.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/before-after-examples.md - `references/fallback-defaults-pattern.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/fallback-defaults-pattern.md - `references/phase-1-analysis-checklist.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/phase-1-analysis-checklist.md - `references/sdk-ai-tracker-patterns.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/migrate/references/sdk-ai-tracker-patterns.md --- name: migrate description: "Migrate an application with hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation in five stages: audit the code, wrap the call, move the tools, add tracking, attach evaluators. Use when the user wants to externalize model/prompt configuration, move from direct provider calls (OpenAI, Anthropic, Bedrock, Gemini, Strands) to a managed config, or stage a full hardcoded-to-LaunchDarkly migration." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "0.1.0" --- # Migrate to AgentControl You're using a skill that will guide you through migrating an application from hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation. Your job is to run the migration in **five stages**, stopping at each stage for the user to confirm: 1. **Audit the code** — read-only scan that produces a structured list of everything hardcoded (prompt, model, parameters, tools, app-scoped knobs). 2. **Wrap the call** — install the SDK, create the config in LaunchDarkly with a fallback that mirrors the hardcoded values, and rewrite the call site to fetch the config fresh on every request. 3. **Move the tools** — extract each tool's JSON schema, attach it to the config, and swap every call site that references the old tool list. 4. **Add tracking** — wire the per-request tracker (duration, tokens, success/error) around the provider call. 5. **Attach evaluators** — either offline evals via the Playground + Datasets, or online judges that score sampled traffic automatically. > **⚠️ Three first-run failure modes to avoid.** > > 1. **Tracker in the wrong scope.** For an agent with a loop, mint `create_tracker()` once per user turn in a `setup_run` entry node — not inside `call_model`. Per-iteration factory calls produce N `runId`s and trip the at-most-once guards. See [agent-mode-frameworks.md § Custom `StateGraph`](references/agent-mode-frameworks.md). > 2. **`load_chat_model` wrapper reuse.** Templates like `langchain-ai/react-agent` ship a `load_chat_model(f"{provider}/{name}")` helper that wraps `init_chat_model(...)` and silently drops every variation parameter. **Delete it** (don't just avoid using it) and replace call sites with `create_langchain_model(ai_config)`. > 3. **Fallthrough not flipped after `/configs-create`.** A freshly-created config's fallthrough points at an auto-generated disabled variation, so the SDK returns `enabled=False` until `/configs-targeting` runs. Flip it before Stage 2 verification. ## Coverage — which shapes are well-trodden vs require extrapolation The skill is optimized for Python and Node.js / TypeScript; other languages are install-only. Within Python and Node the coverage tiers are: | Shape | Python | Node.js | Reference | |-------|--------|---------|-----------| | One-shot completion (direct OpenAI / Anthropic / Bedrock / Gemini call) | ✅ Worked example | ✅ Worked example | [before-after-examples.md](references/before-after-examples.md), per-provider docs in `built-in-metrics/references/` | | Chat loop via managed runner (`ManagedModel`) | ✅ Tier 1 pattern | ✅ Tier 1 pattern | [built-in-metrics SKILL.md](../built-in-metrics/SKILL.md) | | LangChain single-call | ✅ Worked example | ✅ Worked example | [langchain-tracking.md](../built-in-metrics/references/langchain-tracking.md) | | LangGraph prebuilt agent (Python `langchain.agents.create_agent`, Node `createReactAgent`) | ✅ Worked example | ✅ Worked example | [agent-mode-frameworks.md § LangGraph](references/agent-mode-frameworks.md) | | LangGraph custom `StateGraph` with run-scoped tracker (setup_run + call_model + finalize) | ✅ Deep worked example | ⚠️ Mentioned — translate from Python | [agent-mode-frameworks.md § Custom `StateGraph`](references/agent-mode-frameworks.md) | | CrewAI `Agent` | ✅ Worked example | — (not a Node framework) | [agent-mode-frameworks.md § CrewAI](references/agent-mode-frameworks.md) | | Strands `Agent` | ✅ Worked example | ⚠️ BedrockModel + OpenAIModel only (no Anthropic) | [agent-mode-frameworks.md § Strands](references/agent-mode-frameworks.md) | | Custom ReAct loop (hand-rolled, any framework or none) | ✅ Worked example | ⚠️ Apply framework-agnostic invariants; translate from Python | [agent-mode-frameworks.md § Custom ReAct loop](references/agent-mode-frameworks.md) | | Vercel AI SDK (`generateText` / `streamText`) | — (not a Python framework) | ⚠️ Provider package exists; no worked example in skill | `built-in-metrics` provider-package matrix | | Streaming (SSE / WebSocket) | ⚠️ Delegated to `built-in-metrics` streaming doc | ⚠️ Same — use `trackStreamMetricsOf` + manual TTFT | [streaming-tracking.md](../built-in-metrics/references/streaming-tracking.md) | | Multi-agent graph (supervisor + workers) | ⚠️ Out of main scope; see reference | ⚠️ Out of main scope; see reference | [agent-graph-reference.md](references/agent-graph-reference.md) | | Non-LangGraph agent frameworks (Pydantic AI, DSPy, AutoGen, Haystack, LlamaIndex agents, Semantic Kernel) | ⚠️ Apply the three invariants; no framework-specific example | ⚠️ Same | [agent-mode-frameworks.md § Framework-agnostic invariants](references/agent-mode-frameworks.md) | | Go, Ruby, .NET | ℹ️ Install commands only | ℹ️ Install commands only | [phase-1-analysis-checklist.md § SDK routing table](references/phase-1-analysis-checklist.md) | **Reading the key:** ✅ = follow the skill verbatim; ⚠️ = the architecture applies but you'll translate idioms or cross-reference another skill; ℹ️ = skill doesn't go past the install step. If the target app is in the ⚠️ column, start by reading [agent-mode-frameworks.md § Framework-agnostic invariants](references/agent-mode-frameworks.md) — those three rules (one `agent_config` per turn, one tracker per turn, at-most-once methods fire once at turn end) apply regardless of framework, and every code snippet in this skill is an instantiation of them. Translate the Python example's shape onto the target framework's primitives. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment, and an application that already calls an LLM provider with hardcoded model, prompt, and parameter values. **Required environment:** - `LD_SDK_KEY` — server-side SDK key (starts with `sdk-`) from the target LaunchDarkly project **MCP tools used directly by this skill:** none — every LaunchDarkly write happens in a focused sibling skill. **Check the SDK CHANGELOG before applying any pattern.** The API surface described throughout this skill targets the SDK behavior at the time of the skill's last update; SDK releases can rename, remove, or split methods after that. Before you start, fetch the latest CHANGELOG for the SDK(s) you'll target and skim for anything that contradicts the pattern you're about to apply: - Python: https://github.com/launchdarkly/python-server-sdk-ai/blob/main/packages/sdk/server-ai/CHANGELOG.md (and per-provider CHANGELOGs under `packages/ai-providers/server-ai-{openai,langchain}/CHANGELOG.md`) - Node: https://github.com/launchdarkly/js-core/blob/main/packages/sdk/server-ai/CHANGELOG.md (and per-provider CHANGELOGs under `packages/ai-providers/server-ai-{openai,langchain,vercel}/CHANGELOG.md`) If a CHANGELOG entry post-dates this skill and changes an API you're about to use, the CHANGELOG wins — and the skill should be updated. **Hand-off model.** This skill does **not** auto-invoke other skills. At each stage that needs a LaunchDarkly write, this skill prepares the inputs (config key, mode, model, prompt, tool schemas, judge keys) and then **tells the user to run the next slash-command themselves**. After the user finishes that sibling skill, return to the next step here. Treat the "Delegate" lines below as next-step instructions, not auto-handoffs. **Sibling skills the user runs at each stage:** - `projects` — pre-Stage 2, only if no project exists yet - `configs-create` — Stage 2 (creates the config and first variation) - `tools` — Stage 3 (creates tool definitions and attaches them) - `configs-targeting` — between Stage 2 and Stage 4 (promotes the new variation to fallthrough so the SDK actually serves it) - `online-evals` — Stage 5 (attaches judges, creates custom judges) ## Core Principles 1. **Inspect before you mutate.** Every stage begins with a read-only audit. Do not touch code until Step 1 is confirmed by the user. 2. **Replace config, not business logic.** The SDK call is a drop-in for the place where the model, parameters, and prompt are *defined* — not for the provider call itself. OpenAI/Anthropic/Bedrock calls stay where they are. 3. **Fallback mirrors current behavior.** The fallback passed to `completion_config` / `agent_config` must preserve the hardcoded values you removed, so the app is unchanged if LaunchDarkly is unreachable. 4. **Stages are ordered.** Wrap before you add tools. Add tools before you track. Track before you add evals. Skipping ahead produces configs without traffic, metrics without context, and judges with nothing to score. 5. **Hand off to focused skills, manually.** Each stage that needs a LaunchDarkly write tells the user to run a sibling slash-command (`/configs-create`, `/tools`, `/configs-targeting`, `/online-evals`) and waits for them to come back. This skill does **not** auto-invoke other skills. ## Workflow ### Minimum viable migration Stages 1–4 (audit, wrap, tools, tracker) are independently shippable. **A migration that stops after Stage 4 is complete, production-ready, and delivers the core value** — externalized prompts and model config, targeting, variation A/B testing, and Monitoring-tab metrics. Stage 5 (evaluators) is a quality-of-life addition, not a gate. Do not block a Stage-4 rollout on evaluators; ship the run-scoped tracker path, verify metrics flow, then come back for Stage 5 when the team has time to curate a dataset. That said, do not *skip* Stage 4. A migration without the tracker gives you externalized prompts but no visibility, which is most of the payoff left on the floor. ### Step 1: Audit the codebase (Stage 1) This is the first stage. It is **read-only** — no code writes, no LaunchDarkly resources created. The goal is to scan the repo and produce a structured manifest of every hardcoded value that needs to move, then hand the manifest back to the user for confirmation before any code is touched in Stage 2. Use [phase-1-analysis-checklist.md](references/phase-1-analysis-checklist.md) to scan: 1. **Language and package manager** — Python (pip/poetry/uv), TypeScript/JavaScript (npm/pnpm/yarn), Go, Ruby, .NET 2. **LLM provider** — OpenAI, Anthropic, Bedrock, Gemini, LangChain, LangGraph, CrewAI, Strands 3. **Existing LaunchDarkly usage** — any pre-existing `LDClient` or `ldclient` initialization to reuse 4. **Hardcoded model configs** — model name string literals, temperature / max_tokens / top_p, system prompts, instruction strings 5. **Template placeholders in prompts** — `.format()` calls, f-strings in prompt constants, JS/TS template literals, `%(var)s`, hand-rolled `str.replace("__VAR__", ...)`. Flag each placeholder name and its runtime-value source; all get rewritten to Mustache `{{ variable }}` in Stage 2. 6. **Externalized prompt files** — scan YAML / JSON / TOML / Markdown / `.prompt` / `.j2` files **and** prompt-template registries (`langchain.hub.pull(...)`, LangSmith `client.pull_prompt(...)`) for prompts loaded at runtime. Common shapes: CrewAI `agents.yaml` / `tasks.yaml`, LangChain Promptfiles, k8s ConfigMap overlays, Pydantic Settings classes with `prompt_*` fields. Same Mustache rewrite (sub-step 5 of Stage 2) applies if the placeholder syntax differs. See [phase-1-analysis-checklist.md § 4](references/phase-1-analysis-checklist.md). 7. **Hardcoded app-scoped knobs** — search-result limits, retry budgets, tool-timeout overrides, feature toggles, any config-dataclass field that isn't a prompt or model parameter but still governs agent behavior. These belong in `model.custom` on the variation (not `model.parameters`, which is forwarded to the provider SDK and will crash on unknown kwargs). 8. **Mode decision** — completion mode (chat messages array) or agent mode (single instructions string). Completion mode is the default and the only mode that supports judges attached in the UI. For each hardcoded target the audit finds, record: - File path and line range - Current value (model name, full prompt text, parameter dict) - Target config field (`model.name`, `model.parameters.temperature`, `messages[].content`, `instructions`) - Whether the surrounding call uses function calling / tools (drives Stage 3) - Whether the surrounding call has retry logic (affects where Stage 4 tracker calls go) This manifest is the contract for the next four stages. **Stage 1 output** (return to user as a structured summary): ``` Language: Python 3.12 Package manager: uv LLM provider: OpenAI Existing LD SDK: none Target mode: completion Hardcoded targets: - src/chat.py:42 model="gpt-4o" - src/chat.py:43 temperature=0.7, max_tokens=2000 - src/chat.py:45 system="You are a helpful assistant..." Externalized prompt files: none (or e.g. "prompts/agents.yaml — CrewAI role/goal/backstory") Prompt-template registries: none (or e.g. langchain.hub.pull("rlm/rag-prompt") at app.py:14) Coverage totals: 3 hardcoded code targets · 0 externalized prompt files · 0 registry pulls Proposed plan: single config key `chat-assistant`, mirror fallback, Stage 3 (tools) skipped (no function calling), Stage 4 (tracking) inline, Stage 5 (evals) attach built-in accuracy judge. ``` **STOP.** Present this summary, state the coverage totals out loud (e.g. "I found **N** hardcoded code targets and **M** externalized prompt files — does that match what you expected?"), and wait for the user to reply with one of four explicit forms: - **`confirm`** — proceed to Stage 2. - **`add: `** — re-run the audit with the new locations and present an updated summary. - **`fix: `** — update a target in the list (provider, mode, prompt content, etc.) and ask again. - **`stop`** — pause the migration here. Do not interpret any other word — including `skip`, `next`, `go`, `ok`, `proceed` — as confirmation; ask the user to pick one of the four forms. **This is the most important checkpoint in the workflow** — if the audit is wrong, every stage after this will be wrong. The user should cross-check the hardcoded-targets list against what they know is in the code before giving the go-ahead. ### Step 2: Wrap the call in the AI SDK (Stage 2) This is the first stage that writes code. It has nine sub-steps. 1. **Delete any hand-rolled model / tool wrappers the audit flagged.** Do this *before* installing the new SDK so the replacement lands in a repo without confusing fallback imports. The two shapes the Stage 1 audit should have surfaced: - **`load_chat_model(f"{provider}/{name}")` or any `init_chat_model(...)` wrapper.** Ships with `langchain-ai/react-agent` and many derivative repos. Delete the function and its module; the replacement is `create_langchain_model(ai_config)` (installed in the next sub-step). Leaving the wrapper in place means the next edit in this repo will import the familiar helper and silently drop variation parameters. - **Hand-rolled `resolve_tools` / `TOOL_REGISTRY` / `ALL_TOOLS` helpers that hard-code a static tool list.** Delete them; `ldai_langchain.langchain_helper.build_structured_tools(ai_config, TOOL_REGISTRY_DICT)` is the canonical replacement and gets wired in Stage 3. If you leave the hand-rolled version, both shapes will live side-by-side and the next contributor will pick the familiar one. Commit the deletion separately from the SDK install if the repo's review process benefits from it — otherwise bundle with sub-step 2. 2. **Install the AI SDK.** Detect the package manager from Step 1, then install: - Python: `launchdarkly-server-sdk` + `launchdarkly-server-sdk-ai>=0.20.0` - Node.js/TypeScript: `@launchdarkly/node-server-sdk` + `@launchdarkly/server-sdk-ai@^0.20.0` - Go: `github.com/launchdarkly/go-server-sdk/v7` + `github.com/launchdarkly/go-server-sdk/ldai` Tier-2 provider packages (install in Stage 4, only if you're using the matching provider): - OpenAI: `launchdarkly-server-sdk-ai-openai>=0.4.0` (Python) / `@launchdarkly/server-sdk-ai-openai@^0.5.5` (Node) - LangChain / LangGraph: `launchdarkly-server-sdk-ai-langchain>=0.5.0` (Python) / `@launchdarkly/server-sdk-ai-langchain@^0.5.5` (Node) - Vercel AI SDK (Node only): `@launchdarkly/server-sdk-ai-vercel@^0.5.5` - Anthropic, Gemini, Bedrock — no provider package published; use Tier-3 custom extractor (see `built-in-metrics`) 3. **Initialize `LDAIClient` once at startup.** Reuse any existing `LDClient` — do not create a second base client. Place the initialization in the same module that owns existing app config. **Python:** ```python import os import ldclient from ldclient.config import Config from ldai.client import LDAIClient # Order matters: ldclient.get() raises if called before ldclient.set_config(). # The set_config call is what initializes the singleton; .get() just returns it. sdk_key = os.environ.get("LD_SDK_KEY") if sdk_key: ldclient.set_config(Config(sdk_key)) else: # Missing key: init in offline mode so the app still starts and the fallback # path runs on every call. Never raise at import time for a missing env var — # that turns a config gap into a boot failure. import logging logging.getLogger(__name__).warning( "LD_SDK_KEY not set; configs will use fallback values only." ) ldclient.set_config(Config("", offline=True)) ai_client = LDAIClient(ldclient.get()) ``` **Node.js/TypeScript:** ```typescript import { init } from '@launchdarkly/node-server-sdk'; import { initAi } from '@launchdarkly/server-sdk-ai'; // The Node SDK does not have an explicit offline mode — a missing or invalid // key fails fast during waitForInitialization, and every agent_config / // completion_config call returns the fallback. Log a warning; do not throw. if (!process.env.LD_SDK_KEY) { console.warn('LD_SDK_KEY not set; configs will use fallback values only.'); } const ldClient = init(process.env.LD_SDK_KEY ?? 'sdk-offline'); await ldClient.waitForInitialization({ timeout: 10 }).catch(() => { // Swallow init failures in offline mode; fallback path runs. }); const aiClient = initAi(ldClient); ``` 4. **Hand off to `configs-create`.** Print the extracted model, prompt/instructions, parameters, and mode from the Stage 1 manifest, then tell the user: *"Run `/configs-create` with these inputs, then come back here."* Supply the config key you want the code to call (e.g. `chat-assistant`). Do not attempt to auto-invoke the sibling skill — wait for the user to finish it before continuing. **After `configs-create` finishes, the user must also run `/configs-targeting` to promote the new variation to fallthrough.** A freshly created variation returns `enabled=False` to every consumer until targeting is updated. Skip this and Stage 2 verification (sub-step 9 below) will silently take the fallback path on every request. 5. **Rewrite template placeholders to Mustache syntax.** If the hardcoded prompt interpolates runtime values with Python `.format()`, f-strings, JS template literals, or any other non-Mustache syntax (e.g. `{system_time}`, `${userName}`, `%(topic)s`), rewrite every placeholder to `{{ variable }}` Mustache form. Do this in **both** the file you're about to send to `/configs-create` *and* the fallback string you'll write in sub-step 6. The AI SDK interpolates variables through a Mustache renderer on the LD-served path *and* the fallback path using the fourth-argument `variables` dict to `completion_config(...)` / `completionConfig(...)`. Leaving a Python-style `{system_time}` literal in the fallback ships a silent regression when LaunchDarkly is unreachable — the renderer won't match the single-brace form and the literal `{system_time}` goes to the provider as part of the prompt. **Before:** ```python SYSTEM_PROMPT = "You are a helpful assistant. The time is {system_time}." prompt = SYSTEM_PROMPT.format(system_time=datetime.now().isoformat()) ``` **After (in source):** ```python SYSTEM_PROMPT = "You are a helpful assistant. The time is {{ system_time }}." # .format() is removed at the call site — the SDK interpolates via `variables` config = ai_client.completion_config( CONFIG_KEY, context, fallback, variables={"system_time": datetime.now().isoformat()}, ) ``` Common shapes to rewrite: - Python `"{var}"` / `"{var!s}"` / `"%(var)s"` → `"{{ var }}"` - JS/TS `` `${var}` `` template literals inside prompt strings → `"{{ var }}"` - Any hand-rolled `str.replace("__VAR__", value)` scheme → `"{{ var }}"` See [fallback-defaults-pattern.md § Template placeholders](references/fallback-defaults-pattern.md) for the fallback-specific variant. 6. **Build the fallback.** Mirror the hardcoded values you extracted. Use `AICompletionConfigDefault` / `AIAgentConfigDefault` in Python, plain object literals in Node. See [fallback-defaults-pattern.md](references/fallback-defaults-pattern.md) for inline, file-backed, and bootstrap-generated patterns. **Python fallback (completion mode):** ```python from ldai.client import AICompletionConfigDefault, ModelConfig, ProviderConfig, LDMessage fallback = AICompletionConfigDefault( enabled=True, model=ModelConfig(name="gpt-4o", parameters={"temperature": 0.7, "max_tokens": 2000}), provider=ProviderConfig(name="openai"), messages=[LDMessage(role="system", content="You are a helpful assistant...")], ) ``` 7. **Replace the hardcoded call site.** Swap the hardcoded model/prompt/params for a `completion_config` / `completionConfig` (or `agent_config` / `agentConfig`) call, then read the returned fields into the existing provider call. Keep the provider call intact. **Python — before:** ```python response = openai_client.chat.completions.create( model="gpt-4o", temperature=0.7, max_tokens=2000, messages=[ {"role": "system", "content": "You are a helpful assistant..."}, {"role": "user", "content": user_input}, ], ) ``` **Python — after:** ```python context = Context.builder(user_id).set("email", user.email).build() config = ai_client.completion_config("chat-assistant", context, fallback) if not config.enabled: return disabled_response() params = config.model.parameters or {} response = openai_client.chat.completions.create( model=config.model.name, temperature=params.get("temperature"), max_tokens=params.get("max_tokens"), messages=[m.to_dict() for m in (config.messages or [])] + [ {"role": "user", "content": user_input}, ], ) ``` **Python — after (agent mode)** — for LangGraph, CrewAI, or any framework that takes a goal/instructions string: ```python context = Context.builder(user_id).kind("user").build() config = ai_client.agent_config("support-agent", context, FALLBACK) if not config.enabled: return disabled_response() # config is a single AIAgentConfig object — NOT a (config, tracker) tuple. # Obtain the tracker once per execution via the factory: tracker = config.create_tracker() model_name = f"{config.provider.name}/{config.model.name}" instructions = config.instructions params = config.model.parameters or {} # Pass model_name + instructions into your framework's agent constructor. # Example: LangGraph prebuilt agent (Python — `from langchain.agents import create_agent`; # this replaces `langgraph.prebuilt.create_react_agent`, deprecated in LangGraph 1.0 # and removed in 2.0. Same return shape; `prompt=` was renamed to `system_prompt=`.) # agent = create_agent( # create_langchain_model(config), # forwards every variation parameter # TOOLS, # Stage 3 will replace this with a config.tools loader # system_prompt=instructions, # ) ``` See [before-after-examples.md](references/before-after-examples.md) for full Python OpenAI, Node Anthropic, and LangGraph agent-mode paired snippets. 8. **Check `config.enabled`.** If it returns `False`, handle the disabled path without crashing and without calling the provider. The check is required — not optional. 9. **Verify.** Run the app with a valid `LD_SDK_KEY`; confirm the call succeeds and the response matches pre-migration output. Then temporarily set `LD_SDK_KEY=sdk-invalid` (or unset it) and confirm the fallback path runs without error. Both paths must work before moving to Stage 3. Delegate: **`configs-create`** (sub-step 4). ### Step 3: Move tools into the config (Stage 3) Skip this step if the audited app has no function calling / tools. Otherwise: 1. **Enumerate the tools currently registered.** Common shapes to look for: - `openai.chat.completions.create(tools=[...])` — OpenAI direct - `anthropic.messages.create(tools=[...])` — Anthropic direct - `create_agent(llm, tools=[...], system_prompt=...)` — LangGraph prebuilt (Python, `langchain.agents`; replaces deprecated `langgraph.prebuilt.create_react_agent`) - `createReactAgent({ llm, tools: [...] })` — LangGraph.js prebuilt (Node, `@langchain/langgraph/prebuilt`) - `Agent(tools=[...])` — CrewAI - `Agent(tools=[...])` — Strands (Python `@tool`-decorated callables passed through the constructor; TS SDK uses Zod-schema tools) - **Custom `StateGraph`** — module-level `TOOLS = [...]` list referenced in **both** `model.bind_tools(TOOLS)` and `ToolNode(TOOLS)`. This is the `langchain-ai/react-agent` template shape; the list is usually in a `tools.py` module. Grep for `bind_tools(` and `ToolNode(` together — they will point at the same list. Record each tool's name, description, and JSON schema. For LangChain/LangGraph tools defined with `@tool`, extract the schema via `tool.args_schema.model_json_schema()` (or the equivalent Pydantic `model_json_schema()` call). For plain async callables used as tools (common in custom StateGraph shapes), LangChain infers the schema from the function signature at bind time — extract it via `StructuredTool.from_function(fn).args_schema.model_json_schema()`. Do not hand-write the schema. 2. **Hand off to `tools`.** Print the extracted tool names, descriptions, and schemas, then tell the user: *"Run `/tools` with these tools and the variation key, then come back here."* The sibling skill creates tool definitions (`create-ai-tool`) and attaches them to the variation (`update-ai-config-variation`). Wait for the user to finish before proceeding to sub-step 3. Do not auto-invoke. 3. **Replace the hardcoded tools array at the call site** with a read from `config.tools` (or the SDK equivalent for your language). Load the actual implementation functions dynamically from the tool names — see [agent-mode-frameworks.md](references/agent-mode-frameworks.md) for the dynamic-tool-factory pattern from the devrel agents tutorial. **For custom `StateGraph` shapes**, you must update **both** call sites: `.bind_tools(TOOLS)` and `ToolNode(TOOLS)` must both read from the same `config.tools`-derived list. Forgetting one leaves the LLM seeing the new tools but the executor still running the old ones, or vice versa. 4. **Verify.** Run the app; confirm the tool flows still execute correctly. `get-ai-config` (via the delegate) confirms the tools are attached server-side. Delegate: **`tools`** (sub-step 2). ### Step 4: Instrument the tracker (Stage 4) Delegate: **`built-in-metrics`** wires the per-request `tracker.track_*` calls (duration, tokens, success/error, feedback) around the provider call. Use **`custom-metrics`** alongside it if the app needs business metrics beyond the built-in agent ones. Note: do not confuse this with `launchdarkly-metric-instrument`, which is for `ldClient.track()` feature metrics — a different API. See [sdk-ai-tracker-patterns.md](references/sdk-ai-tracker-patterns.md) for the full per-method Python + Node matrix that the delegate skill draws on. Hand off: print the config key, variation key, provider, and whether the call is streaming, then tell the user: *"Run `/built-in-metrics` with these inputs, then come back here."* Do not auto-invoke. Return here for sub-step 5 (verify) once they're done. 1. **Create the tracker.** Obtain a per-execution tracker via the factory on the config returned in Stage 2: `tracker = config.create_tracker()` (Python) or `const tracker = aiConfig.createTracker();` (Node). Call the factory **once per user turn** and reuse the returned `tracker` for every tracking call in that turn — each call mints a fresh `runId` that tags every event emitted from the turn so they can be correlated via exported events or downstream queries. (The Monitoring tab aggregates today; run-level grouping is a downstream concern — but the `runId` is also what the SDK's at-most-once guards are keyed on, so minting a new one mid-turn breaks the guard semantics regardless of where the events end up.) **Where to call the factory depends on the call shape:** - **Completion mode / one-shot provider call:** mint the tracker right after `completion_config(...)` returns, in the same function that handles the request. - **Agent mode with a ReAct loop (LangGraph, LangChain, custom):** mint the tracker in a dedicated `setup_run` entry node that executes **once** before the loop, stash it on graph state, and read it from state in `call_model` / tool handlers / a terminal `finalize` node. Emitting `track_duration` / `track_tokens` / `track_success` inside the loop body will trip the at-most-once guards. See [agent-mode-frameworks.md § Custom `StateGraph` (run-scoped architecture)](references/agent-mode-frameworks.md) for the full `setup_run` + `call_model` + `finalize` pattern. - **Managed runner (Tier 1):** skip this step entirely. `ManagedModel` mints the tracker internally per `run()` / `invoke()`. Move to sub-step 4 if that's what the app uses. 2. **Pick a tier from the four-tier ladder.** See [sdk-ai-tracker-patterns.md § Tier decision table](references/sdk-ai-tracker-patterns.md) for the full table (chat loop → Tier 1; provider-package call → Tier 2; custom extractor → Tier 3; streaming/manual → Tier 4). 3. **Wire the chosen tier.** The delegate skill has full Python + Node examples for each tier plus per-provider files. A condensed Tier 2/3 example for reference — OpenAI via the provider package: **Python:** ```python from ldai_openai import get_ai_metrics_from_response import openai client = openai.OpenAI() tracker = config.create_tracker() def call_openai(): return client.chat.completions.create( model=config.model.name, messages=[{"role": "system", "content": config.messages[0].content}, {"role": "user", "content": user_prompt}], ) # Exceptions are tracked automatically — track_metrics_of catches # exceptions, records tracker.track_error(), and re-raises. Wrap your # own try/except only for local handling (logging, fallback). response = tracker.track_metrics_of(get_ai_metrics_from_response, call_openai) ``` **Node:** ```typescript import { getAIMetricsFromResponse } from '@launchdarkly/server-sdk-ai-openai'; const tracker = aiConfig.createTracker(); // Exceptions are tracked automatically — trackMetricsOf catches // exceptions, records tracker.trackError(), and re-throws. const response = await tracker.trackMetricsOf( getAIMetricsFromResponse, () => openaiClient.chat.completions.create({ model: aiConfig.model!.name, messages: [...aiConfig.messages, { role: 'user', content: userPrompt }], }), ); ``` For Anthropic direct, Bedrock (no provider package), Gemini, and custom HTTP, write a small extractor returning `LDAIMetrics` — see the delegate skill's [anthropic-tracking.md](../built-in-metrics/references/anthropic-tracking.md), [bedrock-tracking.md](../built-in-metrics/references/bedrock-tracking.md), and [gemini-tracking.md](../built-in-metrics/references/gemini-tracking.md). LangChain single-node and LangGraph go through the `launchdarkly-server-sdk-ai-langchain` / `@launchdarkly/server-sdk-ai-langchain` provider package. Build the model with `create_langchain_model(config)` (Python) / `createLangChainModel(config)` (Node) — both forward all variation parameters — and track with `get_ai_metrics_from_response` / `getAIMetricsFromResponse`. See [langchain-tracking.md](../built-in-metrics/references/langchain-tracking.md). 4. **Wire feedback tracking if the app has thumbs-up/down UI.** Both SDKs expose `trackFeedback` with a `{kind}` argument. **Python:** ```python from ldai.tracker import FeedbackKind tracker.track_feedback({"kind": FeedbackKind.Positive}) ``` **Node:** ```typescript import { LDFeedbackKind } from '@launchdarkly/server-sdk-ai'; tracker.trackFeedback({ kind: LDFeedbackKind.Positive }); ``` **Deferred feedback across processes.** If the thumbs-up UI fires in a different process than the one that produced the response, do **not** call `create_tracker()` again in the consumer — that mints a new `runId`. Persist the tracker's resumption token (`tracker.resumption_token` in Python, `tracker.resumptionToken` in Node) alongside the message, then rehydrate the tracker with `LDAIConfigTracker.from_resumption_token(...)` (Python) or `aiClient.createTracker(token, context)` (Node) in the feedback handler. 5. **Verify.** Hit the wrapped endpoint in staging, then open the config in LaunchDarkly → Monitoring tab. Duration, token, and generation counts should appear within 1–2 minutes. If nothing shows up, walk the checklist in [sdk-ai-tracker-patterns.md](references/sdk-ai-tracker-patterns.md) under "Troubleshooting." ### Step 5: Attach evaluations (Stage 5) 1. **Decide between three evaluation paths.** This is the most commonly misunderstood stage — there are **three** paths, not two, and the right default for a migration context is often the one people skip. | Path | When to use | Supports agent mode? | |------|-------------|---------------------| | **Offline eval** (recommended default for migration) | Pre-ship regression: run a fixed dataset through the new variation in the LD Playground and score against baseline. Best fit for migration because you want to prove the new config behaves at least as well as the hardcoded version before shipping. | Yes — all modes | | **UI-attached auto judges** | Attach one or more judges to a variation in the LD UI; judges run on sampled live requests automatically. Zero code changes. | Completion mode only (the UI widget is completion-only today) | | **Programmatic direct-judge** | Call `ai_client.create_judge(...)` inside the request handler and `judge.evaluate(input, output)` on each call. Adds per-request cost and code complexity. Best for continuous live scoring of workflows where sampled auto-judges aren't enough. | Yes — all modes (the SDK handles both identically) | **Most migration users should start with offline eval**, then add programmatic direct-judge only if they need continuous live scoring after the rollout is stable. 2. **For agent-mode migrations, default to offline eval.** UI-attached auto judges are completion-mode only today. The documented path for agent mode is either (a) **offline regression** via the LD Playground + Datasets (works for all modes), or (b) **programmatic direct-judge** wired into the call site. Generate a starter dataset CSV from the audit manifest (one representative input per row) and point the user at the [Offline Evals guide](https://docs.launchdarkly.com/guides/ai-configs/offline-evaluations) for the Playground walkthrough. Only wire programmatic direct-judge into production code if the user explicitly asks for continuous live scoring. **Recommended offline-eval shape for a migration:** - Run the `default` variation (or whichever variation mirrors the pre-migration hardcoded behavior) against the dataset first — this is the baseline. - Clone it into a second variation pointing at a **different model family** (e.g., if the baseline is `anthropic/claude-sonnet-4-5`, clone to `openai/gpt-4o` or `openai/gpt-4o-mini`). The comparison is most informative across families, not across siblings. - Attach the built-in **Accuracy** judge with a pass threshold of **0.85**, and run both variations against the same dataset. - Promote the winner to fallthrough via `/configs-targeting` only if it beats the baseline on Accuracy and does not regress on Relevance or Toxicity. Write this shape into the project's `datasets/README.md` (or equivalent) so the comparison pattern is reproducible after the migration ships. 3. **Hand off to `online-evals`** — only for UI-attached judges (completion mode) or to create custom judge configs that will be referenced by the programmatic path. Tell the user: *"Run `/online-evals` with these inputs, then come back here."* Do not auto-invoke. Pass: - The parent config key and variation key - A list of built-in judges (Accuracy, Relevance, Toxicity) or custom judge keys to create/attach - Target environment The delegate handles creating custom judge configs, attaching them via the variation PATCH endpoint, and setting fallthrough on each judge config. Offline eval does **not** go through this delegate — it's a Playground workflow, not an API write. 4. **For programmatic direct-judge: wire `create_judge` + `evaluate` + `track_judge_result`.** This is the only path at Stage 5 that writes code. The Python shape: ```python from ldai.client import AIJudgeConfigDefault judge = ai_client.create_judge( judge_key, # judge config key in LD ld_context, AIJudgeConfigDefault(enabled=False), # fallback: skip eval on SDK miss ) if judge and judge.enabled: result = await judge.evaluate( input_text, output_text, sampling_rate=0.25, # optional; default 1.0 (always eval) ) if result.sampled: tracker.track_judge_result(result) ``` Four rules: - **`create_judge` returns `Optional[Judge]`.** Always guard with `if judge and judge.enabled:` — it returns `None` if the judge config is disabled for the context or the provider is missing. A direct `.evaluate()` on a `None` return will raise `AttributeError`. - **Pass `AIJudgeConfigDefault`**, not `AICompletionConfigDefault`. The `create_judge` `default` parameter is typed `Optional[AIJudgeConfigDefault]`; passing the completion type will not type-check and is a doc-level bug in some older examples. - **`sampling_rate` is a parameter on `evaluate()`**, not on `create_judge`. It defaults to `1.0` (evaluate every call). For live paths, pass something lower (0.1–0.25) to control cost. - **`evaluate()` returns a `JudgeResult`** (never `None`). Check `result.sampled` to know whether the evaluation actually ran, and call `track_judge_result(result)`. Node uses `trackJudgeResult(result)` and `LDJudgeResult` with the same `sampled` field. **Ask the user which judge config key to use.** LaunchDarkly ships three built-in judges — Accuracy, Relevance, Toxicity — but the actual config **keys** for the built-ins are not canonical SDK constants and aren't documented. Have the user open **AgentControl > Library** in the LD UI and copy the key of the judge they want to reference, or create a custom judge config via `configs-create` first. 5. **Verify.** - **UI-attached auto judges:** trigger a request in staging, open the Monitoring tab → "Evaluator metrics" dropdown. Scores appear within 1–2 minutes at the configured sampling rate. - **Programmatic direct-judge:** hit the wrapped endpoint and confirm `track_judge_result` lands on the parent config's Monitoring tab. - **Offline eval:** run the dataset through the LD Playground, compare baseline vs new-variation scores side by side. No runtime wiring required. Delegate: **`online-evals`** (sub-step 3, optional — only for UI-attached judges or custom-judge creation; offline eval doesn't delegate). ## Edge Cases | Situation | Action | |-----------|--------| | App already initializes `LDClient` for feature flags | Reuse it — pass the existing client to `LDAIClient()` / `initAi()`, do not create a second client | | App uses LangChain `ChatOpenAI(model=...)` | Replace the hand-rolled model construction with `create_langchain_model(config)` (Python) or `createLangChainModel(config)` (Node). Do not read `config.model.name` and pass it to `ChatOpenAI(model=...)` by hand — that pattern drops every variation parameter except the ones you explicitly name | | Retry wrapper around the provider call | The tracker is minted once at the top of the user turn; the retry loop is inside that scope. Every retry attempt shares the same `runId`. Tracker calls (`track_duration` / `track_tokens` / `track_success` / `track_error`) live *outside* the retry body — one call at the end of the turn, on the success path or the final-failure path | | App has no tools — Stage 3 skipped | Move directly from Stage 2 verification to Stage 4 (tracking) | | Mode mismatch: user said agent, audit shows one-shot chat | Choose completion mode unless the app uses a LangGraph prebuilt agent (`langchain.agents.create_agent` in Python or `createReactAgent` in Node), CrewAI `Agent`, Strands `Agent`, or a similar goal-driven framework | | App uses Strands Agents (Python) | Agent mode. Build a `create_strands_model` dispatcher keyed on `agent_config.provider.name` that returns `AnthropicModel(model_id=..., max_tokens=...)` or `OpenAIModel(model_id=..., params=...)`. Drop `parameters.tools` before passing params to the model class — Strands receives tools via `Agent(tools=[...])`. Tracking is Tier 3: wrap `invoke_async` with `tracker.track_duration_of(...)` and record tokens from `result.metrics.accumulated_usage`. See [agent-mode-frameworks.md § Strands Agent](references/agent-mode-frameworks.md) and [strands-tracking.md](../built-in-metrics/references/strands-tracking.md) | | Strands app on TypeScript | TS SDK ships `BedrockModel` and `OpenAIModel` only — cannot serve Anthropic-backed variations. Use the Python SDK if multi-provider variations are required | | TypeScript app using Anthropic SDK | No `trackAnthropicMetrics` helper exists. Use Tier 3: `trackMetricsOf` with a small custom extractor that reads `response.usage.input_tokens` / `response.usage.output_tokens` and returns `LDAIMetrics`. See [anthropic-tracking.md](../built-in-metrics/references/anthropic-tracking.md) in the `built-in-metrics` skill for the exact extractor | | Fallback would silently crash because `LD_SDK_KEY` is missing | Log a startup warning; proceed with the fallback. Never raise at import time | | Multi-agent graph (supervisor + workers) | Stop after migrating a single agent. Agent Graph Definitions are available in **both** SDKs — Python via `launchdarkly-server-sdk-ai.agent_graph` and Node via the graph API in `@launchdarkly/server-sdk-ai`. Read [agent-graph-reference.md](references/agent-graph-reference.md) for the graph-level migration path — it is deliberately out of this skill's main scope | | Single-agent (ReAct, tool loop) + agent mode | Default to offline eval via the LD Playground + Datasets for Stage 5. UI-attached judges are completion-only today, and programmatic direct-judge adds per-call cost that is usually not worth it until after the migration is live and stable. Point at the [Offline Evals guide](https://docs.launchdarkly.com/guides/ai-configs/offline-evaluations) | | Tool with a Pydantic `args_schema` (LangChain `@tool`) | Extract the schema via `tool.args_schema.model_json_schema()`; do not hand-write the JSON schema for the delegate | | Custom `StateGraph` with module-level `TOOLS` list bound via `.bind_tools(TOOLS)` and run through `ToolNode(TOOLS)` (e.g. the `langchain-ai/react-agent` template) | Find the `TOOLS` list (usually in a separate `tools.py` module). Extract schemas the same way. Swap **both** call sites — `.bind_tools(...)` and `ToolNode(...)` — to read from the same `config.tools`-derived list | | App has already externalized config into a `Context` dataclass with env-var fallback (e.g. `react-agent` template's `context.py`) | Replace the consumers of `runtime.context.model` / `runtime.context.system_prompt` with `ai_client.agent_config(...)` and read from the returned `AIAgentConfig`. **Empty the dataclass** rather than keeping it as the fallback shape — the canonical fallback is `FALLBACK = AIAgentConfigDefault(...)` in Python (a top-level constant near the `agent_config` call), not a parallel Python dataclass. Two sources of truth for fallback values drift. An empty `Context` is a placeholder satisfying LangGraph's `context_schema` requirement only; `thread_id` and any other per-request plumbing comes through `config: RunnableConfig` instead (see [agent-mode-frameworks.md § Custom `StateGraph`](references/agent-mode-frameworks.md)) | ## What NOT to Do These are ordered by how likely they are to show up as a first-run failure. The first three rules — about tracker and config lifetime — account for most of the "migration looks done but the Monitoring tab is fragmented / wrong" reports. ### Tracker and config lifetime (most common failure mode) - **Don't call `create_tracker()` / `createTracker()` more than once per user turn.** One turn = the full request/response cycle including every ReAct iteration, tool call, and retry. See Stage 4 Step 1 for the canonical placement in each app shape (completion / agent loop / managed runner). - **Don't call `track_duration` / `track_tokens` / `track_success` / `track_error` / `track_time_to_first_token` inside a loop body.** These are at-most-once per tracker; second calls are dropped. Accumulate inside the loop, emit once in a terminal/finalize node. Per-event methods (`track_tool_call`, `track_tool_calls`, `track_feedback`, `track_judge_result`) are safe to call repeatedly. Full matrix: [sdk-ai-tracker-patterns.md § At-most-once guards](references/sdk-ai-tracker-patterns.md). - **Don't call `agent_config()` / `completion_config()` more than once per user turn.** Each call is a flag evaluation and emits a `$ld:ai:agent:config` event. Re-fetching inside a loop step or a tool body inflates agent-config counts on the Monitoring tab and lets a mid-turn targeting change swap the variation between LLM calls in a single turn. Resolve once at the top, stash on state, and have every subsequent consumer read from state. Tools that need variation-scoped knobs should use the tool-factory pattern (`make_search(ai_config)` that closes over the knob at setup time) — see [agent-mode-frameworks.md § Getting knobs into tools](references/agent-mode-frameworks.md). - Don't cache the config object *across* requests — resolve once per turn, yes, but still resolve once per turn. Caching at module scope defeats the targeting-change mechanism entirely. - Don't delete the fallback once LaunchDarkly is wired up. It is required for the `enabled=False` and SDK-unreachable paths. - Don't tuple-unpack the return of `completion_config` / `agent_config` / `completionConfig` / `agentConfig`. They return a **single** config object (e.g. `AIAgentConfig`, `AICompletionConfig`), not `(config, tracker)`. Obtain the tracker by calling `config.create_tracker()` / `aiConfig.createTracker()`. LLMs hallucinate both the tuple shape and a `config.tracker` property — the actual API is a factory. ### LangChain / LangGraph patterns (second most common failure mode) - **If the repo already contains a `load_chat_model(f"{provider}/{name}")` helper, delete it — don't just avoid using it.** This exact shape ships with `langchain-ai/react-agent` and is copied into dozens of derivative repos; look for `utils.load_chat_model`, `utils.build_model`, or any one-arg `init_chat_model` wrapper that splits a `"provider/model"` string. Re-using it is the first-run failure mode: every variation parameter (temperature, max_tokens, top_p, stop sequences) silently drops on the floor because `init_chat_model` only receives the name and provider. `create_langchain_model(ai_config)` is a one-for-one replacement that forwards the whole `model.parameters` dict. Replace every call site, then delete the wrapper file-side so the next reader can't reach for it. - **Same rule applies to hand-rolled `resolve_tools` / `TOOL_REGISTRY` / `ALL_TOOLS` helpers.** If the template already has a `resolve_tools(tool_keys)` or an `ALL_TOOLS` module-level list, import `build_structured_tools` from `ldai_langchain.langchain_helper` and delete the hand-rolled version. `build_structured_tools(ai_config, TOOL_REGISTRY_DICT)` reads `ai_config.model.parameters.tools` and wraps the matching callables as LangChain `StructuredTool`s with the LD tool key as the `StructuredTool.name` — so `ToolNode` lookup works without a second mapping. Don't leave both in the repo. - Don't put app-scoped knobs directly in `model.parameters`. `create_langchain_model` forwards every key in `parameters` to the provider SDK via `init_chat_model`, so a `max_search_results` / `retry_budget` / `feature_toggle` entry will crash the provider with an unexpected-keyword-argument error. The correct home is `model.custom`, which the provider helpers ignore and the app reads via `ai_config.model.get_custom("key")`. The MCP `update-ai-config-variation` tool does not currently expose top-level `custom`, so pick one of two paths: (a) PATCH the variation via the REST API to set `model.custom` directly, or (b) set it via MCP inside `parameters.custom` (as a nested dict) and use a defensive accessor that reads both locations. Full walk-through with code samples in [langchain-tracking.md § MCP caveat](../built-in-metrics/references/langchain-tracking.md). - Don't re-encode tool schemas inside the fallback. When LaunchDarkly is unreachable the fallback should run without tools (or with whatever minimal provider-bound parameters the app needs to keep operating). Building a `_FALLBACK_TOOLS` array that duplicates the config's tool schema re-introduces the hardcoded config the migration was supposed to move out of code. - Don't import `LaunchDarklyCallbackHandler` from `ldai.langchain` — neither the class nor the dotted module path exists. The Python LangChain helper package is `ldai_langchain` (top-level module, underscore). Use `create_langchain_model(config)` + `track_metrics_of_async(get_ai_metrics_from_response, lambda: llm.ainvoke(messages))` as the canonical pattern. ### Stage / handoff discipline - Don't skip Step 1 even when the user says "just wrap it." Without the audit, the fallback will drift from the hardcoded behavior. - Don't delegate to `configs-create` before extracting the prompt and model — the delegate needs them as inputs. - Don't try to attach tools during initial `setup-ai-config`. Tool attachment is a separate step owned by `tools`. - Don't claim you "delegated to `configs-create`" or any other sibling skill. This skill does not auto-invoke. At each handoff, print the inputs and tell the user to run the sibling slash-command, then wait. Anything else misleads the user about what just happened. - Don't skip the `/configs-targeting` step between Stage 2 and Stage 4. A freshly created variation returns `enabled=False` until targeting promotes it to fallthrough — Stage 2 verification will silently take the fallback path on every request. - Don't attempt a multi-agent graph migration in one pass. Migrate a single agent first; use [agent-graph-reference.md](references/agent-graph-reference.md) as the next-step read. ### Stage 5 evaluations - Don't wire evals before the tracker is in place. Judges score traffic; without Stage 4 traffic, there is nothing to judge. - Don't frame Stage 5 as "either UI or programmatic." There are **three** paths: offline eval (recommended default for migration), UI-attached auto judges (completion-mode only), and programmatic direct-judge. Offline eval is the one most people skip and usually the right starting point. - Don't pass `sampling_rate` to `create_judge` — it's a parameter on `Judge.evaluate()`, not `create_judge()`. - Don't hardcode judge config keys (`"accuracy-judge"`, `"relevance-judge"`, etc). The built-in keys are not canonical SDK constants; ask the user to look them up in **AgentControl > Library** in the LD UI. - Don't forget the `if judge and judge.enabled:` guard after `create_judge`. It returns `Optional[Judge]` and returns `None` when the judge config is disabled for the context. ### API surface gotchas - Don't use `launchdarkly-metric-instrument` for Stage 4 (tracking). That skill is for `ldClient.track()` feature metrics, not agent `tracker.track_*` calls — they are different APIs. - Don't use `track_request()` in Python — it does not exist in `launchdarkly-server-sdk-ai`. Use `track_metrics_of` with a provider-package or custom extractor, or drop to explicit `track_duration` + `track_tokens` + `track_success` / `track_error` if you're on the streaming path. - Don't pass `graph_key=...` to `tracker.track_*()` methods in Python — it is not an accepted argument. Trackers obtained inside a graph traversal are automatically configured with the correct graph key. ## Related Skills - `configs-create` — called by Stage 2 to create the config - `tools` — called by Stage 3 to create and attach tool definitions - `online-evals` — called by Stage 5 to attach judges - `configs-variations` — add variations for A/B testing after migration is complete - `configs-targeting` — roll out new variations to users after migration is complete - `configs-update` — modify config properties as your app evolves - `launchdarkly-metric-instrument` — for `ldClient.track()` feature metrics (NOT for agent tracker calls) ## References - [phase-1-analysis-checklist.md](references/phase-1-analysis-checklist.md) — Step 1 audit checklist, grep patterns, SDK routing table, mode decision tree - [before-after-examples.md](references/before-after-examples.md) — Paired hardcoded-to-wrapped snippets for Python OpenAI, Node Anthropic, Python LangGraph - [sdk-ai-tracker-patterns.md](references/sdk-ai-tracker-patterns.md) — Every `tracker.track_*` method in Python and Node side by side, auto-helper matrix, and common gotchas - [agent-mode-frameworks.md](references/agent-mode-frameworks.md) — How to wire `agent_config` into LangGraph, CrewAI, and custom react loops; dynamic tool loading pattern - [fallback-defaults-pattern.md](references/fallback-defaults-pattern.md) — Three fallback patterns (inline, file-backed, bootstrap-generated) and when to use each - [agent-graph-reference.md](references/agent-graph-reference.md) — Out-of-scope pointer doc for multi-agent migrations --- ## 21. online-evals - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/online-evals/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/online-evals/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/online-evals.md - **Licence:** Other — https://github.com/launchdarkly/ai-tooling Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/online-evals/README.md --- name: online-evals description: Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores. compatibility: Requires LaunchDarkly API access token with ai-configs:write permission. SDK versions Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated `track_judge_result` / `trackJudgeResult` API. metadata: author: launchdarkly version: "0.1.0" --- # Config Online Evaluations Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0. ## Prerequisites - LaunchDarkly account with AgentControl enabled - API access token with write permissions - Existing config with variations (use `configs-create` skill) - For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+ ## API Key Detection 1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY` 2. **Check MCP config** - Claude: `~/.claude/config.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY` 3. **Prompt user** - Only if detection fails ## Core Concepts ### What Are Judges? Judges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results: ```json { "score": 0.85, "reasoning": "Answered correctly with one minor omission" } ``` ### Built-in Judges LaunchDarkly provides three pre-configured judges: | Judge | Metric Key | Measures | |-------|-----------|----------| | Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is | | Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request | | Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) | ### Completion Mode Only Judges can only be attached to **completion mode** configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK. ### Restrictions - Cannot attach judges to judges (no recursion) - Cannot attach multiple judges with the same metric key to a single variation - Cannot view/edit model parameters or tools on judge variations ## Workflow ### Step 1: Create Custom Judges (Optional) For domain-specific evaluation, create judge configs: ```bash # Create judge config curl -X POST "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json" \ -H "LD-API-Version: beta" \ -d '{ "key": "security-judge", "name": "Security Judge", "mode": "judge", "evaluationMetricKey": "security", "isInverted": false }' ``` > **Note:** Set `isInverted: true` for metrics like toxicity where 0.0 is better. Then add a variation with the evaluation prompt: ```bash curl -X POST "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/security-judge/variations" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json" \ -H "LD-API-Version: beta" \ -d '{ "key": "default", "name": "Default", "messages": [ { "role": "system", "content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection." } ], "modelConfigKey": "OpenAI.gpt-4o-mini", "model": { "parameters": { "temperature": 0.3 } } }' ``` ### Step 2: Attach Judges to Variations Use the variation PATCH endpoint: ```bash curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/{configKey}/variations/{variationKey}" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json" \ -H "LD-API-Version: beta" \ -d '{ "judgeConfiguration": { "judges": [ {"judgeConfigKey": "security-judge", "samplingRate": 1.0}, {"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5} ] } }' ``` > **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges. ### Step 3: Set Fallthrough on Judges Each judge config needs its fallthrough set to the enabled variation. Configs default to the "disabled" variation (index 0). > **Note:** `turnTargetingOn` does not work for configs. Use `updateFallthroughVariationOrRollout` instead. ```bash # First get the variation ID for "Default" from GET targeting response curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/security-judge/targeting" \ -H "Authorization: {api_token}" \ -H "Content-Type: application/json; domain-model=launchdarkly.semanticpatch" \ -H "LD-API-Version: beta" \ -d '{ "environmentKey": "production", "instructions": [{ "kind": "updateFallthroughVariationOrRollout", "variationId": "your-default-variation-uuid" }] }' ``` ## Python Implementation ```python import requests import os from typing import Optional class AIConfigJudges: """Manager for config judge attachments""" def __init__(self, api_token: str, project_key: str): self.api_token = api_token self.project_key = project_key self.base_url = "https://app.launchdarkly.com/api/v2" self.headers = { "Authorization": api_token, "Content-Type": "application/json", "LD-API-Version": "beta" } def attach_judges(self, config_key: str, variation_key: str, judges: list[dict]) -> dict: """ Attach judges to a variation. Args: config_key: config key variation_key: Variation key judges: List of {"judgeConfigKey": str, "samplingRate": float} """ url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/variations/{variation_key}" response = requests.patch(url, headers=self.headers, json={ "judgeConfiguration": {"judges": judges} }) if response.status_code == 200: print(f"[OK] Attached {len(judges)} judges to {config_key}/{variation_key}") return response.json() print(f"[ERROR] {response.status_code}: {response.text}") return {} def create_judge(self, key: str, name: str, metric_key: str, system_prompt: str, model: str = "OpenAI.gpt-4o-mini", is_inverted: bool = False) -> dict: """ Create a judge config. Args: key: Judge config key name: Display name metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key}) system_prompt: Evaluation instructions is_inverted: True if lower scores are better (e.g., toxicity) """ # Create config config_url = f"{self.base_url}/projects/{self.project_key}/ai-configs" response = requests.post(config_url, headers=self.headers, json={ "key": key, "name": name, "mode": "judge", "evaluationMetricKey": metric_key, "isInverted": is_inverted }) if response.status_code not in [200, 201]: print(f"[ERROR] Creating config: {response.text}") return {} # Create variation var_url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{key}/variations" response = requests.post(var_url, headers=self.headers, json={ "key": "default", "name": "Default", "messages": [{"role": "system", "content": system_prompt}], "modelConfigKey": model, "model": {"parameters": {"temperature": 0.3}} }) if response.status_code in [200, 201]: print(f"[OK] Created judge: {key}") return response.json() print(f"[ERROR] Creating variation: {response.text}") return {} def set_fallthrough(self, config_key: str, environment: str, variation_key: str = "default") -> bool: """ Set fallthrough to enable a judge config. Note: turnTargetingOn doesn't work for configs. Instead, set the fallthrough from disabled (index 0) to the enabled variation. """ # Get variation ID url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/targeting" response = requests.get(url, headers=self.headers) if response.status_code != 200: print(f"[ERROR] {response.status_code}: {response.text}") return False targeting = response.json() variation_id = None for var in targeting.get("variations", []): if var.get("key") == variation_key or var.get("name") == variation_key: variation_id = var.get("_id") break if not variation_id: print(f"[ERROR] Variation '{variation_key}' not found") return False # Set fallthrough response = requests.patch(url, headers={ **self.headers, "Content-Type": "application/json; domain-model=launchdarkly.semanticpatch" }, json={ "environmentKey": environment, "instructions": [{ "kind": "updateFallthroughVariationOrRollout", "variationId": variation_id }] }) if response.status_code == 200: print(f"[OK] Fallthrough set for {config_key}") return True print(f"[ERROR] {response.status_code}: {response.text}") return False ``` ## SDK: Automatic Evaluation When using `create_model()` + `run()`, attached judges evaluate automatically: ```python import os import json import asyncio import ldclient from ldclient import Context from ldclient.config import Config from ldai import LDAIClient, AICompletionConfigDefault sdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY') ai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config') async def async_main(): ldclient.set_config(Config(sdk_key)) aiclient = LDAIClient(ldclient.get()) context = ( Context.builder('example-user-key') .kind('user') .name('Sandy') .build() ) default_value = AICompletionConfigDefault(enabled=False) # create_model() initializes with judges from Config model = await aiclient.create_model(ai_config_key, context, default_value, {}) if not model: print(f"agent configuration not enabled for: {ai_config_key}") return user_input = 'How can LaunchDarkly help me?' # run() automatically evaluates with attached judges result = await model.run(user_input) print("Response:", result.content) # Await evaluation results if result.evaluations and len(result.evaluations) > 0: eval_results = await asyncio.gather(*result.evaluations) results_to_display = [ r.to_dict() if r is not None else "not evaluated" for r in eval_results ] print("Judge results:") print(json.dumps(results_to_display, indent=2, default=str)) # Always flush events before closing — trailing events are at risk of being # lost otherwise, in short-lived scripts and long-running services alike. ldclient.get().flush() ldclient.get().close() ``` ## SDK: Direct Judge Evaluation For agent mode or custom pipelines, evaluate input/output pairs directly: ```python import os import json import asyncio import ldclient from ldclient import Context from ldclient.config import Config from ldai import LDAIClient, AIJudgeConfigDefault sdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY') judge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy') async def async_main(): ldclient.set_config(Config(sdk_key)) aiclient = LDAIClient(ldclient.get()) context = ( Context.builder('example-user-key') .kind('user') .name('Sandy') .build() ) judge_default_value = AIJudgeConfigDefault(enabled=False) # Get judge configuration from LaunchDarkly judge = aiclient.create_judge(judge_key, context, judge_default_value) if not judge: print(f"agent judge configuration not enabled for key: {judge_key}") return input_text = 'You are a helpful assistant. How can you help me?' output_text = 'I can answer any question you have.' # Evaluate the input/output pair — returns a JudgeResult. judge_result = await judge.evaluate(input_text, output_text) if not judge_result.sampled: print("Judge evaluation was skipped (sample rate or configuration issue)") return # Track the consolidated result on the Config tracker if needed: # tracker = ai_config.create_tracker() # tracker.track_judge_result(judge_result) print("Judge Result:") print(json.dumps(judge_result.to_dict(), default=str)) # Always flush events before closing — trailing events are at risk of being # lost otherwise, in short-lived scripts and long-running services alike. ldclient.get().flush() ldclient.get().close() ``` > **Note:** Direct evaluation does not automatically record metrics. Obtain a tracker via `ai_config.create_tracker()` / `aiConfig.createTracker()` and call `tracker.track_judge_result(result)` / `tracker.trackJudgeResult(result)` to record scores for the config you're evaluating. ## Sampling Rates Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage. You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%. ## Viewing Results 1. Navigate to **configs** > select your config 2. Click **Monitoring** tab 3. Select **Evaluator metrics** from dropdown 4. View scores by variation and time range Results appear within 1-2 minutes of evaluation. ## Use in Guardrails and Experiments Evaluation metrics integrate with: - **Guarded rollouts**: Pause/revert when scores fall below threshold - **Experiments**: Compare variations using evaluation metrics as goals ## Error Handling | Status | Cause | Solution | |--------|-------|----------| | 404 | Config/variation not found | Verify keys exist | | 400 | Invalid judge config | Check judgeConfigKey exists | | 403 | Insufficient permissions | Check API token permissions | | 422 | Duplicate metric key | Cannot attach multiple judges with same metric key | ## Next Steps After attaching judges: 1. **Set fallthrough** on judge configs to an enabled variation (required) 2. **Monitor results** in Monitoring tab 3. **Adjust sampling** based on cost/coverage needs 4. **Set up guarded rollouts** for automatic regression detection ## Related Skills - `configs-create` - Create configs and judges - `configs-targeting` - Configure targeting rules - `configs-variations` - Manage variations ## References - [Online Evaluations](https://docs.launchdarkly.com/home/ai-configs/online-evaluations.md) - [Custom Judges](https://docs.launchdarkly.com/home/ai-configs/custom-judges.md) **Python SDK examples:** - [create_judge_example.py](https://github.com/launchdarkly/hello-python-ai/blob/main/features/create_judge/create_judge_example.py) - Evaluate input/output pairs directly via `create_judge` + `evaluate` - [create_model_example.py](https://github.com/launchdarkly/hello-python-ai/blob/main/features/create_model/create_model_example.py) - Automatic evaluation with `create_model` + `run` (attached judges fire during the run) **Node.js SDK examples:** - [features/create-judge](https://github.com/launchdarkly/js-core/blob/main/packages/sdk/server-ai/examples/features/create-judge/src/index.ts) - Evaluate input/output pairs directly via `createJudge` + `evaluate` - [features/create-model](https://github.com/launchdarkly/js-core/blob/main/packages/sdk/server-ai/examples/features/create-model/src/index.ts) - Automatic evaluation with `createModel` + `run` (attached judges fire during the run) --- ## 22. projects - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/projects/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/projects.md - **Licence:** Other — https://github.com/launchdarkly/ai-tooling Bundled files (10), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/README.md - `references/admin-tooling.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/admin-tooling.md - `references/env-config.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/env-config.md - `references/go-setup.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/go-setup.md - `references/iac-automation.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/iac-automation.md - `references/multi-language-setup.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/multi-language-setup.md - `references/nodejs-setup.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/nodejs-setup.md - `references/project-cloning.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/project-cloning.md - `references/python-setup.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/python-setup.md - `references/quick-start.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/projects/references/quick-start.md --- name: projects description: Guide for setting up LaunchDarkly projects in your codebase. Helps you assess your stack, choose the right approach, and integrate project management that makes sense for your architecture. compatibility: Requires LaunchDarkly API access token with projects:write permission or LaunchDarkly MCP server. metadata: author: launchdarkly version: "0.4.0" --- # LaunchDarkly Projects Setup You're using a skill that will guide you through setting up LaunchDarkly project management in a codebase. Your job is to explore the codebase to understand the stack and patterns, assess what approach makes sense, choose the right implementation path from the references, execute the setup, and verify it works. ## Prerequisites **Choose one:** - LaunchDarkly API access token with `projects:write` permission - LaunchDarkly MCP server configured in your environment ## Core Principles 1. **Understand First**: Explore the codebase to understand the stack and patterns. 2. **Choose the Right Fit**: Select an approach that matches your architecture. 3. **Follow Conventions**: Respect existing code style and structure. 4. **Verify Integration**: Confirm the setup works: the agent performs checks and reports results. ## API Key Detection Before prompting the user for an API key, try to detect it automatically: 1. **Check environment variables**: Look for `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, or `LD_API_KEY` 2. **Check MCP config**: If using Claude, read `~/.claude/config.json` for `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY` 3. **Prompt user**: Only if detection fails, ask the user for their API key See [Quick Start](references/quick-start.md) for API usage patterns. ## What Are Projects? Projects are LaunchDarkly's top-level organizational containers that hold: - All your configs - Feature flags and segments - Multiple environments (Production and Test created by default) Think of projects as separate applications, services, or teams that need their own isolated set of configurations. ## Project Setup Workflow ### Step 1: Explore the Codebase Before implementing anything, understand the existing architecture: 1. **Identify the tech stack:** - What language(s)? (Python, Node.js, Go, Java, etc.) - What framework(s)? (FastAPI, Express, Spring Boot, etc.) - Is there an existing LaunchDarkly integration? 2. **Check environment management:** - How are environment variables stored? (.env files, secrets manager, config files) - Where is configuration loaded? (startup scripts, config modules) - Are there existing LaunchDarkly SDK keys? 3. **Look for patterns:** - Are there existing API clients or service modules? - How is external API integration typically done? - Is there a CLI, scripts directory, or admin tooling? 4. **Understand the use case:** - Is this a new project being set up? - Adding to an existing LaunchDarkly integration? - Part of a multi-service architecture? - Need for project cloning across regions/teams? ### Step 2: Assess the Situation Based on your exploration, determine the right approach: | Scenario | Recommended Path | |----------|------------------| | New project, no LaunchDarkly integration | **Quick Setup** - Create project and save SDK keys | | Existing LaunchDarkly usage | **Add to Existing** - Create new project or use existing | | Multiple services/microservices | **Multi-Project** - Create projects per service | | Multi-region or multi-tenant | **Project Cloning** - Clone template project | | Infrastructure-as-Code (IaC) setup | **Automated Setup** - Script-based creation | | Need project management tooling | **CLI/Admin Tools** - Build project management utilities | ### Step 3: Choose Your Implementation Path Select the reference guide that matches your stack and use case: **By Language/Stack:** - [Python Implementation](references/python-setup.md) - For Python applications (FastAPI, Django, Flask) - [Node.js/TypeScript Implementation](references/nodejs-setup.md) - For Node.js/Express/NestJS applications - [Go Implementation](references/go-setup.md) - For Go services - [Multi-Language Setup](references/multi-language-setup.md) - For polyglot architectures **By Use Case:** - [Quick Start](references/quick-start.md) - Create first project and get SDK keys - [Environment Configuration](references/env-config.md) - Save SDK keys to .env, secrets, or config - [Project Cloning](references/project-cloning.md) - Clone projects for regions/teams - [IaC/Automation](references/iac-automation.md) - Terraform, scripts, CI/CD integration - [Admin Tooling](references/admin-tooling.md) - Build CLI or admin utilities ### Step 4: Implement the Integration Follow the chosen reference guide to implement project management. Key considerations: 1. **API Authentication:** - Store API token securely - Follow existing secrets management patterns - Never commit tokens to version control 2. **Project Naming:** - Use consistent, descriptive names - Follow existing naming conventions - Project keys: lowercase, hyphens, start with letter 3. **SDK Key Management:** - Extract and store SDK keys for each environment - Use the same pattern as other secrets in your codebase - Consider separate keys for test/staging/production 4. **Error Handling:** - Handle existing projects gracefully (409 conflict) - Provide clear error messages - Don't fail silently ### Step 5: Verify the Setup After creating the project, verify it works: 1. **Fetch to confirm it exists.** Prefer the MCP `get-project` tool over raw `curl` — it returns a typed object you can inspect directly. If you must call the REST API: ```bash curl -X GET "https://app.launchdarkly.com/api/v2/projects/{projectKey}?expand=environments" \ -H "Authorization: {api_token}" ``` **Do not pipe the response straight into a `.environments.items[]`-style `jq` filter.** The shape of `environments` varies by `expand` parameter — sometimes it's `{items: [...]}`, sometimes a bare array — and a hand-rolled filter will fail with `Cannot index array with string "items"`. Run `jq -e .` first to inspect the actual shape, or use `jq '.environments | if type == "object" then .items else . end'` to handle both. 2. **Test SDK integration:** Run a quick verification to ensure the SDK key works: ```python import ldclient from ldclient.config import Config ldclient.set_config(Config("{sdk_key}")) # SDK initializes successfully # Always flush events before closing — trailing events are at risk of being # lost otherwise, in short-lived scripts and long-running services alike. ldclient.get().flush() ldclient.get().close() ``` 3. **Report results:** - ✓ Project exists and has environments - ✓ SDK keys are present and valid - ✓ SDK can initialize (or flag any issues) ## Project Key Best Practices Project keys must follow these rules: ``` ✓ Good examples: - "support-ai" - "chat-bot-v2" - "internal-tools" ✗ Bad examples: - "Support_AI" # No uppercase or underscores - "123-project" # Must start with letter - "my.project" # No dots allowed ``` **Naming Recommendations:** - Keep keys short but descriptive - Use team/service/purpose as naming scheme - Be consistent across your organization ## Common Organization Patterns ### By Team ``` platform-ai → Platform Team Agent customer-ai → Customer Success Team Agent internal-ai → Internal Tools Team Agent ``` ### By Application/Service ``` mobile-ai → Mobile App configs web-ai → Web App configs api-ai → API Service configs ``` ### By Region/Deployment ``` ai-us → US Region ai-eu → Europe Region ai-apac → Asia-Pacific Region ``` ## Edge Cases | Situation | Action | |-----------|--------| | Project already exists | Check if it's the right one; use it or create with different key | | Need multiple projects | Create separately for each service/region/team | | Shared configs across services | Use same project, separate by SDK context | | Token lacks permissions | Request `projects:write` or use MCP server | | Project name conflict | Keys must be unique, names can be similar | ## What NOT to Do - Don't create projects without understanding the use case first - Don't commit API tokens or SDK keys to version control - Don't use production SDK keys in test/development environments - Don't create duplicate projects unnecessarily - Don't skip the exploration phase ## Next Steps After setting up projects: 1. **Create configs** - Use the `configs-create` skill 2. **Set up SDK Integration** - Use the `sdk` skill 3. **Configure Targeting** - Use the `configs-targeting` skill ## Related Skills - `configs-create` - Create configs in projects - `sdk` - Integrate SDK in your application - `configs-targeting` - Configure config targeting - `configs-variations` - Manage config variations ## References - [Python Implementation](references/python-setup.md) - [Node.js Implementation](references/nodejs-setup.md) - [Go Implementation](references/go-setup.md) - [Quick Start Guide](references/quick-start.md) - [Environment Configuration](references/env-config.md) - [Project Cloning](references/project-cloning.md) - [IaC/Automation](references/iac-automation.md) - [Admin Tooling](references/admin-tooling.md) --- ## 23. snippets - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/snippets/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/snippets/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/snippets.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: snippets description: "Create and manage prompt snippets — reusable text blocks referenced inside config variation prompts. Keeps common instructions, personas, and guardrails consistent across multiple configs." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "0.1.0" --- # Config Prompt Snippets You're using a skill that will guide you through creating and managing prompt snippets in LaunchDarkly. Your job is to identify reusable text, create snippets, reference them in config variations, and verify everything is wired correctly. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `create-prompt-snippet` -- create a new reusable text block - `list-prompt-snippets` -- browse existing snippets in the project - `get-prompt-snippet` -- inspect a specific snippet's content **Optional MCP tools:** - `update-prompt-snippet` -- edit a snippet's text, name, or tags - `delete-prompt-snippet` -- permanently remove a snippet - `update-ai-config-variation` -- update variation prompts to reference snippets ## Core Concepts ### What Are Prompt Snippets? Prompt snippets are named, versioned text blocks stored at the project level. They contain reusable pieces of prompt text — personas, guardrails, output format instructions, domain knowledge — that can be shared across multiple config variations. When a snippet is updated, a new version is created. Config variations that reference the snippet can pick up the latest version, keeping all your configs in sync. ### When to Use Snippets | Scenario | Example | |----------|---------| | **Shared persona** | "You are a helpful customer support agent for Acme Corp..." used by 5 different configs | | **Safety guardrails** | "Never reveal internal pricing. Never generate code that accesses production databases." | | **Output format** | "Always respond in JSON with keys: answer, confidence, sources." | | **Domain knowledge** | Company-specific terminology, product names, or process descriptions | | **Regulatory text** | Compliance disclaimers that must appear in every response | ### When NOT to Use Snippets - Text that is unique to a single variation — just put it in the prompt directly - Dynamic content that changes per-request — use template variables instead - Entire prompts — snippets are building blocks, not complete prompts ## Core Principles 1. **Reusability First**: Only create a snippet if the text will be used in 2+ places 2. **Single Responsibility**: Each snippet should cover one concern (persona OR guardrails, not both) 3. **Descriptive Keys**: Use keys like `safety-guardrails`, `json-output-format`, `support-persona` 4. **Tag for Discovery**: Add tags so teammates can find snippets by category 5. **Verify References**: After creating a snippet, confirm it appears in the project ## Workflow ### Step 1: Identify Reusable Text Before creating snippets, understand what's shared: 1. List existing configs in the project using `get-ai-config` for each 2. Look for repeated text across variation prompts 3. Identify text that should stay consistent (guardrails, personas, formats) 4. Check existing snippets with `list-prompt-snippets` to avoid duplicates ### Step 2: Create Snippets Use `create-prompt-snippet` with: - `key` -- unique identifier (lowercase, hyphens, e.g. `safety-guardrails`) - `name` -- human-readable display name - `text` -- the reusable prompt text content - `description` (optional) -- explain when/why to use this snippet - `tags` (optional) -- categorize for discovery (e.g. `["guardrails", "safety"]`) ```json { "projectKey": "my-project", "key": "support-persona", "name": "Customer Support Persona", "text": "You are a friendly, knowledgeable customer support agent for Acme Corp. Always greet the customer by name when available. Be empathetic but concise. If you don't know the answer, say so honestly and offer to escalate.", "description": "Standard persona for all customer-facing support configs", "tags": ["persona", "support"] } ``` ### Step 3: Verify 1. Use `get-prompt-snippet` to confirm the snippet was created with the correct text 2. Use `list-prompt-snippets` to see it in the project listing 3. Check that version is 1 for newly created snippets **Report results:** - Snippet created with key, name, and text - Version number confirmed - Tags applied correctly ### Step 4: Update Snippets (When Needed) Use `update-prompt-snippet` to modify an existing snippet. Only pass the fields you want to change: ```json { "projectKey": "my-project", "snippetKey": "safety-guardrails", "text": "Updated guardrail text with new compliance requirements..." } ``` Each update creates a new version. Existing config variations referencing the snippet can pick up the new version. ## Edge Cases | Situation | Action | |-----------|--------| | Snippet key already exists | Use `get-prompt-snippet` to check, then either update or choose a different key | | Very long text | Snippets can hold large blocks — but consider splitting into multiple snippets for modularity | | Snippet referenced by configs | Update carefully — changes propagate to all referencing configs | | Deleting a referenced snippet | Warn the user that configs will lose the reference. Use `delete-prompt-snippet` with `confirm: true` | ## What NOT to Do - Don't create snippets for text used in only one place - Don't put an entire prompt in a single snippet — break it into focused pieces - Don't delete snippets without checking which configs reference them - Don't duplicate existing snippets — check `list-prompt-snippets` first ## More resources To learn more about setting up prompt snippets in the LaunchDarkly UI, read [Prompt snippets](https://launchdarkly.com/docs/home/agentcontrol/snippets.md). --- ## 24. tools - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/agentcontrol/tools/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/tools/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/tools.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (1), referenced from this skill's directory: - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/agentcontrol/tools/README.md --- name: tools description: "Give your agents capabilities through tools (function calling). Helps you identify what your agent needs to do, create tool definitions, and attach them to config variations." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # Config Tools You're using a skill that will guide you through adding capabilities to your agents through tools (function calling). Your job is to identify what your agent needs to do, create tool definitions, attach them to variations, and verify they work. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `create-ai-tool` -- create a new tool definition with a schema - `update-ai-config-variation` -- attach tools to a config variation - `get-ai-config` -- verify tools are attached to the variation **Optional MCP tools:** - `list-ai-tools` -- browse existing tools in the project - `get-ai-tool` -- inspect a specific tool's schema ## Core Principles 1. **Start with Capabilities**: Think about what your agent needs to do before creating tools 2. **Framework Matters**: LangGraph/CrewAI often auto-generate schemas; OpenAI SDK needs manual schemas 3. **Create Before Attach**: Tools must exist before you can attach them to variations 4. **Verify**: The agent fetches the config to confirm attachment 5. **Complete the Full Workflow**: Listing existing tools is a discovery step, not the end goal. After listing, always proceed to create the requested tool, attach it, and verify. Do not stop after exploration. ## Workflow ### Step 1: Identify Needed Capabilities What should the agent be able to do? - Query databases, call APIs, perform calculations, send notifications - Check what exists in the codebase (API clients, functions) - Consider framework: LangGraph/LangChain auto-generate schemas; direct SDK needs manual schemas If the user asks to check existing tools first, or you have no codebase context about what tools exist, follow this exact order: 1. `list-ai-tools` -- explore what exists 2. `create-ai-tool` -- create the new tool (with a key different from existing ones) 3. `update-ai-config-variation` -- attach it 4. `get-ai-config` -- verify Call `list-ai-tools` as your **first** tool call before any creation. Never stop after listing alone -- always proceed through all four steps. ### Step 2: Create Tools Use `create-ai-tool` with: - `key` -- unique identifier for the tool - `description` -- clear description (the LLM uses this to decide when to call the tool) - `schema` -- raw JSON Schema (do NOT use the OpenAI function calling wrapper): ```json { "type": "object", "properties": { "query": {"type": "string", "description": "Search query"}, "limit": {"type": "integer", "default": 10} }, "required": ["query"] } ``` ### Step 3: Attach to Variation Use `update-ai-config-variation` to attach tools. **Pass only the `tools` field.** Do not bundle `instructions`, `messages`, `model`, or `parameters` into this PATCH unless the user has explicitly asked you to also update those fields. Those fields may have been edited in the LaunchDarkly UI since the variation was created, and including them in a tool-attachment PATCH will silently clobber the UI edits. ```json { "projectKey": "my-project", "configKey": "support-chatbot", "variationKey": "default", "tools": [ {"key": "search-knowledge-base", "version": 1} ] } ``` If you observe a UI-clear bug where attaching tools wipes other fields, **do not work around it by re-sending those fields from the previous `get-ai-config` response** — that masks the bug and can resurrect stale values that the user has since edited. Report the bug instead. ### Step 4: Verify 1. Use `get-ai-tool` to confirm the tool exists with a valid schema 2. Use `get-ai-config` to confirm the tool is attached to the variation (check `tools` in the variation's output) **Report results:** - Tool created with valid schema - Tool attached to variation - Flag any issues ## Per-provider schema at the call site LaunchDarkly stores the tool schema once — the flat `{type, name, description, parameters}` shape you passed to `create-ai-tool`. Your application reads it back via `config.model.parameters.tools` (completion mode) or `agent_config.model.parameters.tools` (agent mode), then converts to the shape the provider SDK expects. LaunchDarkly never makes the provider call; your code does. The handlers that implement each tool also stay in application code — LaunchDarkly stores the schema, your application owns the behavior. | Provider / framework | Target shape | Where it goes on the call | |---|---|---| | OpenAI Chat Completions (direct SDK) | `{type: "function", function: {name, description, parameters}}` | top-level `tools=[...]` | | Anthropic direct SDK | `{name, description, input_schema}` — rename `parameters` → `input_schema` | top-level `tools=[...]` | | Bedrock Converse | `{toolSpec: {name, description, inputSchema: {json: parameters}}}` | inside `toolConfig.tools=[...]` | | Gemini (`google-genai`) | `{function_declarations: [{name, description, parameters}]}` (Python) / `{functionDeclarations: [...]}` (Node) | `GenerateContentConfig.tools=[...]` | | OpenAI Responses API | LaunchDarkly's flat shape passes through unchanged | top-level `tools=[...]` | | LangChain / LangGraph | `createLangChainModel(config)` (Node) / `create_langchain_model(config)` (Python) and pass `ai_config.tools` (or your own `StructuredTool` list) into `bind_tools(...)` / `create_react_agent(tools=[...])` | framework-native; no per-call conversion | | Strands Agents | LaunchDarkly's flat shape; drop `parameters.tools` before passing params to the Strands model class (`AnthropicModel`, `OpenAIModel`) — Python `@tool`-decorated callables stay in code | `Agent(tools=[...])` constructor; no per-call conversion | Minimal conversion snippets (Python): ```python ld_tools = (ai_config.model.to_dict().get("parameters") or {}).get("tools", []) or [] # OpenAI Chat Completions openai_tools = [ { "type": "function", "function": { "name": t["name"], "description": t.get("description", ""), "parameters": t.get("parameters", {"type": "object", "properties": {}}), }, } for t in ld_tools ] # Anthropic anthropic_tools = [ { "name": t["name"], "description": t.get("description", ""), "input_schema": t.get("parameters", {"type": "object", "properties": {}}), } for t in ld_tools ] # Bedrock Converse bedrock_tool_config = { "tools": [ { "toolSpec": { "name": t["name"], "description": t.get("description", ""), "inputSchema": {"json": t.get("parameters", {"type": "object", "properties": {}})}, } } for t in ld_tools ] } # Gemini gemini_tools = [ { "function_declarations": [ { "name": t["name"], "description": t.get("description", ""), "parameters": t.get("parameters", {"type": "object", "properties": {}}), } for t in ld_tools ] } ] if ld_tools else [] ``` ## Agent loop with tool calls An agent that uses tools runs a short loop: call the provider, dispatch any tool calls, loop again, stop when the provider returns a final answer. Three rules apply regardless of provider: 1. **Bound the loop.** `MAX_STEPS = 5` is a safe default. A runaway tool loop is almost always a prompt or schema bug, not a case that needs 50 iterations. 2. **Track every tool invocation.** Call `tracker.track_tool_call(tool_name)` / `tracker.trackToolCall(toolName)` for each tool the agent actually executes. This is what the Monitoring tab counts as tool usage. 3. **Break on the provider's "no more tool calls" signal.** The exact signal differs per provider: OpenAI Chat Completions → `choice.finish_reason != "tool_calls"`; Anthropic → `response.stop_reason != "tool_use"`; Bedrock Converse → `response["stopReason"] != "tool_use"`; Gemini → `response.function_calls` empty; OpenAI Responses API → no `function_call` items in `response.output`. Skeleton (Python, Anthropic — the other providers follow the same shape with their own stop-reason check and tool-result formatting): ```python messages = [{"role": "user", "content": initial_input}] MAX_STEPS = 5 for _ in range(MAX_STEPS): response = tracker.track_metrics_of( anthropic_metrics, lambda: anthropic_client.messages.create( model=agent.model.name, system=agent.instructions, messages=messages, tools=anthropic_tools, **params, ), ) if response.stop_reason != "tool_use": break messages.append({"role": "assistant", "content": response.content}) tool_results = [] for block in response.content: if block.type != "tool_use": continue if block.name not in tool_handlers: raise ValueError(f"Unknown tool: {block.name}") result = tool_handlers[block.name](**block.input) tracker.track_tool_call(block.name) tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": result, }) messages.append({"role": "user", "content": tool_results}) ``` Per-provider tool-call payload shapes live in the `built-in-metrics` references: - [openai-tracking.md](../built-in-metrics/references/openai-tracking.md) — Chat Completions + Responses API - [anthropic-tracking.md](../built-in-metrics/references/anthropic-tracking.md) — `tool_use` blocks and `tool_result` payloads - [bedrock-tracking.md](../built-in-metrics/references/bedrock-tracking.md) — `toolUse` / `toolResult` Converse format - [gemini-tracking.md](../built-in-metrics/references/gemini-tracking.md) — `functionCalls` / `functionResponse` parts - [langchain-tracking.md](../built-in-metrics/references/langchain-tracking.md) — LangGraph tool loop inherits from `create_react_agent` ## Orchestrator Note LangGraph, CrewAI, and AutoGen often generate schemas from function definitions. You still need to create tools in LaunchDarkly and attach keys to variations so the SDK knows what's available. ## Edge Cases | Situation | Action | |-----------|--------| | Tool already exists (409) | Use existing or create with different key | | Schema invalid | Use raw JSON Schema format (type: object, properties, required) | | Wrong endpoint assumed | The tools use `/ai-tools`, not `/ai-configs/tools` | ## What NOT to Do - Don't try to attach tools during config creation -- update the variation afterward - Don't skip clear tool descriptions (LLM needs them to decide when to call) - Don't forget to verify attachment after updating the variation - Don't bundle `instructions`, `messages`, `model`, or `parameters` into the tool-attachment PATCH. Send `tools` alone unless the user explicitly asked for a multi-field update — bundled PATCHes silently clobber UI edits to the other fields. ## Related Skills - `configs-create` -- Create config before attaching tools - `configs-variations` -- Manage variations with different tool sets --- ## 25. launchdarkly-experiment-setup - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/experiments/launchdarkly-experiment-setup/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/experiments/launchdarkly-experiment-setup/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-experiment-setup.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html --- name: launchdarkly-experiment-setup description: "Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "0.2.0" --- # LaunchDarkly Experiment Setup You're using a skill that guides you through setting up and running experiments in LaunchDarkly. Your job is to design the experiment, create it with the right metrics, treatments, and flag config, start data collection, evolve the design between iterations when needed, and stop with a winner. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `create-experiment` — create a new experiment with its initial iteration (hypothesis, metrics, treatments, flag config). - `start-experiment-iteration` — begin collecting data for an experiment's current draft iteration. - `get-experiment` — check experiment status, treatments, metrics, and current iteration. **Optional MCP tools:** - `list-experiments` — browse existing experiments in the project. - `update-experiment` — update fields on the experiment or its current iteration. Honours `mutableFieldsByStatus`, so what's editable depends on whether the iteration is `not_started`, `running`, or `stopped`. Returns rejected inputs under `skipped`. - `save-and-start-experiment-iteration` — the API-recommended way to change locked fields on a running experiment. Stops the current iteration, creates a new draft with the supplied field updates, and starts it in one call. - `stop-experiment-iteration` — stop the running iteration. You must declare a winner: pass the `winningTreatmentId` (and a `winningReason`). If no variation outperformed, pick the baseline/control as the winner. - `list-metrics`, `create-metric`, `list-metric-events` — manage metrics referenced by the experiment. ## Core Concepts ### What Are Experiments? Experiments in LaunchDarkly measure the impact of feature flag variations on key metrics. An experiment consists of: - **Treatments**: the flag variations being compared (control vs. test). Each treatment has an `allocationPercent`; the values across treatments should sum to 100. - **Metrics**: what you're measuring (conversion rate, latency, revenue, etc.). One must be the primary metric. - **Flag config**: the `flagKey`, `ruleId`, and `flagConfigVersion` of the targeting rule that drives the experiment. - **Iteration**: a single data-collection window. Created in `not_started` status, becomes `running` when started, transitions to `stopped` when ended. - **Holdout** (optional): a project-level group of users excluded from the experiment for baseline measurement (`holdoutId`). ### Experiment Lifecycle 1. **Create** the experiment with its first iteration (`create-experiment`). 2. **Start the iteration** to begin data collection (`start-experiment-iteration`). 3. **Monitor** results as data accumulates (`get-experiment`). 4. **Evolve the design** mid-experiment if needed — change locked fields like `treatments`, `metrics`, or `methodology` by calling `save-and-start-experiment-iteration`, which stops the current iteration, creates a new draft with your changes, and starts it. 5. **Stop the iteration** when you have a winner or a clear call (`stop-experiment-iteration`). 6. **Ship** the winning variation. ## Core Principles 1. **Metrics first**: ensure the metrics you'll reference exist before creating the experiment. 2. **Clear hypothesis**: every iteration requires a `hypothesis` string; state what you expect to improve and by how much. 3. **Proper controls**: exactly one treatment must have `baseline: true`. 4. **Sufficient sample size**: let iterations run long enough for statistical significance. 5. **One change at a time**: test one variable per experiment for clear attribution. ## Workflow ### Step 1: Prepare Metrics 1. Use `list-metrics` to find existing metrics. 2. If you need a new one, use `create-metric` and note the key. 3. Decide which is the **primary metric** (a single metric or a funnel group). You'll pass its key as `primarySingleMetricKey` or `primaryFunnelKey` on the iteration. | Goal | Metric type | Example key | |------|-------------|-------------| | Conversion | Custom conversion | `checkout-completed` | | Performance | Custom numeric | `page-load-time-ms` | | Engagement | Custom conversion | `feature-clicked` | | Revenue | Custom numeric | `order-value` | ### Step 2: Identify the Targeting Rule You need the `ruleId` and current `flagConfigVersion` of the flag rule that will drive the experiment. Use `get-flag` on the flag (or its environment-scoped status) to find them. The fallthrough rule's id is the string `"fallthrough"`. ### Step 3: Create the Experiment Call `create-experiment`. The top-level fields describe the experiment; the nested `iteration` object describes the first data-collection window. ```json { "projectKey": "my-project", "environmentKey": "production", "key": "checkout-flow-v2-experiment", "name": "Checkout Flow v2 Experiment", "description": "Compare the redesigned checkout against the current flow.", "tags": ["growth", "checkout"], "methodology": "bayesian", "iteration": { "hypothesis": "The redesigned checkout will lift completion rate by 3%.", "primarySingleMetricKey": "checkout-completed", "metrics": [ { "key": "checkout-completed" }, { "key": "checkout-time-seconds" } ], "treatments": [ { "name": "Control", "baseline": true, "allocationPercent": 50, "parameters": [ { "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" } ] }, { "name": "New Checkout", "baseline": false, "allocationPercent": 50, "parameters": [ { "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" } ] } ], "flags": { "checkout-flow-v2": { "ruleId": "fallthrough", "flagConfigVersion": 7 } }, "randomizationUnit": "user" } } ``` Useful optional top-level fields: - `holdoutId` — attach an existing holdout. - `dataSource` — `"launchdarkly"` (default), `"snowflake"`, or `"databricks"`. - `methodology` — `"bayesian"` (default), `"frequentist"`, or `"export_only"`. - `analysisConfig` — set thresholds, multiple-comparison correction, or sequential testing. Useful optional iteration fields: - `attributes` — array of context attribute keys to slice results by (e.g. `["country", "device"]`). - `covariateId` — covariate CSV id for stratified sampling. - `canReshuffleTraffic` — defaults to `true`; set `false` to lock users to their initial variation when allocations change. ### Step 4: Start Data Collection ```json { "projectKey": "my-project", "environmentKey": "production", "experimentKey": "checkout-flow-v2-experiment" } ``` Before starting, the API requires that: - the flag is toggled on, - the iteration has a `randomizationUnit`, and - at least one treatment has a non-zero `allocationPercent`. Pass `changeJustification` if you're restarting after a prior iteration was stopped. ### Step 5: Verify 1. Call `get-experiment` and confirm `currentIteration.status === "running"`. 2. Check that treatments are present with the expected allocations. 3. Check the metric list and the primary metric. ### Step 6: Evolve the Design Mid-Experiment (when needed) Most structural fields (treatments, metrics, methodology, hypothesis, …) are locked while an iteration is `running`. Two ways to change them: - **Light edits while running** — `update-experiment` will let through anything `mutableFieldsByStatus` permits in the `running` state (typically just metadata like `name`, `description`, `maintainerId`, `tags`, plus appending `metrics`/`attributes`). It surfaces rejected fields under `skipped` with a reason. - **Real design changes** — call `save-and-start-experiment-iteration`. It stops the current iteration, creates a new draft with the supplied field updates applied, and starts it in one call. Inputs match `update-experiment`, plus `changeJustification`. Mutability is checked against `not_started` since updates land on the new draft. Example: swap the treatment allocation and add a metric in a single call. ```json { "projectKey": "my-project", "environmentKey": "production", "experimentKey": "checkout-flow-v2-experiment", "changeJustification": "Lowering control allocation now that variant looks safe.", "treatments": [ { "name": "Control", "baseline": true, "allocationPercent": 30, "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-a-id" }] }, { "name": "New Checkout", "baseline": false, "allocationPercent": 70, "parameters": [{ "flagKey": "checkout-flow-v2", "variationId": "variation-b-id" }] } ], "metrics": [ { "key": "checkout-completed" }, { "key": "checkout-time-seconds" }, { "key": "checkout-error-rate" } ] } ``` ### Step 7: Stop the Iteration When you've reached significance or made a call, stop the iteration. **A winning treatment is required to stop** — LaunchDarkly does not let you end an iteration without declaring a winner. Pass the winning treatment's id (returned in `get-experiment` as `_id` on each treatment) plus a `winningReason`. If the experiment was inconclusive or no variation beat the control, declare the **baseline/control treatment as the winner** and say so in `winningReason` (e.g. "Inconclusive — no significant lift, keeping control"). There is no "stop without a winner" path. ```json { "projectKey": "my-project", "environmentKey": "production", "experimentKey": "checkout-flow-v2-experiment", "winningTreatmentId": "treat-002", "winningReason": "Two weeks of data, +4.1% lift on the primary metric with PBBL > 95%." } ``` **Report results:** - Iteration stopped with the declared `winningTreatmentId` (the control/baseline if inconclusive). - Lift / significance summary on the primary metric. - Next steps (ship the winner, roll back, or start a follow-up iteration). ## Edge Cases | Situation | Action | |-----------|--------| | Metric doesn't exist | Create it first with `create-metric`. | | Flag has no variations to compare | Create flag variations before designing treatments. | | You don't know the flag's `ruleId` / `flagConfigVersion` | Use `get-flag` or `get-flag-status-across-envs`. The fallthrough rule's id is the string `"fallthrough"`. | | Experiment already exists | Use `list-experiments` to find it; `get-experiment` for details. | | Need to change locked fields mid-experiment | Use `save-and-start-experiment-iteration` (single call) rather than stopping and recreating by hand. | | `update-experiment` returns `skipped` for a field | Inspect the `currentStatus` and `allowedFields` in the response — that field isn't mutable in the current iteration status. Either stop the iteration first or use `save-and-start-experiment-iteration`. | ## What NOT to Do - Don't omit `iteration` on `create-experiment` — it's required. - Don't set `baseline: true` on more than one treatment. - Don't let `allocationPercent` values fail to sum to 100 across treatments. - Don't try to change locked iteration fields with `update-experiment` while the iteration is `running` — reach for `save-and-start-experiment-iteration` instead. - Don't stop iterations early — wait for statistical significance. - Don't run multiple experiments on the same flag at the same time without a careful holdout design. --- ## 26. flag-and-release-change - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/flag-and-release-change/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-and-release-change/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/flag-and-release-change.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (3), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-and-release-change/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-and-release-change/README.md - `references/pr-wiring.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-and-release-change/references/pr-wiring.md --- name: flag-and-release-change description: "Drive a pull request's change end to end: decide it's flag-worthy, create the guarding flag, wire the new code path behind it on the PR branch, and record an automated release so the change ships safely when the PR merges. A portable orchestrator that composes should-flag-change, launchdarkly-flag-create, and flag-release. Keywords: flag a PR, wrap change in a flag, dark launch, kill switch, auto-release, automated rollout, end-to-end flag workflow." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server and a git CLI with access to the PR's repository metadata: author: launchdarkly version: "0.1.0" --- # Flag & Release a PR Change You're using a skill that takes a pull request whose change should ship behind a feature flag and drives it end to end: decide it needs a flag, create the guarding flag, wire the new behavior behind it on the PR's branch, and record an **automated release** so the change ships safely once the PR merges. **The deploy is not the release.** The merge ships the control path — the flag is created OFF, so deployment always serves the pre-change behavior. The *release* is the flag operation the automated rollout performs afterward, governed by the environment's policy. Creating the flag OFF and recording the release are deliberately separate things. This skill is a **portable PR orchestrator**. It doesn't own the flag mechanics or the release mechanics — it composes three focused skills and adds the PR workflow (read the diff, work in a clone, push to the branch) plus the plan→implement sequencing: | Step | Owned by | This skill's role | |------|----------|-------------------| | Decide *whether* to flag | [`should-flag-change`](../should-flag-change/SKILL.md) (advisory, read-only) | Act on a "yes"; make the call yourself if it wasn't run | | Create the flag + wire the code | [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md) | Invoke it against the change; don't re-teach flag creation or SDK patterns | | Record the release | [`flag-release`](../flag-release/SKILL.md) | Hand off once the flag exists and the code is pushed; don't re-teach rollout mechanics | Don't duplicate any composed skill's mechanics here. This skill's only unique content is the **PR wrapper** (clone, three-dot diff, commit/push to the branch) and the **plan→implement** flow that stitches the three together. > **Automation note.** An orchestrating harness (e.g. a PR pipeline) can skip this skill and invoke the three composed skills directly — `should-flag-change` → `launchdarkly-flag-create` → `flag-release` — driving the git and sequencing itself. This skill is the portable, human-in-the-loop path for a developer working a PR by hand. You work in two phases — **plan**, then **implement** — and you check in with the user in between. **Never create or modify anything during the plan phase.** ## Prerequisites - The remotely hosted LaunchDarkly MCP server. - A `git` CLI that can read and push to the PR's repository. - The composed skills available: [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md) (flag creation + code wiring) and [`flag-release`](../flag-release/SKILL.md) (recording the rollout). [`should-flag-change`](../should-flag-change/SKILL.md) is used if the flag decision hasn't been made. MCP tools are used *via the composed skills* — `create-flag`/`get-flag` through flag-create, `match-release-policies`/`create-automated-rollout-config` through flag-release. This skill calls none directly. ## Working With the Pull Request Work from a clone so you can read the change and push the flag wiring back to its branch. Credentials are provided by the environment — never ask for, print, or store tokens. ```bash git clone https://github.com//.git && cd git fetch origin pull//head git diff origin/HEAD... # three-dot: change relative to the PR's base ``` The three-dot diff (`base...head`) shows exactly what the PR introduces. Read the changed files you need to understand the change and its risk. Stay in this clone through both phases — in implement you edit, commit, and push here. Full PR mechanics (clone, three-dot diff, commit/push to the branch): [references/pr-wiring.md](references/pr-wiring.md). ## Plan Phase **Create nothing in this phase.** 1. **Confirm it should be flagged.** If [`should-flag-change`](../should-flag-change/SKILL.md) already ran, act on its verdict. Otherwise apply the same judgment: favor a flag for user-facing or risky changes; skip config-only, dependency-bump, infra, test-only, or docs changes. If a flag clearly isn't warranted, say so and stop. 2. **Understand the change and conventions.** Read the three-dot diff and changed files — what does it do, what's the blast radius? Then follow **flag-create's Step 1** to learn how this codebase already uses flags (SDK, wrapper, key constants, naming). Don't reinvent that exploration here. 3. **Design the flag.** Usually a single boolean kill-switch around the new path (flag-create's [flag-types](../launchdarkly-flag-create/references/flag-types.md) covers the choice). Don't propose more flags than the change needs. If Step 1 (or `should-flag-change`) surfaced a **dependency on a parent flag/feature that isn't live yet**, note it — the release step can couple them with a prerequisite. 4. **Plan the release.** Follow [`flag-release`](../flag-release/SKILL.md)'s plan phase: pick target environments, preview each with `match-release-policies`, and capture the human's **release intent** (release on merge / hold / `notBefore` / segment / prerequisite). Don't re-derive the rollout model here — that's flag-release's job. 5. **Present the combined plan and stop.** Summarize: the flag (`key`, `name`, boolean, tags) and why it gates *this* change; where in the code the guard goes; the per-environment release plan + captured intent (and anything to be *held*). Then wait. Revise on feedback; proceed only on clear approval. Ask a focused question if you're genuinely missing something (project key, environments, a missing policy) rather than guessing. ## Implement Phase Only after approval: 1. **Create the flag and wire the code** using [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md) (its Steps 3–4): flag created OFF with the agreed key/tags, guarding evaluation added with a safe default matching the codebase's pattern. **Fail closed on creation errors:** only an "already exists" result is success-via-reuse. Any *other* create-flag failure (auth, permissions, not-found, server error) is a hard stop — do **not** wire the code, record the release, or report success. A false "flag created" yields a green PR referencing a flag that doesn't exist, worse than an honest failure. Surface the error and stop. 2. **Add paired flag-on / flag-off tests.** If the repo has a test suite, add a test for each state of the wrapped path — flag ON serves the new behavior, flag OFF preserves the old — matching the repo's framework and flag-mocking convention. Run them and continue only once green. Scope the tests to the flagged path, not general coverage. If the repo has no tests, skip and say so. 3. **Commit and push to the PR branch.** Commit the wiring and push to the PR's existing branch so it lands in the same PR — don't open a new PR or touch the base branch. See [references/pr-wiring.md](references/pr-wiring.md). 4. **Record the release** by handing off to [`flag-release`](../flag-release/SKILL.md)'s implement phase: it records the automated rollout, honoring the captured intent (holding any environment the intent doesn't clear) and returns a `config_id`. Don't re-teach the rollout mechanics here. 5. **Report** the whole change: flag key + LaunchDarkly link (created OFF); the file(s)/code path wired; the tests added; the per-environment release plan + `config_id`; what was **held** (and why) versus what releases on merge. Report only what you verified. ## Edge Cases | Situation | Action | |-----------|--------| | Change isn't flag-worthy | Explain why (config-only, dep bump, infra, test-only, docs) and stop. Don't create a flag. | | Flag already exists | Reuse it — "already exists" is success. Wire the existing key; don't duplicate. | | Flag creation fails for any other reason (auth, permissions, 5xx) | Hard stop. Don't wire code, record a release, or claim success — surface the error. | | Codebase has no LaunchDarkly SDK | Wiring can't evaluate a flag — SDK install is separate ([onboarding/sdk-install](../../onboarding/sdk-install/SKILL.md)). | | Guarding needs more than a boolean | Prefer a boolean kill-switch. Only go multivariate if the change serves distinct variants; see flag-create's [flag-types](../launchdarkly-flag-create/references/flag-types.md). | | Release-specific cases (hold/`notBefore`, prerequisites, no matching policy, duplicate config, no useful metric) | Handled by [`flag-release`](../flag-release/SKILL.md) — see its edge cases. | ## What NOT to Do - **Don't create anything in the plan phase.** Plan proposes; implement creates. - **Don't re-document flag creation, SDK guarding, or rollout mechanics here** — those are [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md) and [`flag-release`](../flag-release/SKILL.md). Link to them. - **Don't turn the flag on yourself.** The recorded release owns that; creating the flag OFF is the point. - **Don't over-flag.** One kill-switch beats several speculative flags. - **Don't handle or print credentials.** Git access is injected. ## References - [references/pr-wiring.md](references/pr-wiring.md): PR mechanics — clone, three-dot diff, committing to the PR branch. *(This skill's only unique reference.)* - [`should-flag-change`](../should-flag-change/SKILL.md): the flag-worthiness decision. - [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md): flag creation + per-SDK guarding patterns. - [`flag-release`](../flag-release/SKILL.md): recording the automated rollout (and its [auto-release.md](../flag-release/references/auto-release.md) release model). --- ## 27. flag-release - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/flag-release/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-release/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/flag-release.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (3), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-release/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-release/README.md - `references/auto-release.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/flag-release/references/auto-release.md --- name: flag-release description: "Record an automated rollout for an existing LaunchDarkly flag that guards a pull request's change, so the change releases safely when the PR merges. Honors a stated release intent (release now / hold / notBefore / segment / prerequisite) and defers per-environment to the project's release policies. Use as the release step once the guarding flag exists and its code is wired. Keywords: record release, automated rollout, release policy, guarded rollout, staged rollout, simple vs policy, release intent, hold release, dark launch." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server. Operates on a flag that already exists; does not create flags or edit code. metadata: author: launchdarkly version: "0.1.0" --- # Record a Flag's Automated Release You're using a skill that takes an **existing** feature flag (created OFF) that guards a pull request's change, and records an **automated rollout** so the change releases safely once the PR merges — no human toggling a flag. It defers per-environment to the team's release policies and honors any human-stated release intent. This is the **release step** of the PR flag workflow, and it's deliberately atomic: | Step | Owned by | |------|----------| | Decide *whether* to flag | [`should-flag-change`](../should-flag-change/SKILL.md) (advisory) | | Create the flag + wire the code | [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md) | | **Record the release** | **this skill** | By the time this skill runs, the flag exists (OFF) and the guarding code is wired and pushed. This skill only records the rollout — it never creates flags or edits code. It can be driven directly by an automation harness, or as the final step of the [`flag-and-release-change`](../flag-and-release-change/SKILL.md) orchestrator. **The deploy is not the release.** The merge ships the control path (flag OFF); the *release* is the flag operation this rollout performs afterward, governed by the environment's policy. > **Honoring a hold is the one thing you must get right.** There is no "hold" release type, and **`policy` is NOT a manual gate.** On merge, `policy` *automatically* performs the environment's release (immediate, progressive, or guarded) with **no human promotion step** — a guarded rollout still *starts on its own* the moment the PR merges. So recording a held environment as `policy` does **not** hold it; it releases it on merge, before any `notBefore` date. The only way to hold an environment is to **omit it from the `environments` array entirely**, which leaves the flag OFF there. If the user wants an environment held (or not released until a date), exclude it from the call and report it as held. When in doubt, omit. ## Prerequisites - The remotely hosted LaunchDarkly MCP server. - The guarding flag already exists in LaunchDarkly, created OFF, with the agreed key/tags. - A pull-request reference (`repoFullName` + `prNumber`, or `prUrl`) so the rollout binds to the right merge. **MCP tools this skill uses:** - `create-automated-rollout-config` — record the rollout for the flag against the PR *(the deliverable)* - `match-release-policies` — resolve which release policy governs each environment (call before proposing the plan) - `list-release-policies` — see the project's release policies and the metrics they auto-attach - `get-flag` — confirm the flag exists and is OFF before recording Full release model — `simple` vs `policy`, precedence, previewing, prerequisites, metric adequacy: [references/auto-release.md](references/auto-release.md). ## Plan Phase **Record nothing in this phase.** 1. **Confirm the flag.** `get-flag` to verify the guarding flag exists and is OFF. If it doesn't exist yet, stop — creation is [`launchdarkly-flag-create`](../launchdarkly-flag-create/SKILL.md)'s job, and recording a rollout for a missing flag fails confusingly. 2. **Pick the target environments.** Use the environments named by the user or harness. Don't hardcode a set — a given change can't always release to every environment. If none are named, enumerate the project's real keys and confirm the set rather than assuming. 3. **Preview each environment's policy.** Call `match-release-policies` (by `flagKey` + `environmentKey`) to resolve, deterministically, what a `policy` release will do per environment — `winningReleaseMethod` (immediate / progressive / guarded / none). Don't reason about policy scope by hand. For a **guarded** winner, check the auto-attached metrics can actually compare this change (see the metric-adequacy note in [references/auto-release.md](references/auto-release.md)). 4. **Capture the human's release intent.** Ask (briefly, only if not already stated): release **on merge**, **hold** (recorded but not released yet), or wait until a **`notBefore`** date? A **cohort/segment** to target first? A **prerequisite** parent flag this must not precede? Intent sits above the policy in precedence and is **honored or explicitly held — never silently dropped**. 5. **Present the per-environment plan and stop.** For each environment, state either the `releaseType` it will be recorded with (`simple` / `policy`, and what that does on merge) **or** that it will be **held** — omitted from the recorded config so the flag stays OFF there — with the reason. Wait for confirmation; revise on feedback. ## Implement Phase Only after confirmation: 1. **Sort every target environment into exactly one bucket — RELEASE or HOLD.** Do this *before* you build the call. There is no third bucket, and `releaseType` does not create one. | Bucket | Meaning | What goes in the call | |--------|---------|-----------------------| | **RELEASE** | Goes live on merge — now, or per its policy | Add `{ environmentKey, releaseType }` to the `environments` array | | **HOLD** | Not yet — waiting on a date, sign-off, segment, or parent flag | **Nothing.** Leave it out of `environments` entirely; name it as held in your report | `releaseType` (`simple` vs `policy`) only chooses *how* a RELEASE environment goes live — it never holds one. **Both release on merge:** `simple` serves `true` immediately; `policy` runs that environment's policy (immediate / progressive / guarded) automatically, with no human promotion step. "`policy` defers to the *policy*" — not to you, and not until a date. **Do not reach for `policy` to park a held environment: it ships that environment on merge, before any `notBefore`.** The only encoding of a hold is *absence from the array*. 2. **Build `environments` from the RELEASE bucket only — then read the keys back.** The array must contain every RELEASE environment and no HOLD environment. Before you send the call, scan the `environmentKey`s in the array: if any environment you're holding appears there, delete that entry. The call has **no field for a date or a hold** — if you catch yourself wanting to add `holdUntil`, `notBefore`, or `hold` to an entry (or to keep it as `policy` "so it waits"), that is the signal the environment is HOLD: **drop the entry, don't invent a field.** The date and reason go in your report, not the call. Worked example — *"release staging on merge, hold production until 2026-09-01"*: staging is RELEASE, production is HOLD. ```json { "projectKey": "default", "flagKey": "new-checkout-flow", "environments": [{ "environmentKey": "staging", "releaseType": "simple" }], "repoFullName": "acme/storefront", "prNumber": 482 } ``` `production` appears nowhere in the call. You report it held until 2026-09-01, with the reason (legal sign-off). Fail closed: if an environment's intent is unclear, it's HOLD, not RELEASE. 3. **Record the rollout.** Call `create-automated-rollout-config` with `projectKey`, `flagKey`, the RELEASE-only `environments` array, and the PR reference. If a **prerequisite** parent flag was agreed, wire it if the MCP surface supports it; otherwise report it as a manual step. Details: [references/auto-release.md](references/auto-release.md). 4. **Verify.** The call returns `created`, `config_id`, and the normalized per-environment plan — record `config_id`. Report only what you verified; flag anything you couldn't confirm rather than asserting it. 5. **Report** the per-environment release plan + `config_id`; what was **held** (and why) versus what releases on merge; and what happens on merge (e.g. "production resolves policy X → guarded rollout on merge; staging serves true immediately; production held until 2026-08-01 per intent"). ## Edge Cases | Situation | Action | |-----------|--------| | Flag doesn't exist yet | Stop — creation is `launchdarkly-flag-create`. Recording a rollout for a missing flag fails confusingly. | | A rollout config already exists for this flag + PR | Don't record a second one — a duplicate confuses the scheduler. Point the user at the existing config to change the plan. | | Registering before the PR exists | `simple` envs work without a PR, but `policy` envs need `repoFullName`/`prNumber` to trigger on merge. Prefer recording *after* the PR is open; if you record early, say `policy` won't fire until the PR is wired. | | No release policy matches an env | `policy` falls back to defaults (often immediate). Tell the user; offer `simple`, or point at release-policy setup. | | User wants to hold, or set a `notBefore` date | Skip the releasing plan for those environments; report them as held with the reason. Never silently release against stated intent. | | Change depends on a parent flag not yet live | Couple them with a prerequisite (set it if the MCP surface supports it); otherwise report the coupling as a required manual step. Don't let this flag release before its parent. | | A `policy` env resolves to guarded but has no relevant metric | Say so — a guarded rollout with no meaningful metric guards nothing. Recommend `simple`, or point at metric setup. | ## What NOT to Do - **Don't create the flag or edit code** — that's `launchdarkly-flag-create`. This skill only records the release. - **Don't turn the flag on yourself, or toggle it after recording the config.** The rollout owns that; double-toggling causes audit noise and confuses the scheduler. - **Don't skip `match-release-policies`.** Proposing `policy` without knowing what it resolves to is guessing. - **Don't silently release against a stated hold/`notBefore`.** Honor intent or hold — never drop it. - **Don't handle or print credentials.** Access is injected by the environment. ## References - [references/auto-release.md](references/auto-release.md): the automated-rollout / release-policy model, `simple` vs `policy`, precedence (intent → override → policy → default), previewing, prerequisites, metric adequacy. *(Core of this skill.)* - [`launchdarkly-guarded-rollout`](../launchdarkly-guarded-rollout/SKILL.md): for a *bespoke* rollout no policy expresses — set that env to `simple` here and drive the guarded rollout by hand after merge. --- ## 28. launchdarkly-flag-cleanup - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-cleanup/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-cleanup/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-cleanup.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-cleanup/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-cleanup/README.md - `references/pr-template.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-cleanup/references/pr-template.md - `references/sdk-patterns.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-cleanup/references/sdk-patterns.md --- name: launchdarkly-flag-cleanup description: "Safely remove a feature flag from code while preserving production behavior. Use when the user wants to remove a flag from code, delete flag references, or create a PR that hardcodes the winning variation after a rollout is complete." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # LaunchDarkly Flag Cleanup You're using a skill that will guide you through safely removing a feature flag from a codebase while preserving production behavior. Your job is to explore the codebase to understand how the flag is used, query LaunchDarkly to determine the correct forward value, remove the flag code cleanly, and verify the result. If you haven't already identified which flag to clean up, use the [flag discovery skill](../launchdarkly-flag-discovery/SKILL.md) first to audit the landscape and find candidates. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `check-removal-readiness`: detailed safety check (orchestrates flag config, cross-env status, dependencies, code references, and expiring targets in parallel) - `get-flag`: fetch flag configuration for a specific environment **Optional MCP tools:** - `archive-flag`: archive the flag in LaunchDarkly after code removal - `delete-flag`: permanently delete the flag (irreversible, prefer archive) ## Core Principles 1. **Safety First**: Always preserve current production behavior. 2. **LaunchDarkly as Source of Truth**: Never guess the forward value. Query the actual configuration. 3. **Follow Conventions**: Respect existing code style and structure. 4. **Minimal Change**: Only remove flag-related code. No unrelated refactors. ## Workflow ### Step 1: Explore the Codebase Before touching LaunchDarkly or removing code, understand how this flag is used in the codebase. 1. **Find all references to the flag key.** Search for the flag key string (e.g., `new-checkout-flow`) across the codebase. Check for: - Direct SDK evaluation calls (`variation()`, `boolVariation()`, `useFlags()`, etc.) - Constants/enums that reference the key - Wrapper/service patterns that abstract the SDK - Configuration files, tests, and documentation - See [SDK Patterns](references/sdk-patterns.md) for the full list of patterns by language 2. **Understand the branching.** For each reference, identify: - What code runs when the flag is `true` (or variation A)? - What code runs when the flag is `false` (or variation B)? - Are there side effects, early returns, or nested conditions? 3. **Note the scope.** How many files, components, or modules does this flag touch? A flag used in one `if` block is simpler than one threaded through multiple layers. ### Step 2: Run the Removal Readiness Check Use `check-removal-readiness` to get a detailed safety assessment. This single tool call orchestrates multiple checks in parallel: - Flag configuration and targeting state - Cross-environment status - Dependent flags (prerequisites) - Expiring targets - Code reference statistics The tool returns a readiness verdict: **`safe`**: No blockers or warnings. Proceed with removal. **`caution`**: No hard blockers but warnings exist (e.g., code references in other repos, expiring targets scheduled, flag marked as permanent). Present warnings and let the user decide. **`blocked`**: Hard blockers prevent safe removal (e.g., dependent flags, actively receiving requests, targeting is on with active rules). Present blockers: the user must resolve them first. ### Step 3: Determine the Forward Value Use `get-flag` to fetch the flag configuration in each critical environment. The **forward value** is the variation that replaces the flag in code. | Scenario | Forward Value | |----------|---------------| | All critical envs ON, same fallthrough, no rules/targets | Use `fallthrough.variation` | | All critical envs OFF, same offVariation | Use `offVariation` | | Critical envs differ in ON/OFF state | **NOT SAFE**: stop and inform the user | | Critical envs serve different variations | **NOT SAFE**: stop and inform the user | ### Step 4: Present the Cleanup Plan Before modifying any code, present a summary to the user and wait for confirmation: 1. **The forward value** — which variation will be hardcoded and why (based on the flag's current state). 2. **All code references found** — file paths and line numbers from Step 1. 3. **Planned changes** — for each reference, describe what will be removed and what will be kept. 4. **Readiness verdict** — the result from `check-removal-readiness` (safe, caution, or blocked) and any warnings. 5. **LaunchDarkly action** — confirm the flag will be archived after code changes are complete. **Do not proceed with code changes until the user explicitly confirms.** ### Step 5: Remove the Flag from Code Now execute the removal using what you learned in Step 1. 1. **Replace flag evaluations with the forward value.** - Preserve the code branch matching the forward value - Remove the dead branch entirely - If the flag value was assigned to a variable, replace the variable with the literal value or inline it 2. **Clean up dead code.** - Remove imports, constants, and type definitions that only existed for the flag - Remove functions, components, or files that only existed for the dead branch - Check for orphaned exports, hooks, helpers, styles, and test files - If the repo uses an unused-export tool (Knip, ts-prune, lint rules), run it and remove any flag-related orphans 3. **Don't over-clean.** - Only remove code directly related to the flag - Don't refactor, optimize, or "improve" surrounding code - Don't change formatting or style of untouched code **Example transformation (boolean flag, forward value = `true`):** ```typescript // Before const showNewCheckout = await ldClient.variation('new-checkout-flow', user, false); if (showNewCheckout) { return renderNewCheckout(); } else { return renderOldCheckout(); } // After return renderNewCheckout(); ``` ### Step 6: Create Pull Request Use the template in [references/pr-template.md](references/pr-template.md) for a structured PR description. The PR should clearly communicate: - What flag was removed and why - What the forward value is and why it's correct - The readiness assessment results (from `check-removal-readiness`) - What code was removed and what behavior is preserved - Whether other repos still reference this flag ### Step 7: Verify Before considering the job done: 1. **Code compiles and lints.** Run the project's build and lint steps. 2. **Tests pass.** If the flag was used in tests, the tests should be updated to reflect the hardcoded behavior. 3. **No remaining references.** Search the codebase one more time for the flag key to make sure nothing was missed. 4. **PR is complete.** The description covers the readiness assessment, forward value rationale, and any cross-repo coordination needed. ## Edge Cases | Situation | Action | |-----------|--------| | Flag not found in LaunchDarkly | Inform user, check for typos in the key | | Flag already archived | Ask if code cleanup is still needed (flag is gone from LD but code may still reference it) | | Multiple SDK patterns in codebase | Search all patterns: `variation()`, `boolVariation()`, `variationDetail()`, `allFlags()`, `useFlags()`, plus any wrappers | | Dynamic flag keys (`flag-${id}`) | Warn that automated removal may be incomplete: manual review required | | Different default values in code vs LD | Flag as inconsistency in the PR description | | Orphaned exports/files remain after removal | Run unused-export checks and remove dead files | ## What NOT to Do - Don't change code unrelated to flag cleanup. - Don't refactor or optimize beyond flag removal. - Don't remove flags still being actively rolled out. - Don't guess the forward value: always query LaunchDarkly. ## After Cleanup Once the PR is merged and deployed: 1. **Archive the flag in LaunchDarkly** using `archive-flag`. Archival is reversible; deletion is not. Always archive first. 2. **Notify other teams** if `check-removal-readiness` reported code references in other repositories. 3. **If the flag had targeting changes pending,** they can be ignored: the flag is being removed. ## References - [PR Template](references/pr-template.md): Structured PR description for flag removal - [SDK Patterns](references/sdk-patterns.md): Flag evaluation patterns by language/framework - [Flag Discovery](../launchdarkly-flag-discovery/SKILL.md): Find cleanup candidates before using this skill - [Flag Targeting](../launchdarkly-flag-targeting/SKILL.md): If you need to change targeting instead of removing --- ## 29. launchdarkly-flag-command - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-command/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-command/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-command.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (2), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-command/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-command/README.md --- name: launchdarkly-flag-command description: "Resolve `/flag` style requests into the right LaunchDarkly flag lookup flow. Use when the user types `/flag`, asks to quickly find a flag by name/key, wants a direct flag detail summary, or needs fast disambiguation between similar flags." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # LaunchDarkly Flag Command Router You're using a skill that standardizes quick `/flag` requests. Your job is to parse the user intent, resolve the requested flag with minimal friction, return an actionable summary, and route to deeper workflows when needed. ## Scope Boundary This skill is a **read-only lookup entrypoint**. It returns flag details and routes forward. **Hard constraints — you MUST NOT:** - Create, toggle, update, or delete flags - Assess whether a flag is safe to remove, stale, or ready for cleanup - Provide a "verdict", "safe to remove" conclusion, removal steps, or "before removing" advice - Offer to archive or delete the flag **When the user asks about removal or staleness**, your entire response for that part must be the flag summary table followed by this exact routing message (you may rephrase slightly but must keep the substance): > This quick lookup can only show you the flag's current config. To assess whether it's safe to remove, you need the **flag discovery** or **flag cleanup** skill — they scan code references, check status across all environments, and analyze downstream dependencies. That's it. No analysis. No bullet points. No verdict. The removal question is answered by the routing message, not by you. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `list-flags` — search and disambiguate flag candidates - `get-flag` — fetch detailed configuration for a resolved flag **Optional MCP tools:** - `get-flag-status-across-envs` — compare lifecycle status across environments - `get-flag-health` — quick health snapshot for a single flag ## Command Contract Treat these forms as equivalent intents: - `/flag ` - `flag ` - "find flag " - "show me flag" Use `production` as the default environment unless the user specifies another environment. ## Workflow ### Step 1: Parse and Normalize Input 1. Extract the query text after `/flag`. 2. If no query is provided, ask for one concise identifier (flag key, name fragment, or tag). 3. Capture optional hints from the request: - Environment (`staging`, `production`, etc.) - Project key - Preference for exact key vs fuzzy search ### Step 2: Resolve the Flag Use `list-flags` first unless the user clearly provided an exact key and project. 1. Search with `list-flags` using the query. 2. If one clear exact match exists, resolve to that flag. 3. If multiple plausible matches exist, return a short disambiguation list (key + name + state) and ask the user to pick. 4. If no matches exist, tell the user and suggest one broader query. ### Step 3: Return a Useful Summary For a resolved flag, call `get-flag` and return: 1. Flag key and name 2. Environment state (`on`/`off`) 3. Off variation and fallthrough behavior 4. Rule/target complexity (simple vs complex) 5. Direct LaunchDarkly URL for the flag (when project + key are known) **If the user asked about removal, staleness, or cleanup** (e.g., "is this safe to remove?", "can I clean this up?", "is this stale?"): Show ONLY the summary table above, then write: > This quick lookup can only show you the flag's current config. To assess whether it's safe to remove, you need the **flag discovery** or **flag cleanup** skill — they scan code references, check status across all environments, and analyze downstream dependencies. Do not add a verdict, bullet-point analysis, removal steps, "before removing" checklist, or an offer to archive/delete. The removal question is **fully answered by the routing message above**. Proceed to Step 4. ### Step 4: Route to the Right Follow-up Workflow After returning the summary, check whether the user's request implies a deeper workflow. If it does, **name the skill and stop** — do not attempt the workflow yourself. | User intent | Route to | |---|---| | Create or modify a flag | [flag create skill](../launchdarkly-flag-create/SKILL.md) | | Change targeting or rollout | [flag targeting skill](../launchdarkly-flag-targeting/SKILL.md) | | "Is this safe to remove?", "Is this stale?", cleanup | [flag discovery](../launchdarkly-flag-discovery/SKILL.md) / [flag cleanup](../launchdarkly-flag-cleanup/SKILL.md) | For removal/staleness questions specifically: follow the Scope Boundary instructions above — summary table only, then route. No verdict. ## Output Style Keep `/flag` responses brief and operational: - Start with the resolved flag (or disambiguation list) - Include only the minimum config details needed for the next action - End with one clear next step question when user intent is ambiguous ## Important Context - `/flag` is a fast entrypoint, not a full lifecycle workflow. - Prefer disambiguation over guessing when multiple flags match. - Treat project + environment as first-class context; avoid hidden assumptions. - When sharing rollout percentages, always use human-readable percentages. - **Never improvise removal, staleness, or cleanup analysis.** Always route to the dedicated skill. --- ## 30. launchdarkly-flag-create - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-create/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-create/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-create.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-create/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-create/README.md - `references/flag-types.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-create/references/flag-types.md - `references/sdk-evaluation-patterns.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-create/references/sdk-evaluation-patterns.md --- name: launchdarkly-flag-create description: "Create and configure LaunchDarkly feature flags in a way that fits the existing codebase. Use when the user wants to create a new flag, wrap code in a flag, add a feature toggle, or set up an experiment. Guides exploration of existing patterns before creating." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.1.0-experimental" --- # LaunchDarkly Flag Create & Configure You're using a skill that will guide you through introducing a new feature flag into a codebase. Your job is to explore how flags are already used in this codebase, create the flag in LaunchDarkly in a way that fits, add the evaluation code matching existing patterns, and verify everything is wired up correctly. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `create-flag`: create a new feature flag in a project - `get-flag`: verify the flag was created correctly **Optional MCP tools (enhance workflow):** - `list-flags`: browse existing flags to understand naming conventions and tags - `update-flag-settings`: update flag metadata (name, description, tags, temporary/permanent status) ## Workflow ### Step 1: Explore the Codebase Before creating anything, understand how this codebase uses feature flags. 1. **Find the SDK.** Search for LaunchDarkly SDK imports or initialization: - Look for `launchdarkly`, `ldclient`, `ld-client`, `LDClient` in imports - Check `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, or equivalent for the SDK dependency - Identify which SDK is in use (server-side Node, React, Python, Go, Java, etc.) 2. **Find existing flag evaluations.** Search for variation calls to understand the patterns this codebase uses: - Direct SDK calls: `variation()`, `boolVariation()`, `useFlags()`, etc. - Wrapper patterns: Does this codebase abstract flags behind a service or utility? - Constant definitions: Are flag keys defined as constants somewhere? - See [SDK Evaluation Patterns](references/sdk-evaluation-patterns.md) for patterns by language 3. **Understand conventions.** Look at existing flags to learn: - **Naming convention**: Are keys `kebab-case`, `snake_case`, `camelCase`? - **Organization**: Are flag keys co-located with features, or centralized in a constants file? - **Default values**: What defaults do existing evaluations use? - **Context/user construction**: How does this codebase build the user/context object passed to the SDK? This determines which context kinds and attributes any future targeting can use — and it differs by surface (server vs client vs anonymous). See [Context Availability](../launchdarkly-flag-targeting/references/context-availability.md) before planning a rule, individual target, or rollout. 4. **Check LaunchDarkly project conventions.** Optionally use `list-flags` to see existing flags: - What tags are commonly used? - Are flags marked as temporary or permanent? - What naming patterns exist in the project? ### Step 2: Determine the Right Flag Type Based on what the user needs, choose the appropriate flag configuration. See [Flag Types and Patterns](references/flag-types.md) for the full guide. **Quick decision:** | User intent | Flag kind | Variations | |-------------|-----------|------------| | "Toggle a feature on/off" | `boolean` | `true` / `false` | | "Gradually roll out a feature" | `boolean` | `true` / `false` | | "A/B test between options" | `multivariate` (string) | User-defined values | | "Configure a numeric threshold" | `multivariate` (number) | User-defined values | | "Serve different config objects" | `multivariate` (JSON) | User-defined values | **Defaults to apply:** - Set `temporary: true` unless the user explicitly says this is a permanent/long-lived flag. Most flags are release flags that should eventually be cleaned up. - Generate a `key` from the name if not provided (e.g., "New Checkout Flow" -> `new-checkout-flow`), but match the codebase's naming convention if one exists. - Suggest relevant tags based on the feature area, team, or context the user mentions. ### Step 3: Create the Flag in LaunchDarkly Use `create-flag` with the configuration determined in Step 2. After creation: - The flag is created with **targeting OFF** in all environments. - The flag serves the `offVariation` to everyone until targeting is turned on. - Remind the user they'll need to use the [flag targeting skill](../launchdarkly-flag-targeting/SKILL.md) to toggle it on and optionally set up rollout rules. ### Step 4: Add Flag Evaluation to Code Now add the code to evaluate the flag, **matching the patterns you found in Step 1**. 1. **Use the same SDK patterns** the codebase already uses. If there's a wrapper, use the wrapper. If there are constants, add the new key to the constants file. 2. **Use an appropriate default value.** The default (fallback) value in code should be the "safe" behavior: typically the existing behavior before the flag. This ensures the feature stays off if the SDK can't reach LaunchDarkly. 3. **Add the conditional logic.** Wrap the new behavior in a flag check. 4. **Handle both branches.** Make sure the code path for each variation is clear and complete. See [SDK Evaluation Patterns](references/sdk-evaluation-patterns.md) for implementation examples by language and framework. ### Step 5: Verify Confirm the flag is properly set up: 1. **Code compiles/passes linting.** Run the project's build or lint step. 2. **Flag exists in LaunchDarkly.** Use `get-flag` to confirm it was created with the right configuration. 3. **Both code paths work.** The flag-off path preserves existing behavior; the flag-on path enables the new feature. 4. **Default value is safe.** If LaunchDarkly is unreachable, the code falls back to the default: make sure that's the existing/safe behavior. ## Updating Flag Settings If the user wants to change flag metadata (not targeting), use `update-flag-settings`. Supported changes: | Change | Instruction | |--------|-------------| | Rename | `{kind: "updateName", value: "New Name"}` | | Update description | `{kind: "updateDescription", value: "New description"}` | | Add tags | `{kind: "addTags", values: ["tag1", "tag2"]}` | | Remove tags | `{kind: "removeTags", values: ["old-tag"]}` | | Mark as temporary | `{kind: "markTemporary"}` | | Mark as permanent | `{kind: "markPermanent"}` | Multiple instructions can be batched in a single call. These changes are project-wide, not environment-specific. **Important:** Metadata updates (above) are separate from targeting changes (toggle, rollout, rules). If the user wants to change who sees what, direct them to the [flag targeting skill](../launchdarkly-flag-targeting/SKILL.md). ## Important Context - **Flag keys are immutable.** Once created, a flag's key cannot be changed. Choose carefully. - **Flags start OFF.** Creation never enables a flag. This is a safety feature. - **The default value in code is your safety net.** It's what gets served when the SDK can't connect to LaunchDarkly. Always use the "safe" / existing behavior as the default. - **Follow existing codebase conventions.** The most common mistake is introducing a flag pattern that doesn't match what the team already does. Step 1 exists to prevent this. ## References - [Flag Types and Patterns](references/flag-types.md): Boolean vs multivariate, naming conventions, configuration best practices - [SDK Evaluation Patterns](references/sdk-evaluation-patterns.md): How to evaluate flags in each SDK, including common wrapper patterns - [Context Availability](../launchdarkly-flag-targeting/references/context-availability.md): Which context kinds/attributes targeting can use, matched to the surface where the flag is read (relevant once the flag will target rather than being a plain on/off switch) --- ## 31. launchdarkly-flag-discovery - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-discovery/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-discovery/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-discovery.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-discovery/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-discovery/README.md - `references/flag-health-signals.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-discovery/references/flag-health-signals.md - `references/removal-readiness-checklist.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-discovery/references/removal-readiness-checklist.md --- name: launchdarkly-flag-discovery description: "Audit your LaunchDarkly feature flags to understand the landscape, find stale or launched flags, and assess removal readiness. Use when the user asks about flag debt, stale flags, cleanup candidates, flag health, or wants to understand their flag inventory." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # LaunchDarkly Flag Discovery You're using a skill that will guide you through auditing and understanding the feature flag landscape in a LaunchDarkly project. Your job is to explore the project, assess the health of its flags, identify what needs attention, and provide actionable recommendations. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `list-flags`: search and browse flags with filtering by state, type, tags - `get-flag`: get full configuration for a single flag in a specific environment - `get-flag-status-across-envs`: check a flag's lifecycle status across all environments **Optional MCP tools (enhance depth):** - `find-stale-flags`: find flags that are candidates for cleanup, sorted by staleness - `get-flag-health`: get combined health view for a single flag (merges status + config) - `check-removal-readiness`: detailed safety check for a specific flag ## Workflow ### Step 1: Understand the Project Before diving into flag data, establish context: 1. **Identify the project.** Confirm the `projectKey` with the user. If they haven't specified one, ask. 2. **Understand scope.** Ask the user what they're trying to accomplish: - Broad audit? ("What's the state of our flags?") - Targeted investigation? ("Is this specific flag still needed?") - Cleanup planning? ("What flags can we remove?") ### Step 2: Explore the Flag Landscape Adapt your approach to the user's goal: **For a broad audit:** - Use `list-flags` scoped to a critical environment (default to `production`). - Note the total count: this tells you the scale of the flag surface area. - Filter by `state` (active, inactive, launched, new) to segment the landscape. - Filter by `type` (temporary vs permanent): temporary flags are the primary cleanup targets. **For cleanup planning:** - Use `find-stale-flags`: this is the most efficient entry point. It returns a prioritized list of cleanup candidates sorted by staleness, categorized as: - `never_requested`: created but never evaluated (possibly abandoned) - `inactive_30d`: no SDK evaluations in the specified period - `launched_no_changes`: fully rolled out, no recent changes - Default `inactiveDays` is 30. Increase for conservative cleanup (60, 90) or decrease for aggressive cleanup (7, 14). - Default `includeOnly` is `temporary`. Set to `all` to include permanent flags. **For a targeted investigation:** - Use `get-flag-health` for a single-flag deep dive. It merges status data with configuration context in one call, returning lifecycle state, last-requested timestamp, targeting summary, age, and whether it's temporary. - Or use `get-flag` for the full configuration including rules, targets, and fallthrough details. ### Step 3: Assess Flag Health For flags that need deeper investigation, assess health signals. See [Flag Health Signals](references/flag-health-signals.md) for the full interpretation guide. Key signals to evaluate: | Signal | What it tells you | |--------|-------------------| | **Lifecycle state** | Where the flag is in its journey (new -> active -> launched -> inactive) | | **Last requested** | When an SDK last evaluated this flag: staleness indicator | | **Targeting complexity** | Number of rules and targets: removal complexity indicator | | **Cross-environment consistency** | Whether the flag behaves the same everywhere | | **Flag age + temporary status** | Old temporary flags are strong cleanup candidates | Use `get-flag-status-across-envs` to check if a flag is consistent across environments. A flag inactive in production but active in staging tells a different story than one inactive everywhere. ### Step 4: Categorize and Prioritize Group flags into actionable categories: 1. **Ready to remove**: Inactive everywhere, temporary, no dependencies. Direct the user to the [flag cleanup skill](../launchdarkly-flag-cleanup/SKILL.md) for code removal. 2. **Likely safe, needs verification**: Launched (fully rolled out), no rule changes recently. The user should confirm the rollout is intentionally complete. 3. **Needs investigation**: Active in some environments but not others, or has complex targeting. Don't recommend action without more context. 4. **Leave alone**: Active flags doing their job, or permanent flags that are intentionally long-lived. ### Step 5: Assess Removal Readiness (When Applicable) If the user wants to know whether a specific flag can be removed, use `check-removal-readiness`. This tool orchestrates multiple API calls in parallel and returns a structured verdict: - **`safe`**: No blockers or warnings. Proceed with cleanup. - **`caution`**: Warnings exist (code references, expiring targets, permanent flag type). Present and let the user decide. - **`blocked`**: Hard blockers (dependent flags, active requests, targeting rules). Must resolve first. See [Removal Readiness Checklist](references/removal-readiness-checklist.md) for the full details on interpreting each signal. ### Step 6: Present Findings Structure your response based on what the user asked for: **For audits:** Lead with a summary (total flags, breakdown by state and type), then highlight what needs attention, then provide specific recommendations. **For specific flags:** Lead with the verdict (healthy / needs attention / ready to remove), then support it with the signals you found. **For cleanup planning:** Lead with the count of cleanup candidates, prioritize by confidence (safest removals first), and link to the cleanup workflow for execution. ## Important Context - **"Launched" means fully rolled out**: targeting is on, a single variation is served to everyone, and no changes have been made recently. It doesn't mean "recently deployed." - **"Inactive" doesn't always mean safe to remove.** The flag might be used in code that hasn't shipped yet, or referenced as a prerequisite by another flag. - **Permanent flags can be inactive on purpose.** Some flags are designed to be dormant until needed (kill switches, emergency toggles). Don't automatically flag these for cleanup. - **Weights are scaled by 1000 in the API.** A weight of `60000` means 60%. Always convert to human-readable percentages. - **This skill is for discovery, not action.** If the user wants to remove a flag from code, direct them to the [flag cleanup skill](../launchdarkly-flag-cleanup/SKILL.md). If they want to change targeting, direct them to the [flag targeting skill](../launchdarkly-flag-targeting/SKILL.md). ## References - [Flag Health Signals](references/flag-health-signals.md): How to interpret lifecycle states, staleness, and health data - [Removal Readiness Checklist](references/removal-readiness-checklist.md): Full safety assessment before recommending flag removal --- ## 32. launchdarkly-flag-drift - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-drift/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-drift/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-drift.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (4), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-drift/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-drift/README.md - `references/pr-template.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-drift/references/pr-template.md - `references/sdk-default-patterns.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-drift/references/sdk-default-patterns.md --- name: launchdarkly-flag-drift description: "Detect and reconcile drift between a feature flag's in-code SDK fallback default and its LaunchDarkly default rule (fallthrough). Use when a flag's default rule changed, when the user asks to detect flag drift, check whether a hardcoded default still matches LaunchDarkly, sync an in-code default, or open a PR reconciling a fallback value, without removing the flag or its evaluation." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "0.1.0" --- # LaunchDarkly Flag Drift Detection You're using a skill that will guide you through checking whether a feature flag's **in-code SDK fallback default** has drifted from its **LaunchDarkly default rule (fallthrough)**, and reconciling the code if it has. Your job is to determine the flag's current default rule value from LaunchDarkly, locate the default argument passed to every SDK evaluation in code, compare them, and, only when they differ, update the in-code default so it matches. You never remove the flag or change its evaluation logic. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `get-flag`: fetch the flag's configuration **in a specific environment** (fallthrough, variations, offVariation). Fallthrough is environment-specific, so call this **once per critical environment**. **Optional MCP tools:** - `list-flags`: find the flag key if the user only described the flag by name - `get-environments`: enumerate the project's environments to determine which are critical - `get-flag-status-across-envs`: quick snapshot of a flag's state across environments (a fast way to spot cross-environment differences before pulling each `get-flag`) ## Core Concept: The SDK Fallback Default Is Not the Off Variation Every SDK evaluation call takes a **fallback default**: the value returned when LaunchDarkly is unreachable, the client is uninitialized, or the flag is unavailable. This is a code-side safety value, distinct from the flag's `offVariation` (served when the flag is toggled off) and from its `fallthrough` (the default rule served when no targeting rule matches). This skill treats one specific invariant as "correct": **the in-code fallback default should match the current default rule (fallthrough) value**. When they diverge, that's *drift*. Reconciling it keeps the value returned during an outage consistent with what most users would otherwise receive. ## Core Concept: One In-Code Default, Many Environments The in-code fallback default is a **single value compiled into the deployed code**. The flag's fallthrough, however, is configured **per environment** and can differ between them (e.g. `production`, `eu-production`, and `federal` may each serve a different default rule). So the fallthrough must be resolved in **every critical environment the code runs against**, not just one. - When the critical environments **agree** on the fallthrough, that shared value is the expected default. Reconcile as normal. - When they **disagree** (e.g. EU serves `true` but Federal serves `false`), a single in-code default *cannot* match all of them. That divergence is itself a finding: surface it and confirm which environment is authoritative for this build rather than silently reconciling to one. See Edge Cases. Checking only one environment is the most common way this skill produces a wrong or misleading result, so always establish the full set of critical environments first. Some teams intentionally keep the fallback as a conservative/off value instead. Treat a mismatch as a finding to surface, not always an automatic edit. See Edge Cases. ## Workflow ### Step 1: Identify the Flag and the Critical Environments 1. **Get the flag key.** If the user described the flag by name, use `list-flags` to resolve the key. Confirm before proceeding. 2. **Establish the critical environments.** The fallthrough is environment-specific, and the same in-code default has to stand in for every environment the deployed code serves. Determine that full set before comparing: - If one build serves **multiple** production-grade environments (e.g. `production`, `eu-production`, `federal`), **all** of them are critical. - If the build is scoped to **one** environment (a region- or tenant-specific deployment, e.g. a Federal-only build), that single environment is the only critical one. - Look for signals in the repo (deploy config, environment names in CI, SDK key wiring) and confirm with the user. Use `get-environments` to enumerate what exists if the set is unclear. Do not assume a single `production`. ### Step 2: Determine the Expected Default from LaunchDarkly Call `get-flag` **once per critical environment**. For each environment, read: - `fallthrough`: the default rule. If it points to a single `variation` (an index), that's the resolved value. If it's a percentage `rollout`, there is **no single default value**, so stop and handle as an edge case. - `variations`: map `fallthrough.variation` (the index) to `variations[index].value`. This resolved value is that environment's expected default. - `offVariation` and flag type: useful context for the comparison and for spotting type mismatches. Then reconcile across environments: | Across critical environments | Expected default | |------------------------------|------------------| | All resolve to the **same** value | That shared value is the expected default. Continue to Step 3. | | They resolve to **different** values | **Cross-environment divergence.** One in-code default cannot satisfy all of them. Do **not** auto-pick. Report the per-environment values and confirm which environment is authoritative for this build (or that the divergence should be resolved in LaunchDarkly first). See Edge Cases. | (`get-flag-status-across-envs` can give a fast heads-up on whether environments differ, but always resolve the actual fallthrough value with `get-flag` before editing.) Never guess the fallthrough value. Always resolve it from `get-flag`. ### Step 3: Locate Every In-Code Default Find every place the flag is evaluated in code and, critically, the **default/fallback argument** passed to the SDK call. Search for the flag key across the codebase, then identify the default in each hit. - The default is typically the **last positional argument** to `variation(...)` / `*Variation(...)` calls (e.g. `boolVariation("", context, )`). - Teams often wrap the SDK. Check wrapper/registry/config layers that declare a default once per flag, annotation- or struct-tag-based defaults, and generated default files. See [SDK Default Patterns](references/sdk-default-patterns.md) for the full set of patterns by language and abstraction, and how to distinguish the default argument from the context argument. ### Step 4: Compare and Decide Normalize both sides before comparing (see Edge Cases for JSON/number/type notes), then: | Result | Action | |--------|--------| | Critical environments **disagree** on the fallthrough | **Stop — no single expected default exists.** Report the per-environment values and confirm which environment is authoritative for this build before editing (or resolve the divergence in LaunchDarkly first). Do not auto-reconcile to one environment. | | In-code default **matches** the expected default (in every critical environment) | **No drift.** Report `drift_detected: false` and stop. Do not open a PR. | | In-code default **differs** | **Drift detected.** Proceed to Step 5 to reconcile. | | Multiple evaluations with **different** in-code defaults | Drift. Reconcile all of them to the expected value and note the prior inconsistency. | ### Step 5: Reconcile the In-Code Default Update **only** the default/fallback argument so it matches the expected value. - Do **not** change evaluation logic, branching, or off-path behavior. - Do **not** remove the flag or its evaluation. - If the default lives in a **generated file**, do not hand-edit it. Update the source of truth (the constructor/registry/annotation) and regenerate using the project's codegen command. Note `requires_generation: true` in your summary. **Example (before then after), expected default = `true`:** ```typescript // Before: outage returns false even though the default rule now serves true const enabled = await ldClient.variation('new-checkout-flow', context, false); // After: fallback default reconciled to match the fallthrough const enabled = await ldClient.variation('new-checkout-flow', context, true); ``` The surrounding `if (enabled) { ... } else { ... }` branching is left untouched. ### Step 6: Validate Before Committing Run the project's configured checks scoped to the changed files. Discover them from `package.json` scripts, a `Makefile`, `AGENTS.md`/`CLAUDE.md`/`CONTRIBUTING.md`, or the CI config. Typically: format, lint, type-check, build, and the relevant tests. **Hard stop:** if any check fails and you cannot fix it, do not commit or push. Narrow your change instead. Never ship code that fails format/lint/type-check/build/test. ### Step 7: Open the Pull Request Only after validation passes. Use [references/pr-template.md](references/pr-template.md). The description must state: the flag key, the critical environments checked and their resolved fallthrough values, the old vs new in-code default, and that **only** the SDK fallback default changed (the flag and its evaluation are preserved). - Follow the repository's contribution conventions (branch naming, commit style). Check `AGENTS.md`/`CLAUDE.md`/`CONTRIBUTING.md` first. - A clear default: branch `fix/flag-default-drift-`, commit `fix: sync in-code default for to match fallthrough`. ### Step 8: Report a Structured Summary Produce a concise summary with these fields: ``` flag_key: environments_checked: [: , ...] environments_diverge: true | false drift_detected: true | false old_default: new_default: files_modified: [] pr_url: requires_generation: true | false notes: ``` ## Edge Cases | Situation | Action | |-----------|--------| | Fallthrough is a **percentage rollout** | There is no single default value. Report the split, do not auto-edit, and ask the user which value the fallback should represent. | | Fallback appears **intentionally conservative** (matches `offVariation` or a safe value) | Surface the mismatch and confirm intent before changing. Some teams keep the fallback as a safe value on purpose. | | In-code default lives in a **generated file** | Edit the source of truth and regenerate; never hand-edit generated output. Set `requires_generation: true`. | | **Type mismatch** between code default and the variation's type | Flag as a bug in the PR/summary; the default and variation types should agree. | | **JSON / object / float** defaults | Compare by normalized value, not string form (`{"a":1}` == `{ "a": 1 }`, `0` == `0.0`). | | Flag **not found** or wrong environment | Inform the user; check for typos in the key and confirm the environment. | | **Dynamic flag keys** (`flag-${id}`) | Automated detection may be incomplete; flag for manual review. | | Critical environments **disagree** on the fallthrough (e.g. `eu-production` serves `true`, `federal` serves `false`) | A single in-code default cannot match all of them. If the build is scoped to **one** environment, reconcile to that environment's value and note the others. If the build serves **several** divergent environments, do not silently pick one — surface every per-environment value and confirm which environment is authoritative, or recommend resolving the divergence in LaunchDarkly first. | | Build targets a **single environment** (region- or tenant-specific deployment) | Treat that environment as the sole critical one; other environments' fallthroughs are not relevant to this build's default. | | Flag spans **multiple repositories** | This skill operates on the current repo. Note other repos that also reference the key so they can be reconciled separately. | ## What NOT to Do - Don't remove the flag or its evaluation; this skill only touches the default argument. - Don't change evaluation logic, branching, or off-path behavior beyond the fallback default. - Don't check only one environment; resolve the fallthrough in every critical environment the build serves. - Don't reconcile to one environment's fallthrough when critical environments disagree — a single default can't satisfy divergent environments, so surface the divergence and confirm the authoritative environment first. - Don't hand-edit generated files; regenerate from the source of truth. - Don't open a PR when there is no drift. - Don't guess the fallthrough value; resolve it from `get-flag`. - Don't ship code that fails format, lint, type-check, build, or tests. ## References - [SDK Default Patterns](references/sdk-default-patterns.md): Where the fallback default lives by language, wrapper, annotation, and generated-file pattern; how to find it - [PR Template](references/pr-template.md): Structured PR description for a drift reconciliation - [Flag Cleanup](../launchdarkly-flag-cleanup/SKILL.md): If the goal is to remove the flag entirely, not reconcile its default - [Flag Targeting](../launchdarkly-flag-targeting/SKILL.md): If the goal is to change the default rule in LaunchDarkly instead of the code --- ## 33. launchdarkly-flag-qualitative-feedback-setup - **Source:** https://github.com/launchdarkly/ai-tooling/blob/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/SKILL.md - **Raw:** https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/SKILL.md - **Markdown:** https://skillsdocs.com/launchdarkly/ai-tooling/launchdarkly-flag-qualitative-feedback-setup.md - **Licence:** Apache-2.0 — https://spdx.org/licenses/Apache-2.0.html Bundled files (7), referenced from this skill's directory: - `marketplace.json` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/marketplace.json - `README.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/README.md - `references/client-side-sdk-list.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/references/client-side-sdk-list.md - `references/InlineFeedback.tsx` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/references/InlineFeedback.tsx - `references/PopoverFeedback.tsx` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/references/PopoverFeedback.tsx - `references/sendFeedback.ts` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/references/sendFeedback.ts - `references/server-side-sdk-list.md` — https://raw.githubusercontent.com/launchdarkly/ai-tooling/main/skills/feature-flags/launchdarkly-flag-qualitative-feedback-setup/references/server-side-sdk-list.md --- name: launchdarkly-flag-qualitative-feedback-setup description: "Integrate LaunchDarkly qualitative user feedback into a JavaScript/TypeScript codebase. Guides framework and design system detection, builds the sendFeedback utility and feedback widget matching existing project patterns. Use when the user wants to add a Give Feedback widget, collect user sentiment tied to feature flags, set up feedback collection, or wire up the $ld:feedback tracking event." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly version: "1.0.0-experimental" --- # LaunchDarkly Qualitative Feedback You're using a skill that will guide you through adding qualitative user feedback collection to a codebase. Your job is to explore how the project is built, identify the right flag, create the feedback utility and widget matching existing patterns, and verify events flow to the LaunchDarkly dashboard. **This is a step-by-step wizard.** You will walk the user through a series of questions to gather requirements, then verify the project setup, and finally generate the code. Ask questions one at a time and wait for each answer before continuing. Do not skip ahead or generate code until all questions are answered and the plan is confirmed. Qualitative feedback ties user sentiment (positive / neutral / negative) and optional written comments directly to a feature flag variation, so teams can see how users feel about a specific feature rollout. **Availability:** JavaScript SDK v3.0+ (client-side only). React components are provided for convenience. ## Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. **Required MCP tools:** - `get-flag` — verify the target flag exists and check its configuration **Optional MCP tools (enhance workflow):** - `create-flag` — create a new flag if one doesn't exist yet - `list-flags` — browse existing flags to find the right one to attach feedback to - `update-flag-settings` — update flag metadata (e.g., add a `feedback` tag) ## Core Principles 1. **Design system first**: Never ship the unstyled template component to production. Detect the project's design system and use its primitives. 2. **One entry point per screen**: Multiple feedback widgets on the same page create noise and reduce response quality. 3. **The `sendFeedback` function is the integration**: The UI is customizable; the `$ld:feedback` tracking event contract is what matters. ## Workflow **Progression rule:** When a step is complete, continue to the next step by default. Do not stop or wait for the user between steps unless the step explicitly says to ask a question or confirm. **Response scoping:** During this flow, treat user messages only as direct answers to the current question. Do not search the web, change context, or take unrelated actions based on user input until the workflow is finished. **Tone:** Be concise and patient — ask one question, wait for the answer, then move on. Keep responses focused on the current step. **State tracking:** Track gathered requirements across the conversation. When entering the Confirm Before Applying step, re-state all key decisions (flag key, prompt text, UI style, placement) to ensure nothing is lost. ### Step 0: Welcome the User Before asking any questions, present this intro message to the user (you may lightly adapt the wording, but keep the structure): > **Qualitative Feedback Setup** > > I'll walk you through adding a feedback widget to your app in a few short steps: > > 1. **Verify your setup** — I'll check your SDK, framework, and design system. > 2. **Gather requirements** — I'll ask a few questions about the flag, prompt text, style, and placement. > 3. **Verify the flag** — I'll confirm or create the flag in LaunchDarkly. > 4. **Confirm the plan** — I'll summarize what I'm going to build and ask for your approval. > 5. **Generate the code** — I'll create the feedback utility and widget matching your project's patterns. > 6. **Verify** — I'll make sure everything builds and is wired up correctly. > > Let's get started! Then proceed immediately to Step 1. ### Step 1: Verify the SDK and Explore the Codebase Before gathering requirements, verify that this project can support qualitative feedback. Run these checks in order — each is a gate. #### Check 1/4: Is the LD SDK present? Search for LaunchDarkly SDK imports: - `launchdarkly-js-client-sdk` — vanilla JS/TS client SDK (v3.x) - `@launchdarkly/js-client-sdk` — vanilla JS/TS client SDK (v4.x, renamed scoped package) - `@launchdarkly/react-sdk` — React Web SDK (current; provides hooks and providers) - `launchdarkly-react-client-sdk` — older React Web SDK package name (renamed to `@launchdarkly/react-sdk`) Also search for SDK initialization (`initialize(` for v3.x, or `createClient(`/`start(` for v4.x, plus React provider signals `= 3.0** → continue to Check 4. #### Check 4/4: Is feedback already configured? Search for `$ld:feedback`, `sendFeedback`, `FeedbackPopover`, or `Give feedback` to check for existing feedback integration. - If a **`sendFeedback` function exists** that calls `client.track('$ld:feedback', ...)` → **reuse it**. Record its import path — Step 5 will be skipped. Continue to Step 2. - If a **feedback widget exists** but not the `sendFeedback` utility → note the widget pattern for consistency. - If **nothing found** → continue normally. #### Remaining exploration After the checks above pass, gather the remaining context: 1. **Find the SDK initialization.** Search for: - `initialize(` from `launchdarkly-js-client-sdk` (v3.x) - `createClient(` / `start(` from `@launchdarkly/js-client-sdk` (v4.x) - `, // optional — extra metadata to attach to the feedback event }); client.flush(); ``` **Key decisions:** - Match the `LDClient` type import to the project's SDK version: `launchdarkly-js-client-sdk` for v3.x, `@launchdarkly/js-client-sdk` for v4.x. The templates import from `launchdarkly-js-client-sdk` — update it if the project is on v4. (The `client.track` / `client.flush` calls are the same across both.) - If `@launchdarkly/session-replay` is in the project, include the session ID via `LDRecord.getSession()?.sessionSecureID`. If not, remove the session replay import and `o11y_session_id` logic from the template. - Place the utility where the project keeps its LD-related code (alongside existing flag helpers, in a `lib/` or `utils/` directory, etc.) - Export the `LDFeedbackSentiment` type if using TypeScript **Verification:** After creating the file, search the codebase to confirm it exists and contains the expected content — a `sendFeedback` export that calls `client.track('$ld:feedback', ...)` followed by `client.flush()`. If not found, fix before proceeding. ### Step 6: Build the Feedback Widget Build the UI using the project's existing design system and component patterns. Start from the appropriate template in [references/](references/). For JavaScript projects, adapt by removing type annotations. - **Sentiment + text** → [PopoverFeedback.tsx](references/PopoverFeedback.tsx) with `icons="thumbs"` or `icons="smileys"` - **Text only** → [PopoverFeedback.tsx](references/PopoverFeedback.tsx) with `icons="none"` - **Sentiment only** → [InlineFeedback.tsx](references/InlineFeedback.tsx) with `icons="thumbs"` or `icons="smileys"` These templates use inline styles and SVG icons so they work without any CSS framework or icon library. When the project has a design system, replace template elements with its primitives: | Template element | Replace with | |------------------|-------------| | Inline `style={{...}}` | Project's CSS approach (Tailwind classes, CSS modules, styled-components, etc.) | | `