Phase 1 Analysis Checklist

Check the top-level files for the target service:

Extract: language, package manager, and any LLM provider SDKs already installed.

2. Provider imports

Grep the source tree for provider SDK imports so you know which one the app actually uses (dependencies can be unused):

3. Hardcoded model configs

Look for the three things that need to move into the config:

1. Model name — grep for string literals:
   • "gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "o1", "o1-mini"
   • "claude-opus-", "claude-sonnet-", "claude-haiku-", "claude-3-"
   • "gemini-", "mistral-", "meta.llama", "anthropic.claude-"
2. Parameters — grep for keys: temperature=, max_tokens=, maxTokens:, top_p=, topP:, top_k=, stop_sequences=
3. System prompts / instructions — grep for:
   • "role": "system" (OpenAI/Anthropic completion)
   • system=" or system: (Anthropic top-level system)
   • instructions=" (agent frameworks, CrewAI, LangGraph Python create_agent(system_prompt=) or legacy create_react_agent(prompt=), LangGraph.js createReactAgent({ prompt }))
   • Long triple-quoted strings above provider calls

For each hit, record the file path, line number, and current value.

4. External prompt files & registries

Prompts often live outside .py / .ts source. Open every plausible config file and read it before declaring the audit complete — code-only grep will miss prompts loaded from YAML, prompt-template registries, or framework-specific manifests, and the resulting config will not cover the real prompt surface area.

File-extension scan targets (run from repo root):

• **/*.yaml, **/*.yml
• **/*.json (exclude package.json, package-lock.json, tsconfig.json, *.lock)
• **/*.toml (exclude pyproject.toml already covered in §1)
• **/*.md under prompts/, templates/, agents/, personas/
• **/*.prompt, **/*.prompty, **/*.j2, **/*.jinja, **/*.tmpl

Content signals to grep inside those files:

• Keys named system, system_prompt, instructions, prompt, template, role, goal, backstory, persona, messages
• Mustache ({{ }}) or Jinja ({% %}) blocks suggesting a prompt template
• Long multi-line string values (>200 chars) under any of those keys

Framework-specific shapes to call out by name:

For each hit, record: file path, line range, the key holding the prompt, the loader call site that reads it, and any template placeholder syntax (Mustache vs Jinja vs Python .format()). The placeholder rewrite in Stage 2 sub-step 5 needs to know the source syntax to convert it to Mustache.

If a fallback file is already in use (see fallback-defaults-pattern.md — JSON/YAML loaded at startup), distinguish it from prompts that flow into the provider call. The fallback path is intentional infrastructure; only the latter migrates into the config.

5. Template placeholders in prompts

Anything the app currently interpolates into a prompt at runtime must be rewritten to Mustache {{ variable }} syntax in Stage 2 so the fallback path renders identically to the LD-served path. Grep for:

Record placeholder name + where the runtime value comes from (env var, function arg, datetime.now(), etc.). These get routed through variables={...} on completion_config / agent_config calls in Stage 2, and the literal prompt string gets rewritten to {{ placeholder }}. Leaving a non-Mustache placeholder in the fallback is a silent regression mode: LaunchDarkly-served prompts interpolate correctly, the fallback ships unrendered.

6. Hardcoded app-scoped knobs

Configuration that governs tool or app behavior rather than model behavior — easy to miss in an audit because it looks like ordinary application config. Common shapes:

• Context / Settings dataclass fields referenced by tools (max_search_results, chunk_size, retry_budget, timeout_ms, enable_reranking)
• Environment variables read inside tool implementations
• Constants declared in tools.py or a config module that a tool reads at call time

If a value changes agent behavior between variations — it belongs in the config. Stage 2 sub-step 5 (fallback) puts these in ModelConfig(custom={...}), not parameters (which is forwarded to the provider SDK and will crash on unknown kwargs). Tools read them via ai_config.model.get_custom("key").

7. Existing LaunchDarkly SDK usage

If LDClient / ldclient is already initialized in the codebase, reuse it — do not create a second base client in Stage 2. Grep for:

• Python: import ldclient, ldclient.set_config, ldclient.get()
• TypeScript/JS: @launchdarkly/node-server-sdk, init(LD_SDK_KEY), @launchdarkly/react-client-sdk
• Environment variables: LD_SDK_KEY, LAUNCHDARKLY_SDK_KEY, LAUNCHDARKLY_API_KEY

8. Mode decision: completion or agent

Walk the decision tree once per call site, using the call shape as the primary signal:

Default to completion mode when unclear — it is more flexible and is the only mode that supports judges attached via the LaunchDarkly UI (Stage 5).

9. Monorepo / multi-service scope

If the repo contains multiple services, ask the user which service to instrument. Do not migrate every service in one pass.

SDK routing table

Feeds into Stage 2 (install + wrap). Quoted from the ai-configs-relaunch-guides/AGENT-SETUP-PROMPT.md SDK routing table.

Node.js provider-specific helper packages (optional, for auto-tracking in Stage 4):

Python currently ships helper packages for OpenAI (ldai_openai) and LangChain (ldai_langchain). The LangChain Python package exposes create_langchain_model(config) (builds a LangChain chat model from the config, forwarding every variation parameter and mapping LD provider names to LangChain equivalents), convert_messages_to_langchain, and get_ai_metrics_from_response — the same package covers LangGraph. Use create_langchain_model(config) + track_metrics_of_async(get_ai_metrics_from_response, lambda: llm.ainvoke(messages)) as the canonical single-call pattern. See langchain-tracking.md (opens in a new tab) for both LangChain and LangGraph patterns and sdk-ai-tracker-patterns.md for the full tracker-method matrix.

Phase 1 output format

Return this shape to the user, then stop and wait for confirmation:

STOP

Do not proceed to Stage 2 until the user confirms all of:

1. The service boundary is right
2. The hardcoded targets list is complete
3. The mode choice matches their intent
4. The stage plan is acceptable (e.g. skip tools? skip evals for now?)
5. Coverage check. State the totals out loud: “I scanned the repo and found N hardcoded code targets, M externalized prompt files, and K registry pulls. If you expected more (e.g. you know there’s a prompts/ directory, a CrewAI agents.yaml, or a config service I didn’t reach), tell me where to look before we proceed.” A number the user can react to surfaces under-detection that a yes/no on a list cannot.

Reply with one of:

• confirm — the audit looks complete; proceed to Stage 2.
• add: <files or paths> — I missed something; here’s where to look. (Re-run the audit with the new locations and present an updated summary.)
• fix: <correction> — a target in the list is wrong (provider, mode, prompt content, etc.). (Update the summary and ask again.)
• stop — pause the migration here.

Do not interpret any other word — including skip, next, go, ok, proceed — as confirmation. If the reply doesn’t match one of the four forms above, ask the user to pick one. Do not proceed under ambiguity.