Run Source Model Baseline

References

• source_provider: openai | anthropic | google
• source_model_id: the model ID to call (e.g., gpt-4o, claude-3-5-sonnet-20241022, gemini-1.5-pro)
• golden_dataset_path: usually <REPO>/.saws-migrate/golden-dataset/prompts.jsonl
• output_path: usually <REPO>/.saws-migrate/eval-results/source_baselines.jsonl

Preconditions

• <REPO>/.saws-migrate/.source-provider-env exists (written by the orchestration skill’s Phase B3 when the user provided a source-provider API key). The file contains a single KEY=VALUE line, one of: OPENAI_API_KEY=... / ANTHROPIC_API_KEY=... / GEMINI_API_KEY=....
• The golden dataset JSONL exists at golden_dataset_path and each entry has at least id, user_prompt, optionally system_prompt.

If the env file is absent, do NOT run this skill — the caller should set live_source_baseline: false and skip to static baselines.

Procedure

Step 1: Verify the env file

• absent → return immediately with status: "skipped".
• present + format_bad → the file exists but has no parseable KEY=VALUE line (e.g. a bare key was written without the env-var prefix). Do NOT proceed — the resolver would silently hit the no_key path. Return status: "skipped" with a note telling the caller the env file is malformed and needs re-collection in KEY=VALUE form. Never print the file’s contents.
• present + format_ok → continue.

Step 1.5: Resolve the source model ID against the live provider catalog

The source_model_id you were handed is the user’s STATED source model (extracted by llm2bedrock-code-analyzer / log-ingestor from the customer’s source code or plan). It might be a slight misspelling, a stale alias, or a date-suffixed variant compared to what the provider actually exposes right now. Before running the baseline, check that the ID exists in the provider’s live catalog and resolve to the closest valid variant of the SAME model line if needed.

🚫 Hard rule — model line is sacred. You may auto-resolve gpt-5.4 to gpt-5.4-2024-08-06 (a date-pinned variant of the same model). You MUST NOT resolve it to gpt-5, gpt-5.5, gpt-4o, gpt-5.4-mini, gpt-5.4-pro, gpt-5.4-latest, or any other model line / alias. The migration report’s pass rate is meaningful only when the live baseline is the SAME deterministic model the customer said they were running. A cross-line swap or a moving alias is worse than no baseline.

Run the committed resolver script <scriptsDir>/resolve_source_model.py (do NOT write an ad-hoc script — instructions only ever run committed scripts, with paths passed as arguments). The resolver:

1. Calls the provider’s list-models endpoint with the env key:
   • OpenAI: GET https://api.openai.com/v1/models
   • Anthropic: GET https://api.anthropic.com/v1/models
   • Gemini: GET https://generativelanguage.googleapis.com/v1beta/models (key via x-goog-api-key header, never a query parameter)
2. Looks for, in order:
   • Exact match for source_model_id → use unchanged.
   • Safe prefix match: a catalog ID that starts with <source_model_id>- AND whose suffix is a date (YYYY-MM-DD) or pure version number (e.g. 2, 3.1). These are the same model pinned to a date or version — same-line. Examples: gpt-5.4 matches gpt-5.4-2026-03-05; claude-3-5-sonnet matches claude-3-5-sonnet-20241022. Does NOT match gpt-5, gpt-5.5, gpt-54 (different lines), nor gpt-5.4-mini, gpt-5.4-pro, gpt-5.4-latest (alphabetic suffixes — those are either different model lines or non-deterministic aliases). If multiple safe variants, pick the shortest ID (most general).
   • Ambiguous prefix (catalog has prefix hits but ALL of them have alphabetic / alias suffixes — mini, nano, pro, turbo, latest, codex, etc.): do NOT auto-pick. Surface the prefix hits as not_found candidates so the user picks the right model line themselves.
   • No match → emit a JSON record with the top 5 catalog IDs whose names share the longest common prefix with source_model_id, for the caller to show the user.

The script detects the provider from whichever *_API_KEY the env file carries (all three are implemented — OpenAI, Anthropic, Gemini) and calls only that provider’s list-models endpoint. The resolution rules above are the behavior contract, unit-locked in test_resolve_source_model.py.

Run it, passing the plan model id via env and the env-file path as the argument:

SOURCE_PROVIDER (this skill’s declared input) is authoritative for provider selection — if the env file carries several provider keys, the stated provider’s key is used and a missing key is a hard no_key, never a guess.

Interpret the JSON output:

If the orchestration skill later re-invokes the baseline with a user-chosen candidate, the caller appends to notes: live baseline used <chosen_id> (selected from candidates after plan id <source_model_id> not found, confirmed by user).

NEVER silently substitute a different model line. The prefix rule above is the only automatic substitution allowed.

Step 2: Run the baseline script

Run the committed <scriptsDir>/source_baseline.py (do NOT write an ad-hoc script). It detects the provider from the env file, builds the provider-correct request shape (OpenAI max_completion_tokens, Anthropic top-level system, Gemini systemInstruction — shapes unit-locked in test_source_baseline.py), applies the partial-resume guard (prompts with a live row in the output are never re-billed; failed rows are retried), and writes the output contract below.

Pass SOURCE_MODEL_ID, GOLDEN_DATASET_PATH, OUTPUT_PATH via env and the env-file path as the argument:

Step 3: Classify the result

Parse the printed live K/N line and inspect the JSONL.

Security

• Never echo, log, or include the API key value in any returned notes or output (including any blocked detail). The key only lives in <REPO>/.saws-migrate/.source-provider-env on the local host.
• The script reads the key from the env file into os.environ only — never writes it to stdout or to the output JSONL.
• The key is only ever sent as an auth header to its OWN provider’s official endpoint — api.openai.com, api.anthropic.com, or generativelanguage.googleapis.com — never to any other host, and never as a URL query parameter (query strings end up in logs). The endpoint set is pinned by test_source_baseline.py.

Output contract

source_baselines.jsonl, one record per line:

Only records with status: "live" carry a usable source_response. The caller (llm2bedrock-prompt-evaluator Step 4) merges these into raw_results.jsonl and falls back to the dataset’s stored assistant_response for any prompt without a live response.