Plan Materialization

Write out every checkpoint and record-section into the plan (mandatory)

The plan is the durable, resumable source of truth: Phase 0 resumes from it, and it is what survives context compaction. Any step that exists only as prose in a approach reference — not as a discrete, tracked item in this plan — WILL be skipped after compaction. A parse-clean signal (has_import_errors: false) does NOT satisfy the test-run checkpoint and must never be recorded in a way that reads as “validated” or “complete”. Generate these sections BEFORE presenting the plan for confirmation; do not defer their creation to execution time.

Compaction-survival rule for skill delegations: When a step requires invoking another skill (e.g., testing-mwaa-workflow, debugging-mwaa-workflow, authoring-mwaa-workflow), the text you write for that step MUST include the word “skill” and name that skill explicitly. After compaction, the original approach reference text is gone — the plan file is the only artifact the agent reads on re-entry. A step that says “Test-run checkpoint [PENDING]” will be ad-hoc’d with manual REST API calls; a step that says “invoke skill testing-mwaa-workflow (delegated mode) and not manual DAG run triggers [PENDING]” will trigger the correct skill invocation. This applies to every step that delegates to a skill, not just the test-run checkpoint.

Open the chosen approach reference now and enumerate (a) every numbered step, (b) every step marked REQUIRED or flagged as a checkpoint, and (c) every ## record-section it names. Write each into the plan as a discrete PENDING, trackable item. At minimum the generated plan MUST contain:

• A ## Checklist section mirroring the chosen reference’s numbered steps plus Phase 8, each with a status field. Flag every action that needs confirmation (deploy to your current environment, create-environment, each version jump, switchover, decommission) and every step labeled REQUIRED. Multi-action steps must be decomposed — see the approach-specific sub-step rules below.

• Every required checkpoint from the reference as its own PENDING entry. By approach: New-environment upgrade — Step 4-checkpoint (test-run the new environment); Rehearsed in-place upgrade — Step 1-checkpoint (test-run the test copy; a hard prerequisite for the live upgrade), Step 3 re-check your current environment (REQUIRED); Direct in-place upgrade — Step 1 extra confirmation checkpoint, Step 4 test-run your current environment.

• A pre-seeded ## Test-run results section, with one row per deployed DAG in state PENDING (columns: DAG ID, run_id, State, Duration, Notes) and a header stating it is REQUIRED and must be all-SUCCESS or NEEDS-DECISION before switchover (New-environment upgrade) / before the live upgrade (Rehearsed in-place upgrade) / before marking complete (Direct in-place upgrade).

• The test-run checkpoint step in the Checklist MUST use this exact wording pattern (the location named varies by approach):

  • New-environment upgrade: Test-run checkpoint: run DAGs on the new environment — invoke skill testing-mwaa-workflow (delegated mode) and not manual DAG run triggers [PENDING, REQUIRED CHECKPOINT]
  • Rehearsed in-place upgrade: Test-run checkpoint: run DAGs on the test copy — invoke skill testing-mwaa-workflow (delegated mode) and not manual DAG run triggers [PENDING, REQUIRED CHECKPOINT]
  • Direct in-place upgrade: Test-run checkpoint: run DAGs on your current environment — invoke skill testing-mwaa-workflow (delegated mode) and not manual DAG run triggers [PENDING, REQUIRED CHECKPOINT]

  The word “skill” and the “not manual DAG run triggers” clause are not decorative — they are the re-entry instruction that survives compaction. Without them, a resuming agent will ad-hoc trigger-and-poll via the REST API instead of invoking the structured testing skill.

• For the New-environment upgrade, a ## Switchover progress table pre-seeded with one row per batch and columns: Batch, Current environment paused, New environment unpaused, Monitored one full cycle?, Advanced. The plan must not advance a batch until its Monitored one full cycle? cell is yes. The Rehearsed in-place upgrade has no switchover — omit this table for it; use the batch table’s per-DAG status for test-copy deployment tracking instead.

• A ## Where we are block: working_on, current_jump, jump_status, phase (which of the skill’s Phases 0–8 the upgrade is in), test_run_checkpoint (NOT-PASSED / PASSED). For the Rehearsed in-place upgrade, add live_upgrade_allowed — yes / no (yes only after the test-run checkpoint is PASSED) so a resuming agent cannot start the irreversible live upgrade on parse-only evidence.

• Phase 8 broken into discrete tracked items: 8.1: Batched switchover [PENDING] (New-environment upgrade only), 8.2: Stability period [PENDING], 8.3: Decommission [PENDING, NEEDS CONFIRMATION]. Do not track Phase 8 as a single line.

The batch table’s status vocabulary MUST distinguish parse-clean from test-run-verified. Use LOADED (uploaded, parses, zero import errors) as distinct from VERIFIED; LOADED is never a completion state. For multi-jump New-environment upgrade paths, add a Deploy Phase column (A: original or B: fixed) to distinguish starting-version original DAGs from post-target AF3-fixed DAGs — deploying Phase B code before the target version jump completes causes ModuleNotFoundError.

Approach-specific sub-step decomposition rules

Many numbered steps are multi-action procedures. If context compacts mid-step, the plan must record which sub-actions completed. The Checklist MUST decompose these steps into individually-tracked sub-items:

New-environment upgrade:

• Step 1 (create the new environment): track as 1a: create-environment + 1b: poll AVAILABLE. Do not proceed to Step 2 until status = AVAILABLE.
• Step 2 (Migrate Operational Metadata): track per-category: Variables: [PENDING/SKIP/DONE], Connections: [PENDING/SKIP/DONE], Pools: [PENDING/SKIP/DONE]. Pre-populate SKIP for categories handled by a secrets backend (detected in Phase 3 Step 4).
• Step 4 (Deploy Fixed DAGs): for multi-jump paths, split into 4A: Deploy original DAGs (starting-version validation) and 4B: Deploy fixed DAGs (post-target). The batch table must record which deployment phase each DAG is in. AF3-only imports (from airflow.sdk) MUST NOT be deployed until the target version jump completes.

Rehearsed in-place upgrade:

• Step 1b (Migrate Operational Config): track as a discrete item with per-category status (same as the New-environment upgrade’s Step 2). If skipped, the test run on the test copy fails with misleading “connection not found” errors unrelated to the version upgrade.
• Step 3 (Re-check your current environment): treat as REQUIRED. After the irreversible live upgrade, do not proceed to Step 5 (mark complete) without confirming your current environment is healthy (scheduler heartbeat advancing, no import errors).

Direct in-place upgrade: no additional decomposition required beyond the existing Phase 4 mandate (Steps 1 and 4 are already individually tracked).

Phase 5 decomposition (cross-major only)

When any version jump crosses into a new major version, Phase 5 (Fix Code) must be written into the plan as individually-tracked sub-items in the Checklist. The re-scan step is the critical one — --unsafe-fixes can introduce runtime-breaking patterns (notably _TaskDecorator.output) that are invisible to Ruff but cause AttributeError at parse time on the target version. If context compacts before the re-scan runs, the plan must still instruct the agent to perform it.

Write Phase 5 into the plan as:

The 5.FINAL step is a required checkpoint: do not proceed to deployment (Phase 7/8) until the re-scan is recorded as DONE with zero hits. Its ordering constraint (“AFTER all --unsafe-fixes”) must appear in the plan text — without it, a resuming agent may run the re-scan before applying fixes, get a clean result, and then introduce the very patterns the re-scan was meant to catch.

Plan format contract

The plan is parsed by the agent itself on resume, so its structure must be predictable. Every plan MUST follow this contract.

• Status vocabulary (fixed, no synonyms): PENDING, DONE, SKIP, DROPPED, LOADED (uploaded + parses; not yet test-run), VERIFIED (test run passed), NEEDS-DECISION (blocked on an external/infra dependency or a user call), and for checkpoints PASSED / NOT-PASSED. Do not invent other status words (no “in progress”, “partial”, “done-ish”).
• Last updated line: the first line after the plan title MUST be Last updated: <UTC ISO-8601 timestamp> - <one-line what-changed>, rewritten each time progress is saved. It is the freshness signal a resuming agent reads first.
• ## Where we are keys (fixed): working_on, current_jump, jump_status, phase (which of the skill’s Phases 0–8 the upgrade is in), test_run_checkpoint (PASSED / NOT-PASSED), and for the Rehearsed in-place upgrade live_upgrade_allowed — yes / no (yes only after the test-run checkpoint is PASSED).

Saving progress (maintain the plan during execution)

After completing any tracked item - a step, sub-step, version jump, checkpoint, batch, or per-DAG status - and BEFORE issuing the next action or command, save that change to the plan file. Never batch plan updates. Saving progress means updating three things together:

1. the item’s status token (from the fixed vocabulary above);
2. any newly-discovered durable facts - run_ids, environment ARNs, S3 object versions, NEEDS-DECISION reasons, and decisions - recorded in the relevant table or section, not just held in context;
3. the Last updated line and the ## Where we are block.

Recording durable facts (not only status flips) is what prevents detail - not just progress - from being lost when context compacts. “Before the next action” is what keeps multi-action steps (e.g. the rehearsal’s create/migrate/version-jump/deploy sub-steps) from drifting behind reality.

Never write secret values into the plan. The plan is a durable on-disk artifact. Record connection/variable migration as status tokens only (PENDING/SKIP/DONE); never store passwords, connection strings, variable values, or connection extra contents in it. Reference secrets by key/name only.

Self-consistency invariants

These invariants MUST always hold. A violation means the plan is mid-drift and Plan Reconciliation (below) must run before any further action. They are written against the plan’s roles so they hold for every approach; the “test-run checkpoint” is whichever checkpoint the chosen approach defines - the New-environment upgrade’s Step 4-checkpoint, the Rehearsed in-place upgrade’s Step 1-checkpoint, or the Direct in-place upgrade’s Step 4.

Plan reconciliation on resume

At session start and after any compaction, re-read the entire plan file before taking any action; never act on a remembered or summarized version. Then, before resuming work, run this reconciliation:

1. Check every invariant above.
2. If all hold, resume at ## Where we are.
3. If any invariant is violated, do NOT trust the plan to resolve itself - establish ground truth with read-only AWS calls:
   • aws mwaa get-environment --name <env> - does the env exist, at what version/status?
   • aws mwaa invoke-rest-api --name <env> --method GET --path /dags (paginate: loop offset by 100 until total_entries, per discovery-preflight.md Step 1.4) - are the DAGs deployed, and is has_import_errors false?
   • aws mwaa invoke-rest-api --name <env> --method GET --path /health - metadatabase/scheduler healthy, heartbeat advancing?
4. Repair the plan to match observed reality (correct the drifted statuses/checkpoint), and record what was repaired in the Last updated line.
5. If reality is genuinely ambiguous, surface it to the user - never guess.

A stale PASSED test-run checkpoint invites a different destructive next action per approach; reconciliation guards each: