Overview

Use this capability when the user asks to:

• “run my script on SageMaker” or execute Python on managed compute with S3 I/O
• run a transform with PyTorch, TensorFlow, Hugging Face, XGBoost, or sklearn dependencies
• run PySpark/Spark JAR distributed ETL or process large datasets
• run a custom processing container
• query Athena or Redshift directly into a processing job
• diagnose a failed, stopped, or stuck processing job

Dataset format conversion that is already covered by dataset-transformation stays there; that capability may call SageMaker Processing internally.

Operating assumptions

Decision flow

1. Check conversation and project files before asking for values.
2. Choose one branch:
   • Single Python script or BYOC → code_templates/script_processing.py
   • Framework source directory + requirements → code_templates/framework_processing.py
   • PySpark or Spark JAR → code_templates/spark_processing.py
   • Athena or Redshift query input → code_templates/dataset_definition_processing.py
   • Existing job failure/status → references/troubleshooting.md
3. Use a prebuilt image resolved through image_uris.retrieve when it fits. Follow the managed-image resolution and reproducibility rules in references/configuration.md; review the resolved URI before launch.
4. For a custom ECR image, use the read-only aws ecr describe-images procedure in references/configuration.md, present candidate tags/digests and timestamps, obtain user selection, and pin the selected digest. Never silently select a mutable latest tag.
5. For Athena/Redshift, SageMaker executes the query and stages results to the configured S3 URI; do not add a separate manual export step.

Gather inputs

Use cascading fallback: conversation → project files → ask one unresolved question at a time.

Required for launch: script/app or image, role, instance type/count, max runtime, input/output mapping, and wait behavior. Additionally:

• Framework: framework, exact supported version, Python version, source directory, optional requirements.txt
• Spark: PySpark script or JAR, optional Spark/Python/container version overrides, dependencies, multi-instance count. Apply the version policy from references/configuration.md.
• BYOC: ECR image URI and command/entrypoint expectations
• Athena: catalog, database, workgroup (optional), SQL, query-result S3 URI, output format
• Redshift: cluster, database, DB user, cluster role, SQL, query-result S3 URI, output format

For a Spark JAR, infer the fully qualified main class from the JAR manifest, build file (pom.xml, build.sbt, Gradle), or application documentation. Ask the user if it remains unresolved. SPARK_MAIN_CLASS is required only for APPLICATION_KIND="jar"; clear it for PySpark.

GPU-backed Processing Jobs are supported only with a GPU processing instance, compatible CUDA/framework or BYOC image, and sufficient regional quota. This reference has not validated a dedicated GPU-processing branch, so state that limitation and verify image compatibility/quota rather than assuming a CPU image will work.

Preflight and confirmation

Before generating code or submitting a job:

1. Verify region, execution-role trust, S3 paths, exact image availability, query permissions, and requested compute quota.
2. Validate that input/output local paths start with /opt/ml/processing/, because SageMaker Processing mounts input channels and collects declared outputs under that container path; paths outside it are not automatically populated or uploaded.
3. For GPU requests, additionally verify GPU-compatible image metadata and the relevant Processing instance quota.
4. Summarize branch, image/framework/version, script/query, S3 I/O, instance type/count, volume, max runtime, and wait behavior.
5. Link to SageMaker pricing (opens in a new tab) rather than calculating a dollar estimate.
6. Ask for explicit approval. Stop until approved.

Generate and launch

1. Read references/configuration.md and the selected template.
2. Write the customized output under the project directory in notebook or script mode.
3. Keep WAIT_FOR_COMPLETION=False unless the user asks to block.
4. Templates submit with wait=False, print job name/ARN immediately, and use logs_for_processing_job only when waiting.
5. Report output S3 locations after completion.

Diagnose an existing job

Read references/troubleshooting.md. Start with DescribeProcessingJob, then inspect the job’s CloudWatch stream. Return failure signature, root cause category, evidence, and fix; state what was not checked.

Out of Scope

The following are supported by SageMaker and AWS but do not have a validated workflow in this reference. If the user’s request matches one of these, let them know and proceed with best-effort guidance using general AWS knowledge:

• scheduled orchestration through SageMaker Pipelines, Step Functions, or EventBridge
• Ground Truth labeling jobs
• EMR/Glue cluster administration outside the processing job
• a dedicated, end-to-end validated GPU Processing branch

References

• code_templates/script_processing.py — plain script and BYOC
• code_templates/framework_processing.py — framework image, source directory, requirements
• code_templates/spark_processing.py — PySpark or Spark JAR
• code_templates/dataset_definition_processing.py — Athena or Redshift query input
• references/configuration.md — placeholders and branch settings
• references/troubleshooting.md — status and failure classification
• SageMaker Processing SDK reference (opens in a new tab)
• SageMaker prebuilt containers (opens in a new tab)