Setting the file. One moment.
Skill 123 · Creating Data Lake Table
Subchapter 123.4
references/table-creation-glue-etl.mdMarkdown6 KBView on GitHub
Use when building Glue ETL pipelines that create and write to S3 Tables. Tables created via the S3 Tables API (aws s3tables create-table --metadata) are also readable by Spark.
--datalake-formats iceberg MUST be set as a job argumentIn Glue 5.x, catalog configs are static and MUST go in --conf job arguments. Using spark.conf.set() throws:
AnalysisException: Cannot modify the value of a static config: spark.sql.extensionsRule: All spark.sql.catalog.* configuration goes in --conf, never in the PySpark script.
Rule: Catalog and database names containing hyphens MUST be backtick-escaped in Spark SQL (e.g., `my-catalog`.`my-db`.my_table). Without backticks, Spark returns INVALID_IDENTIFIER.
| Method | Athena/Redshift access | Recommended |
|---|---|---|
| Analytics Integration (GlueCatalog) | Yes | Yes, if multi-service |
| REST Endpoint | No (Glue-only) | Yes, if Glue-only |
spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<name>.type=rest
spark.sql.catalog.<name>.uri=https://s3tables.<region>.amazonaws.com/iceberg
spark.sql.catalog.<name>.warehouse=<table_bucket_arn>
spark.sql.catalog.<name>.rest.sigv4-enabled=true
spark.sql.catalog.<name>.rest.signing-name=s3tables
spark.sql.catalog.<name>.rest.signing-region=<region>
spark.sql.catalog.<name>.io-impl=org.apache.iceberg.aws.s3.S3FileIOspark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<name>.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.<name>.glue.id=<account_id>:s3tablescatalog/<table_bucket_name>
spark.sql.catalog.<name>.warehouse=<table_bucket_arn>The warehouse parameter is required — without it Spark fails with “Cannot derive default warehouse location”.
The --conf argument is a single string with space-separated --conf key=value pairs:
"--conf": "spark.sql.catalog.s3tables=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.s3tables.type=rest --conf ..."First key-value has no --conf prefix. Use --cli-input-json file://config.json to avoid shell escaping.
job-config.json:
{
"Name": "my-etl-job",
"Role": "arn:aws:iam::<ACCOUNT>:role/<GLUE_ROLE>",
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://<BUCKET>/scripts/my_etl.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--datalake-formats": "iceberg",
"--conf": "spark.sql.catalog.s3tables=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.s3tables.type=rest --conf spark.sql.catalog.s3tables.uri=https://s3tables.<REGION>.amazonaws.com/iceberg --conf spark.sql.catalog.s3tables.warehouse=<TABLE_BUCKET_ARN> --conf spark.sql.catalog.s3tables.rest.sigv4-enabled=true --conf spark.sql.catalog.s3tables.rest.signing-name=s3tables --conf spark.sql.catalog.s3tables.rest.signing-region=<REGION> --conf spark.sql.catalog.s3tables.io-impl=org.apache.iceberg.aws.s3.S3FileIO",
"--catalog_name": "s3tables",
"--namespace": "<NAMESPACE>",
"--table_name": "<TABLE_NAME>"
},
"GlueVersion": "5.1",
"NumberOfWorkers": 2,
"WorkerType": "G.1X"
}For Analytics Integration, replace the --conf value with: spark.sql.catalog.s3tablescatalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.s3tablescatalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.s3tablescatalog.glue.id=<ACCOUNT>:s3tablescatalog/<BUCKET_NAME> --conf spark.sql.catalog.s3tablescatalog.warehouse=<TABLE_BUCKET_ARN>
Catalog config is in --conf, so the script is clean:
import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME', 'catalog_name', 'namespace', 'table_name'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
# Do NOT call spark.conf.set() for catalog config in Glue 5.x
# catalog_name must match spark.sql.catalog.<name> from --conf
spark.sql(f"""
CREATE TABLE IF NOT EXISTS {args['catalog_name']}.{args['namespace']}.{args['table_name']} (
col1 STRING,
col2 INT
)
USING iceberg
PARTITIONED BY (col1)
""")
# No LOCATION clause -- S3 Tables manages storage
job.commit()The Glue service role needs: AWSGlueServiceRole plus s3tables:GetTableBucket, s3tables:GetNamespace, s3tables:ListNamespaces, s3tables:CreateTable, s3tables:GetTable, s3tables:ListTables, s3tables:UpdateTableMetadataLocation, s3tables:GetTableMetadataLocation, s3tables:GetTableData, s3tables:PutTableData, and glue:GetCatalog, glue:GetDatabase, glue:GetTable, glue:passConnection. Table bucket, namespace, and Glue catalog MUST be created before the Glue job runs (Steps 3-5 in SKILL.md). For exact resource ARN scoping, see access-control.md.
| Issue | Fix |
|---|---|
| “Cannot modify static config” | Remove spark.conf.set(). Use --conf job argument. |
| “Access Denied” on S3 Tables | Check Glue role has granular s3tables: permissions. See IAM Requirements above. |
Shell escaping breaks --conf | Use --cli-input-json file://config.json. |
| Table not visible in Athena | REST endpoint tables aren’t in Athena. Use Analytics Integration. |
| Catalog not found | Ensure catalog name in script matches spark.sql.catalog.<name> from --conf. |
For latest Glue ETL guidance, search AWS docs for "Running ETL jobs on Amazon S3 tables with AWS Glue".