Iceberg Catalog Config And Usage

Managed Iceberg tables with automatic compaction, snapshot management, and multi-engine access.

• Catalog path: The table bucket is configured in --conf via glue.id, so the write path is 3-part: s3tablescatalog.<namespace>.<table>
• No LOCATION clause in CREATE TABLE
• Table and column names must be lowercase
• Requires Glue 5.1 or higher and --datalake-formats iceberg job argument
• All spark.sql.catalog.* config goes in --conf job arguments, never in spark.conf.set() (Glue 5.x static config restriction)
• Delegate table creation to creating-data-lake-table (opens in a new tab)

Two access methods exist. Use Analytics Integration when the table needs to be visible to Athena, Redshift, or EMR. Use REST Endpoint when only Glue Spark jobs access the table.

Analytics Integration (recommended for multi-engine access):

The warehouse parameter is required. Without it Spark fails with “Cannot derive default warehouse location”.

REST Endpoint (Glue-only access):

Tables created via REST are NOT visible in Athena or Redshift.

--conf format in Glue DefaultArguments: Pass as a single string. First pair has no --conf prefix; subsequent pairs are space-separated with --conf prefix:

Use --cli-input-json file://config.json to avoid shell escaping issues.

Write path (PySpark):

Standard Iceberg on General Purpose Bucket

Self-managed Iceberg tables on regular S3 buckets. User handles compaction and snapshot cleanup.

• Catalog path: glue_catalog.<database>.<table> (via Glue Data Catalog)
• LOCATION clause IS required: LOCATION 's3://<bucket>/<prefix>/'
• Registered in Glue Data Catalog as normal
• Works with Glue 5.1 or higher and --datalake-formats iceberg job argument
• All spark.sql.catalog.* config goes in --conf job arguments, never in spark.conf.set()

Glue job catalog config:

The warehouse parameter sets the default base path for new tables.

Write path (PySpark):

Athena DDL:

Parquet / ORC / CSV on S3

Raw files written to S3 with no Iceberg table metadata. Queryable via external tables in Athena.

• No table management (no compaction, no snapshots, no schema evolution)
• User must create an external table in Glue catalog to query with Athena
• Suitable when the user explicitly wants raw files, not a managed table

Write path (PySpark):

External table for querying:

Gotchas

• S3 Tables CREATE TABLE must NOT include a LOCATION clause. Standard Iceberg MUST include one.
• The s3tablescatalog federated catalog uses slash-separated paths in Athena: "s3tablescatalog/<bucket>"."<namespace>"."<table>". Spark uses dot-separated: s3tablescatalog.<namespace>.<table> (the bucket is configured in --conf via glue.id).
• Parquet/ORC/CSV targets do not create Iceberg metadata – they are raw files only. No schema evolution, time travel, or ACID transactions.
• Discover available MCP tools by keyword search – do not hardcode tool names.

Engine Access Patterns

How each engine reads and writes to each target format. Use this when building jobs that read from one format and write to another, or when validating ingested data.

S3 Tables

Spark with Analytics Integration and Athena both use 4 levels, but Athena uses slash-separated catalog paths while Spark uses dots. Spark with REST uses 3 levels because the table bucket is embedded in the --conf warehouse ARN.

Standard Iceberg

Standard Iceberg tables are registered in the default Glue Data Catalog. Athena queries them without a catalog prefix. Spark requires the catalog name from --conf (e.g., glue_catalog).

Parquet / ORC / CSV

No catalog registration needed for Spark reads — point directly at the S3 path. Athena requires an external table definition in the Glue catalog.

Decision Guide