Setting the file. One moment.
Skill 54 · Ingesting Into Data Lake
Subchapter 54.2
references/bigquery-ingest.mdMarkdown4 KBView on GitHub
Move data from Google BigQuery into the data lake. Assumes a Glue BIGQUERY connection exists. If not, delegate to connecting-to-data-source.
BIGQUERY with service account credentials in Secrets Managerproject.dataset.table)bigquery.googleapis.com (public internet or Google Private Service Connect)bigquery_df = glueContext.create_dynamic_frame.from_options(
connection_type="bigquery",
connection_options={
"connectionName": args['connection_name'],
"parentProject": args['gcp_project'],
"sourceType": "table",
"table": "my_dataset.customers"
}
).toDF()For custom SQL:
connection_options={
"connectionName": args['connection_name'],
"parentProject": args['gcp_project'],
"sourceType": "query",
"query": "SELECT id, name, updated_at FROM `project.dataset.customers` WHERE country = 'US'"
}BigQuery billing note: the query reads bytes from table storage. Filter aggressively at source to minimize bytes scanned.
BigQuery has strong timestamp semantics. Watermark columns commonly used:
updated_at / last_modified_PARTITIONTIME / _PARTITIONDATE on partitioned tablesINFORMATION_SCHEMA.PARTITIONS.last_modified_time for partition-level freshnessExample incremental read with watermark filter:
query = f"""
SELECT *
FROM `{project}.{dataset}.{table}`
WHERE updated_at > TIMESTAMP('{last_watermark}')
"""See incremental-loading.md for watermark storage.
For time-partitioned BigQuery tables, use partition decorators to target specific partitions and reduce bytes scanned:
# Read only 2026-04 partitions
query = f"""
SELECT *
FROM `{project}.{dataset}.{table}`
WHERE _PARTITIONTIME BETWEEN TIMESTAMP('2026-04-01') AND TIMESTAMP('2026-04-30')
"""Clustered tables benefit similarly from filter push-down on clustering columns. Check clustering:
SELECT clustering_fields FROM `<project>.<dataset>.INFORMATION_SCHEMA.TABLES` WHERE table_name = '<table>';| BigQuery | Iceberg | Notes |
|---|---|---|
| STRING | STRING | |
| INT64, INTEGER | BIGINT | All BQ integers are 64-bit |
| NUMERIC | DECIMAL(38,9) | BQ NUMERIC is fixed precision |
| BIGNUMERIC | STRING | Iceberg DECIMAL caps at (38,38); store as STRING, cast on read |
| FLOAT64, FLOAT | DOUBLE | |
| BOOL, BOOLEAN | BOOLEAN | |
| BYTES | BINARY | |
| DATE | DATE | |
| TIME | STRING | Iceberg has no TIME type |
| DATETIME | TIMESTAMP | No timezone |
| TIMESTAMP | TIMESTAMPTZ | UTC-anchored |
| GEOGRAPHY | STRING | WKT or GeoJSON |
| STRUCT | STRUCT | |
| ARRAY | ARRAY | |
| JSON | STRING | Parse if needed |
BIGNUMERIC (up to 76.38 precision) exceeds Iceberg DECIMAL’s 38-digit cap. For full-precision needs, store as STRING and cast on read.