Setting the file. One moment.
Skill 50 · Connecting To Data Source
Subchapter 50.1
references/bigquery-setup.mdMarkdown3 KBView on GitHub
AWS Glue native BigQuery connection (type BIGQUERY). Authentication is via a GCP service account; credentials flow through AWS Secrets Manager.
roles/bigquery.dataViewer plus roles/bigquery.jobUser for running jobs)Service account and key generation happen in GCP, not AWS. For current steps see GCP service account docs (opens in a new tab) and BigQuery access control (opens in a new tab).
Minimum GCP IAM roles for read-only ingestion:
roles/bigquery.dataViewer on the target datasetroles/bigquery.jobUser on the project (to run queries)For cross-project reads, grant both roles in each source project.
Base64-encode the service account JSON and store in Secrets Manager. The Glue BigQuery connection expects the secret value to be the base64 string directly, not a JSON wrapper.
base64 -i <service-account>.json | tr -d '\n' > sa.b64
aws secretsmanager create-secret \
--name glue/bigquery/<project-id>/credentials \
--secret-string file://sa.b64 \
--region <region>
rm sa.b64Rotate by creating a new key in GCP and updating the secret value. Glue picks up the new value on next job run.
{
"Name": "bigquery-<project-id>",
"ConnectionType": "BIGQUERY",
"ConnectionProperties": {
"SECRET_ID": "glue/bigquery/<project-id>/credentials"
}
}Glue’s BigQuery connection talks to Google APIs over the internet. No PhysicalConnectionRequirements needed unless the Glue job itself must run in a specific VPC for other reasons (e.g., also reading from a private RDS). In that case, ensure the subnet has NAT gateway egress so Glue can reach bigquery.googleapis.com.