Subchapter 18.6
references/troubleshooting.mdMarkdown16 KBView on GitHub
Common issues and solutions when working with Terraform Stacks.
Issue: Component A references Component B, and Component B references Component A.
Error Message:
Error: Cycle detected in component dependenciesSolutions:
# Before (circular dependency)
component "vpc" {
source = "./modules/vpc"
inputs = {
security_group_id = component.app.security_group_id # References app
}
}
component "app" {
source = "./modules/app"
inputs = {
vpc_id = component.vpc.vpc_id # References vpc
}
}
# After (broken circular reference)
component "vpc" {
source = "./modules/vpc"
inputs = {
# Remove reference to app
}
}
component "security_group" {
source = "./modules/security-group"
inputs = {
vpc_id = component.vpc.vpc_id
}
}
component "app" {
source = "./modules/app"
inputs = {
vpc_id = component.vpc.vpc_id
security_group_id = component.security_group.id
}
}Issue: Variable block validation errors during terraform stacks validate.
Error Message:
Error: Unsupported argument
on variables.tfcomponent.hcl line 5:
5: validation {
Validation blocks are not supported in Stack configurationsSolution: Remove validation blocks from variable declarations. Stacks do not support validation blocks:
# Incorrect
variable "instance_count" {
type = number
validation {
condition = var.instance_count > 0
error_message = "Instance count must be positive"
}
}
# Correct
variable "instance_count" {
type = number
description = "Number of instances (must be positive)"
}Move validation logic into the underlying modules if needed.
Issue: Variables fail validation when type is not specified.
Error Message:
Error: Missing required argument
on variables.tfcomponent.hcl line 3:
3: variable "region" {
The argument "type" is required in Stack variable declarationsSolution: Always specify type for variables - it’s required in Stacks (unlike traditional Terraform):
# Incorrect
variable "region" {
default = "us-west-1"
}
# Correct
variable "region" {
type = string
default = "us-west-1"
}Issue: Modules with embedded provider blocks cause errors.
Error Message:
Error: Provider configuration not allowed in module
Modules used with Terraform Stacks cannot contain provider blocksSolution:
Issue: The HCP Terraform UI doesn’t provide an option to destroy Stack deployments.
Why: Stack deployment destruction is only available through configuration, not the UI.
Solution: Set destroy = true in the deployment block and upload the configuration:
deployment "old_environment" {
inputs = {
aws_region = "us-west-1"
instance_count = 2
role_arn = local.role_arn
identity_token = identity_token.aws.jwt
}
destroy = true # Marks deployment for destruction
}Workflow:
destroy = true to the deployment blockterraform stacks configuration uploadImportant: You cannot destroy deployments from the UI. This is by design to prevent accidental destruction.
Issue: Deployment remains in “planning” state indefinitely.
Possible Causes:
.terraform.lock.hcl matches required providersDiagnosis:
# Get deployment step diagnostics
terraform stacks deployment-run list
# Note the run ID, then:
curl -s -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-runs/{run-id}/stack-deployment-steps" | \
jq '.data[] | {id, status: .attributes.status, component: .attributes["component-name"]}'Solutions:
Issue: Deployment is waiting for approval but CLI doesn’t show approval prompt.
Why: CLI monitoring commands are non-blocking and don’t automatically prompt for approval.
Solution:
Option 1: Approve via CLI
# Approve all pending plans in a deployment run
terraform stacks deployment-run approve-all-plans -deployment-run-id=sdr-ABC123
# Or approve all plans in a deployment group
terraform stacks deployment-group approve-all-plans -deployment-group=canaryOption 2: Configure auto-approve (Premium feature)
deployment_auto_approve "safe_changes" {
deployment_group = deployment_group.canary
check {
condition = context.plan.applyable
reason = "Plan must be successful"
}
}Issue: Provider authentication fails with OIDC/workload identity.
Error Messages:
Error: Error assuming role with web identity
Error: Failed to retrieve credentials
Error: Invalid identity tokenDiagnosis Steps:
# Check identity_token block exists
identity_token "aws" {
audience = ["aws.workload.identity"]
}
# Check deployment references the token
deployment "production" {
inputs = {
identity_token = identity_token.aws.jwt
}
}provider "aws" "this" {
config {
region = var.aws_region
assume_role_with_web_identity {
role_arn = var.role_arn
web_identity_token = var.identity_token
}
}
}AWS - Verify trust policy includes HCP Terraform:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account-id>:oidc-provider/app.terraform.io"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"app.terraform.io:aud": "aws.workload.identity"
},
"StringLike": {
"app.terraform.io:sub": "organization:<org-name>:project:<project-name>:stack:<stack-name>:deployment:<deployment-name>"
}
}
}
]
}Azure - Verify federated credential:
organization:<org>:project:<project>:stack:<stack>:deployment:<deployment>https://app.terraform.ioGCP - Verify workload identity pool:
google.subject from token claimsSolutions:
Issue: Provider version conflicts or “could not retrieve provider” errors.
Error Messages:
Error: Failed to install provider
Error: Provider version not found
Error: Checksum mismatch for providerSolutions:
terraform stacks providers-lockterraform stacks providers-lock \
-platform=linux_amd64 \
-platform=darwin_amd64 \
-platform=darwin_arm64required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.7.0" # Ensure version constraint is valid
}
}.terraform.lock.hcl to version controlIssue: Modules from the Terraform public registry cause errors during plan or apply.
Common Errors:
Error: Unsupported attribute
Error: Invalid reference
Error: Missing required argumentKnown Problematic Modules:
terraform-aws-modules/alb/aws - Some versions have compatibility issuesterraform-aws-modules/ecs-service/aws - May have issues with certain configurationsSolutions:
Test modules in dev deployment first before using in production
Check module compatibility by reviewing recent issues on the module repository
Use specific module versions rather than latest:
component "alb" {
source = "terraform-aws-modules/alb/aws"
version = "8.7.0" # Use specific version known to work
# ...
}# Instead of using a module that has issues
component "alb" {
source = "./modules/alb" # Create local module with raw resources
# ...
}Fork and fix modules if you have the resources to maintain them
Report compatibility issues to module maintainers
Issue: Stack can’t find local module sources.
Error Message:
Error: Module not found
Could not load module ./modules/vpcSolutions:
# Correct
component "vpc" {
source = "./modules/vpc"
}
# Incorrect (absolute paths don't work)
component "vpc" {
source = "/Users/username/project/modules/vpc"
}my-stack/
├── components.tfcomponent.hcl
└── modules/
└── vpc/
├── main.tf
├── variables.tf
└── outputs.tfIssue: Component output is not available to referencing component.
Error Message:
Error: Reference to unknown component
Component "vpc" has not been definedSolutions:
component "vpc" {
source = "./modules/vpc"
# Must define component before referencing it
}
component "app" {
source = "./modules/app"
inputs = {
vpc_id = component.vpc.vpc_id # Now valid
}
}# In modules/vpc/outputs.tf
output "vpc_id" {
value = aws_vpc.main.id
}component "regional" {
for_each = var.regions
# ...
}
component "app" {
inputs = {
# Correct - reference specific instance
vpc_id = component.regional["us-west-1"].vpc_id
# Incorrect - can't reference for_each component directly
# vpc_id = component.regional.vpc_id
}
}Issue: Deployment with deferred changes doesn’t complete after multiple iterations.
Error Message:
Error: Maximum deferred change iterations reachedCause: Dependency cycle or values that never stabilize.
Solutions:
Issue: API request for diagnostics returns empty results.
Request:
curl "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics"Response:
{
"data": []
}Solution: Add required stack_deployment_step_id query parameter:
curl "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics?stack_deployment_step_id={step-id}"Issue: No CLI command to retrieve Stack outputs after deployment.
Why: Currently no direct CLI command for outputs retrieval.
Solution: Use the artifacts API endpoint:
# Get final apply step ID first
APPLY_STEP=$(terraform stacks deployment-run list --json | \
jq -r '.[0].deployment_steps[] | select(.operation_type == "apply") | .id' | tail -1)
# Get outputs
curl -L -s -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-steps/$APPLY_STEP/artifacts?name=apply-description" | \
jq -r '.outputs | to_entries | .[] | "\(.key): \(.value.change.after)"'Issue: Commands like terraform stacks deployment-run watch never return in CI/CD pipelines.
Why: Watch commands stream output indefinitely and are designed for interactive use.
Solution: Use API polling instead of watch commands. See api-monitoring.md for complete workflow.
Issue: Request to artifacts endpoint returns 404 Not Found.
Possible Causes:
Solution:
# Check step status first
curl -s -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}" | \
jq '.data.attributes.status'
# Only request artifacts when status is "completed"
if [ "$STATUS" = "completed" ]; then
curl -L -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-description"
fiIssue: Artifacts endpoint returns redirect response instead of artifact content.
Why: The endpoint returns HTTP 307 redirect to the actual artifact URL.
Solution: Configure HTTP client to follow redirects:
# curl: Use -L flag
curl -L -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-description"
# Python requests: allow_redirects=True (default)
import requests
response = requests.get(url, headers=headers, allow_redirects=True)
# Node.js fetch: redirect: 'follow' (default)
const response = await fetch(url, {
headers: headers,
redirect: 'follow'
});For more detailed error information, enable debug logging:
# CLI commands
TF_LOG=DEBUG terraform stacks validate
TF_LOG=DEBUG terraform stacks configuration upload
# API artifacts
# Request the debug-log artifact instead of description
curl -L -H "Authorization: Bearer $TOKEN" \
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-debug-log"If experiencing widespread issues, check HCP Terraform status page:
List recent configurations to identify when issues started:
terraform stacks configuration listFor issues not covered here: