Seeing And Managing AI Crawler Traffic

Domain expertise for deciding, per AI scraper, whether to allow or block it in AWS WAF. Covers the AI activity visibility surface, the AI and large language model (LLM) bot labels Bot Control applies, category-based handling, and composing the AI labels into the existing bot confidence signal.

Does not cover turning Bot Control on (the protecting-against-bots reference) or the confidence signal chain in depth (its own references). This reference depends on Bot Control being enabled.

Execute commands using the AWS MCP server when connected (sandboxed execution, audit logging, observability). Fall back to the AWS CLI otherwise.

Table of Contents

• Overview
• Workflow
• See the AI traffic first
• Handle AI traffic by category
• Compose into the confidence signal
• Troubleshooting
• Procedure
• Security Considerations
• Additional Resources

Workflow

To see and manage AI crawler traffic end to end, follow the procedure exactly. See the Procedure section below.

The procedure covers:

• Using the AI activity surface to see which AI and LLM crawlers are arriving
• Reading the AI and LLM bot labels
• Handling that traffic by category (allow wanted crawlers, block or challenge unwanted ones)
• Optionally composing the AI labels into the existing confidence signal

See the AI traffic first

Customers want to allow some AI crawlers and block others but cannot see the AI traffic, so they cannot make the decision.

Constraints:

• You MUST surface the AI activity view and the AI and LLM bot labels so the customer can see which crawlers arrive before deciding on each
• You SHOULD confirm Bot Control is enabled, since the AI labels come from it

Handle AI traffic by category

Customers treat all AI crawler traffic as one block-or-allow decision when they want different handling per crawler.

Constraints:

• You MUST handle AI traffic by category using the bot labels, so the customer can allow wanted crawlers and block or challenge unwanted ones, rather than a single blanket decision

Compose into the confidence signal

A customer already forwarding a bot confidence signal can feed AI labels into the same chain rather than building a separate path.

Constraints:

• You SHOULD compose the AI labels into the existing confidence signal (see turning-bot-control-labels-into-a-confidence-signal) rather than building a separate AI path
• You MUST NOT duplicate the forwarding mechanism for AI traffic when a confidence signal already exists

Troubleshooting

The customer cannot see which AI crawlers are arriving

The AI activity surface or AI labels are not in view, or Bot Control is not enabled. Enable Bot Control and use the AI activity view (See the AI traffic first).

AI handling is all-or-nothing

The customer is making one blanket decision. Handle by category using the labels (Handle AI traffic by category).

Procedure

Overview

This procedure surfaces AI crawler traffic, handles it by category, optionally composes it into the confidence signal, and surfaces the console link.

Parameters

• web_acl_name, web_acl_id, scope (required): Identify the web ACL.
• category_handling (required): Which AI or LLM categories to allow, block, or challenge.

Constraints for parameter acquisition:

• You MUST ask for all required parameters upfront in a single prompt
• You MUST confirm Bot Control is enabled and producing AI labels

Steps

1. Verify dependencies

Constraints:

• You MUST confirm credentials with aws sts get-caller-identity
• You MUST confirm Bot Control is enabled and AI labels are present

2. Review AI traffic and add category handling

Constraints:

• You MUST review the AI activity and labels before deciding

• You MUST add label-match rules handling the AI categories per the customer’s decision, fetching the current LockToken before update-web-acl and passing the full rule set

• You MUST preserve the web ACL’s existing DefaultAction from the get-web-acl response and pass it back as {default_action}; do not assume Allow={}, since that would silently open all unmatched traffic on a web ACL whose default action is Block. For example, a Block rule on an unwanted AI crawler label:

  Copy

  aws wafv2 get-web-acl --name {web_acl_name} --scope {scope} --id {web_acl_id} --region {region}
  aws wafv2 update-web-acl --name {web_acl_name} --scope {scope} --id {web_acl_id} \
    --lock-token {lock_token} --default-action {default_action} \
    --rules '[{"Name":"BlockUnwantedAICrawler","Priority":1,"Action":{"Block":{}},"Statement":{"LabelMatchStatement":{"Scope":"LABEL","Key":"awswaf:managed:aws:bot-control:bot:category:ai"}},"VisibilityConfig":{"SampledRequestsEnabled":true,"CloudWatchMetricsEnabled":true,"MetricName":"BlockUnwantedAICrawler"}}]' \
    --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName={web_acl_name} \
    --region {region}

3. Optionally compose into the confidence signal and surface the console link

Constraints:

• You SHOULD feed the AI labels into the existing confidence signal rather than a separate path

• You MUST present the web ACL console link and tell the customer to confirm the AI handling rules:

  Copy

  https://us-east-1.console.aws.amazon.com/wafv2/homev2/web-acls?region={region}

Example

Example input

Example output

Troubleshooting

Cannot see AI crawlers

Enable Bot Control and use the AI activity view (Step 1).

Handling is all-or-nothing

Add per-category label-match rules (Step 2).

Security Considerations

This procedure modifies a security control, so misconfiguration directly weakens the application’s defenses.

• Least-privilege IAM. You MUST grant only the specific wafv2: actions a task needs (for example wafv2:GetWebACL and wafv2:UpdateWebACL) rather than wafv2:* or the AWSWAFFullAccess managed policy.
• Ephemeral credentials. You MUST use IAM roles with temporary credentials (such as an EC2 instance profile, SSO session, or aws sts assume-role) rather than long-lived IAM user access keys when running these WAF CLI commands.
• Monitor configuration changes. You SHOULD enable AWS CloudTrail on wafv2 management events and set CloudWatch alarms on critical web ACL configuration changes (such as DeleteWebACL and UpdateWebACL rule removals) and on the web ACL’s BlockedRequests and CountedRequests metrics, so rule changes and sudden spikes in blocked or counted traffic are detected.

Additional Resources

• AWS WAF Bot Control rule group (AWS WAF Developer Guide) (opens in a new tab)
• AWS WAF label match rule statement (AWS WAF Developer Guide) (opens in a new tab)