Skills

Skill 27 of 48

Dd Audit Key Compromise

Investigate a potentially compromised Datadog API key — timeline of actions, geo/IP breakdown, endpoints called, anomaly flags, and remediation steps.

1 minute · 167 words · 12 sections

Install

$ npx skills add datadog-labs/agent-skills --skill dd-audit-key-compromise

$ npx skills add datadog-labs/agent-skills

The first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.

Audit Trail: API Key Compromise Investigation

Reconstruct what a Datadog API key did, where requests originated, and which resources were affected.

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

You need the key ID of the suspect key (not the key value). Find it in Datadog UI under Organization Settings > API Keys, or from context showing @metadata.api_key.id.

Investigation Workflow

Step 1 — Establish timeline

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      endpoint: .attributes.attributes.http.url_details.path,
      method: .attributes.attributes.http.method,
      ip: .attributes.attributes.network.client.ip,
      city: .attributes.attributes.network.client.geoip.city.name,
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name
    }]'

Step 2 — Geo/IP breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name,
      ip: .attributes.attributes.network.client.ip
    }]
    | group_by(.country)
    | map({
        country: .[0].country,
        count: length,
        asns: [.[].asn] | unique,
        ips: [.[].ip] | unique
      })
    | sort_by(-.count)'

Step 3 — Endpoint breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      method: .attributes.attributes.http.method,
      path: .attributes.attributes.http.url_details.path
    }]
    | group_by(.path)
    | map({path: .[0].path, methods: [.[].method] | unique, count: length})
    | sort_by(-.count)'

Step 4 — Destructive action check

pup audit-logs search --query "@metadata.api_key.id:KEY_ID @action:deleted" --from 90d -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Step 5 — When was the key created and by whom?

pup audit-logs search --query "@asset.type:api_key @asset.id:KEY_ID @action:created" --from 90d -o json \
  | jq '[.data[] | {
      created_at: .attributes.timestamp,
      created_by: .attributes.attributes.usr.email,
      creator_ip: .attributes.attributes.network.client.ip,
      creator_country: .attributes.attributes.network.client.geoip.country.name
    }]'

Anomaly Flags

Signal	Why it matters
Country not in org’s normal baseline	Possible exfiltration from unexpected region
ASN is a cloud/VPN provider (AWS, Cloudflare, NordVPN, etc.)	Proxied traffic; obscured origin
DELETE actions on monitors, dashboards, or log pipelines	Potential sabotage
Burst of activity in short window	Automated scraping or bulk exfiltration
Activity outside business hours	Off-hours access
Key used from multiple IPs simultaneously	Key shared or stolen

Investigation Output Format

Key ID: <key_id>
Created: <timestamp> by <user_email>
Active period: <first_seen> to <last_seen>
Total events: <N>

Origins:
  - <Country> (<ASN>): <N> events — [NORMAL / FLAG: first-time origin]

Endpoints called (top 5):
  - <METHOD> <path>: <N> calls

Destructive actions: <N> deletions — [resource types affected]

Recommended actions:
  1. Revoke the key immediately if not already done
  2. Review affected resources: [list]
  3. Check if any deleted resources need restoration
  4. Audit who else had access to this key

Remediation

Revoke in Datadog UI: Organization Settings > API Keys > Revoke.

Or via API (requires manage_api_keys scope):

pup api-keys delete KEY_ID

Colophon

How this issue was made

Source

github.com/datadog-labs/agent-skills, branch main, last pushed 17 September 2026.

Licence

MIT — the text of every skill is reproduced unmodified, frontmatter included, under the upstream licence.

Discovery

48 skills found by walking the repository tree for SKILL.md, not by matching a directory convention. 11 distinct layouts observed: agent-observability/*/SKILL.md, */SKILL.md, dd-apm/k8s-ssi/*/SKILL.md, dd-apm/linux-ssi/*/SKILL.md, dd-apm/*/SKILL.md, dd-apps/*/SKILL.md, dd-audit/*/SKILL.md, dd-browser-sdk/*/SKILL.md, dd-security/csm/*/SKILL.md, dd-software-delivery/*/SKILL.md, SKILL.md (repo root).

Issue colours

Resolved from a curated brand profile — hue 299°, chroma 0.183. Two accent tones are generated per issue and each is proven against its own ground before it ships: a single accent that passes AA on both light and dark paper is arithmetically impossible.

Heading repairs

1 repair applied to this skill so the document has one h1 and no skipped levels:

Shifted “12 headings” from h1 to h2 so the skill title is the only h1.

Spec compliance

12 editorial notes across 12 of 48 skills. They are printed in the margin of each skill rather than as errors here.

Agent Observability Build Eval From Annotations — 1 note
Agent Observability Eval Pipeline — 1 note
Agent Observability Replay Trace — 1 note
Dd Audit AI Activity — 1 note
Dd Audit Compliance Report — 1 note
Dd Audit Cost Spike Investigation — 1 note
Dd Audit Key Compromise — 1 note
Dd Audit Security Investigation — 1 note
Upgrade Browser SDK V5 — 1 note
Upgrade Browser SDK V6 — 1 note

and 2 more.

Images

Images inside a skill come from the upstream repository. Where the author gave no alternative text we mark the image decorative rather than inventing a description — a plausible caption we made up is worse than none for the reader who depends on it.

Signal

Install counts come from skills.sh. They measure downloads, not quality, and an unranked repository is not an unread one.

Agent surfaces

The whole issue is available as one markdown document at /datadog-labs/agent-skills.md, and each skill at its own .md URL.

Publication

Set by Skills Docs from the source repository. Body text is Literata at the reader’s chosen size and measure; code is Geist Mono. Nothing on this page was written by us except this paragraph.

Audit Trail: API Key Compromise Investigation

Reconstruct what a Datadog API key did, where requests originated, and which resources were affected.

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

You need the key ID of the suspect key (not the key value). Find it in Datadog UI under Organization Settings > API Keys, or from context showing @metadata.api_key.id.

Investigation Workflow

Step 1 — Establish timeline

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      endpoint: .attributes.attributes.http.url_details.path,
      method: .attributes.attributes.http.method,
      ip: .attributes.attributes.network.client.ip,
      city: .attributes.attributes.network.client.geoip.city.name,
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name
    }]'

Step 2 — Geo/IP breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name,
      ip: .attributes.attributes.network.client.ip
    }]
    | group_by(.country)
    | map({
        country: .[0].country,
        count: length,
        asns: [.[].asn] | unique,
        ips: [.[].ip] | unique
      })
    | sort_by(-.count)'

Step 3 — Endpoint breakdown

pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
  | jq '[.data[] | {
      method: .attributes.attributes.http.method,
      path: .attributes.attributes.http.url_details.path
    }]
    | group_by(.path)
    | map({path: .[0].path, methods: [.[].method] | unique, count: length})
    | sort_by(-.count)'

Step 4 — Destructive action check

pup audit-logs search --query "@metadata.api_key.id:KEY_ID @action:deleted" --from 90d -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Step 5 — When was the key created and by whom?

pup audit-logs search --query "@asset.type:api_key @asset.id:KEY_ID @action:created" --from 90d -o json \
  | jq '[.data[] | {
      created_at: .attributes.timestamp,
      created_by: .attributes.attributes.usr.email,
      creator_ip: .attributes.attributes.network.client.ip,
      creator_country: .attributes.attributes.network.client.geoip.country.name
    }]'

Anomaly Flags

Signal	Why it matters
Country not in org’s normal baseline	Possible exfiltration from unexpected region
ASN is a cloud/VPN provider (AWS, Cloudflare, NordVPN, etc.)	Proxied traffic; obscured origin
DELETE actions on monitors, dashboards, or log pipelines	Potential sabotage
Burst of activity in short window	Automated scraping or bulk exfiltration
Activity outside business hours	Off-hours access
Key used from multiple IPs simultaneously	Key shared or stolen

Investigation Output Format

Key ID: <key_id>
Created: <timestamp> by <user_email>
Active period: <first_seen> to <last_seen>
Total events: <N>

Origins:
  - <Country> (<ASN>): <N> events — [NORMAL / FLAG: first-time origin]

Endpoints called (top 5):
  - <METHOD> <path>: <N> calls

Destructive actions: <N> deletions — [resource types affected]

Recommended actions:
  1. Revoke the key immediately if not already done
  2. Review affected resources: [list]
  3. Check if any deleted resources need restoration
  4. Audit who else had access to this key

Remediation

Revoke in Datadog UI: Organization Settings > API Keys > Revoke.

Or via API (requires manage_api_keys scope):

pup api-keys delete KEY_ID

Dd Audit Key Compromise

Audit Trail: API Key Compromise Investigation

Prerequisites

Investigation Workflow

Step 1 — Establish timeline

Step 2 — Geo/IP breakdown

Step 3 — Endpoint breakdown

Step 4 — Destructive action check

Step 5 — When was the key created and by whom?

Anomaly Flags

Investigation Output Format

Remediation

References

Colophon

Dd Audit Key Compromise

Audit Trail: API Key Compromise Investigation

Prerequisites

Investigation Workflow

Step 1 — Establish timeline

Step 2 — Geo/IP breakdown

Step 3 — Endpoint breakdown

Step 4 — Destructive action check

Step 5 — When was the key created and by whom?

Anomaly Flags

Investigation Output Format

Remediation

References

Colophon