Skills

Skill 28 of 48

Dd Audit Security Investigation

Answer “who did what” security questions from Audit Trail — deletions, config changes, login activity, permission changes, actions from a specific user or IP.

1 minute · 243 words · 15 sections

Install

$ npx skills add datadog-labs/agent-skills --skill dd-audit-security-investigation

$ npx skills add datadog-labs/agent-skills

The first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.

Audit Trail: Security Investigation

Answer common security investigation questions using pup audit-logs.

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

Command Execution Order

Clarify the investigation scope: who, what resource type, what time window.
Run the most specific query first; broaden only if results are empty.
If results are large, pipe to jq to group or summarize.
Highlight anomalies: bulk operations, unusual geo, off-hours activity, support user actions.

Common Investigation Queries

Who deleted resources in a time window?

pup audit-logs search --query "@action:deleted" --from 24h -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Who modified a specific resource (by ID)?

pup audit-logs search --query "@asset.id:RESOURCE_ID" --from 7d -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name
    }]'

What did a specific user do?

pup audit-logs search --query "@usr.email:user@example.com" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

pup audit-logs search --query "@evt.name:Authentication @action:login" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      status: .attributes.attributes.status,
      ip: .attributes.attributes.network.client.ip,
      city: .attributes.attributes.network.client.geoip.city.name,
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name
    }]'

Failed logins only

pup audit-logs search --query "@evt.name:Authentication @action:login @status:error" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Who changed roles or permissions?

pup audit-logs search --query "@evt.name:\"Access Management\"" --from 30d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

What actions came from a specific IP?

pup audit-logs search --query "@network.client.ip:1.2.3.4" --from 30d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type
    }]'

Who created or deleted API keys?

pup audit-logs search --query "@evt.name:Authentication @asset.type:api_key" --from 90d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      key_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Event Category Reference

Category (`@evt.name`)	What it covers
`Authentication`	Logins, API key create/delete/modify
`Access Management`	Roles, user add/remove, restriction policies
`Dashboard`	Create, modify, delete, share
`Monitor`	Create, modify, delete, resolve
`Log Management`	Pipelines, indexes, archives, exclusion filters
`Integration`	Add/modify/delete integrations
`Metrics`	Custom metric create/modify/delete
`Organization Management`	Child org creation, org settings
`Notebook`	Create, modify, delete
`APM`	Retention filters, sampling config
`Cloud Security Platform`	CWS rules, security signal state changes
`Bits AI SRE`	MCP tool calls, AI investigations

Anomaly Flags to Surface

When presenting investigation results, call out:

Actor type SUPPORT_USER — Datadog support accessed the org
Bulk deletions — same user, same action, many resources in a short window
Unexpected geography — country not seen in prior logins for this user
Off-hours activity — actions at unusual times for the user’s typical timezone
First-time ASN — action from a cloud provider or VPN not seen before (@network.client.geoip.as.name)

Colophon

How this issue was made

Source

github.com/datadog-labs/agent-skills, branch main, last pushed 17 September 2026.

Licence

MIT — the text of every skill is reproduced unmodified, frontmatter included, under the upstream licence.

Discovery

48 skills found by walking the repository tree for SKILL.md, not by matching a directory convention. 11 distinct layouts observed: agent-observability/*/SKILL.md, */SKILL.md, dd-apm/k8s-ssi/*/SKILL.md, dd-apm/linux-ssi/*/SKILL.md, dd-apm/*/SKILL.md, dd-apps/*/SKILL.md, dd-audit/*/SKILL.md, dd-browser-sdk/*/SKILL.md, dd-security/csm/*/SKILL.md, dd-software-delivery/*/SKILL.md, SKILL.md (repo root).

Issue colours

Resolved from a curated brand profile — hue 299°, chroma 0.183. Two accent tones are generated per issue and each is proven against its own ground before it ships: a single accent that passes AA on both light and dark paper is arithmetically impossible.

Heading repairs

1 repair applied to this skill so the document has one h1 and no skipped levels:

Shifted “15 headings” from h1 to h2 so the skill title is the only h1.

Spec compliance

12 editorial notes across 12 of 48 skills. They are printed in the margin of each skill rather than as errors here.

Agent Observability Build Eval From Annotations — 1 note
Agent Observability Eval Pipeline — 1 note
Agent Observability Replay Trace — 1 note
Dd Audit AI Activity — 1 note
Dd Audit Compliance Report — 1 note
Dd Audit Cost Spike Investigation — 1 note
Dd Audit Key Compromise — 1 note
Dd Audit Security Investigation — 1 note
Upgrade Browser SDK V5 — 1 note
Upgrade Browser SDK V6 — 1 note

and 2 more.

Images

Images inside a skill come from the upstream repository. Where the author gave no alternative text we mark the image decorative rather than inventing a description — a plausible caption we made up is worse than none for the reader who depends on it.

Signal

Install counts come from skills.sh. They measure downloads, not quality, and an unranked repository is not an unread one.

Agent surfaces

The whole issue is available as one markdown document at /datadog-labs/agent-skills.md, and each skill at its own .md URL.

Publication

Set by Skills Docs from the source repository. Body text is Literata at the reader’s chosen size and measure; code is Geist Mono. Nothing on this page was written by us except this paragraph.

Audit Trail: Security Investigation

Answer common security investigation questions using pup audit-logs.

Prerequisites

pup auth login   # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope

Command Execution Order

Clarify the investigation scope: who, what resource type, what time window.
Run the most specific query first; broaden only if results are empty.
If results are large, pipe to jq to group or summarize.
Highlight anomalies: bulk operations, unusual geo, off-hours activity, support user actions.

Common Investigation Queries

Who deleted resources in a time window?

pup audit-logs search --query "@action:deleted" --from 24h -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Who modified a specific resource (by ID)?

pup audit-logs search --query "@asset.id:RESOURCE_ID" --from 7d -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name
    }]'

What did a specific user do?

pup audit-logs search --query "@usr.email:user@example.com" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

pup audit-logs search --query "@evt.name:Authentication @action:login" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      status: .attributes.attributes.status,
      ip: .attributes.attributes.network.client.ip,
      city: .attributes.attributes.network.client.geoip.city.name,
      country: .attributes.attributes.network.client.geoip.country.name,
      asn: .attributes.attributes.network.client.geoip.as.name
    }]'

Failed logins only

pup audit-logs search --query "@evt.name:Authentication @action:login @status:error" --from 7d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Who changed roles or permissions?

pup audit-logs search --query "@evt.name:\"Access Management\"" --from 30d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

What actions came from a specific IP?

pup audit-logs search --query "@network.client.ip:1.2.3.4" --from 30d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      action: .attributes.attributes.action,
      event: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type
    }]'

Who created or deleted API keys?

pup audit-logs search --query "@evt.name:Authentication @asset.type:api_key" --from 90d --limit 200 -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      key_id: .attributes.attributes.asset.id,
      ip: .attributes.attributes.network.client.ip,
      country: .attributes.attributes.network.client.geoip.country.name
    }]'

Event Category Reference

Category (`@evt.name`)	What it covers
`Authentication`	Logins, API key create/delete/modify
`Access Management`	Roles, user add/remove, restriction policies
`Dashboard`	Create, modify, delete, share
`Monitor`	Create, modify, delete, resolve
`Log Management`	Pipelines, indexes, archives, exclusion filters
`Integration`	Add/modify/delete integrations
`Metrics`	Custom metric create/modify/delete
`Organization Management`	Child org creation, org settings
`Notebook`	Create, modify, delete
`APM`	Retention filters, sampling config
`Cloud Security Platform`	CWS rules, security signal state changes
`Bits AI SRE`	MCP tool calls, AI investigations

Anomaly Flags to Surface

When presenting investigation results, call out:

Actor type SUPPORT_USER — Datadog support accessed the org
Bulk deletions — same user, same action, many resources in a short window
Unexpected geography — country not seen in prior logins for this user
Off-hours activity — actions at unusual times for the user’s typical timezone
First-time ASN — action from a cloud provider or VPN not seen before (@network.client.geoip.as.name)

Dd Audit Security Investigation

Audit Trail: Security Investigation

Prerequisites

Command Execution Order

Common Investigation Queries

Who deleted resources in a time window?

Who modified a specific resource (by ID)?

What did a specific user do?

Failed logins only

Who changed roles or permissions?

What actions came from a specific IP?

Who created or deleted API keys?

Event Category Reference

Anomaly Flags to Surface

References

Colophon

Dd Audit Security Investigation

Audit Trail: Security Investigation

Prerequisites

Command Execution Order

Common Investigation Queries

Who deleted resources in a time window?

Who modified a specific resource (by ID)?

What did a specific user do?

Failed logins only

Who changed roles or permissions?

What actions came from a specific IP?

Who created or deleted API keys?

Event Category Reference

Anomaly Flags to Surface

References

Colophon