Subchapter 78.11
references/redshift-storage-remediation.mdMarkdown38 KBView on GitHub
The action side of references/redshift-storage-utilization.md (attribute there first) — establish Serverless vs provisioned before acting (SKILL.md STEP 0), since VACUUM/resize behavior differs.
A cluster stuck in modifying/restarting is not always ~100% storage (S12) — that pattern-match onto storage has been the single most costly mistake in this scenario family. Before landing on S12 or S18, check, in this order:
STL_/STV_ disk views don’t even exist there to confirm one. There is no customer-side force, cancel, or pg_terminate_backend that clears it and deleting/recreating is not the first move: escalate to the Redshift service team to unstick the workflow. Do not prescribe adding/removing RPU or reducing data as the fix.aws redshift describe-events --source-identifier <cluster-id> --source-type cluster --duration <minutes, integer, no quotes> shows reboots, maintenance/patch windows, node replacement and auto-recovery. This is the direct answer to “why did it restart / why did it enter modifying by itself” (auto-recovery after a node fault and scheduled maintenance windows both do this legitimately) and it distinguishes an AWS-initiated action from anything the customer did.PercentageDiskSpaceUsed in CloudWatch (it still reports while the cluster is down, unlike the SQL views) plus stv_node_storage_capacity once reachable. Checking 1–5 does not replace this: if the fill really is near 100%, run the full S12 playbook, including adding capacity for headroom as an immediate action alongside the support case. What’s true across all causes: a wedged/unavailable production cluster needs the Redshift service team to recover it — the customer cannot force a stuck modify/restart. Escalate in parallel with checking 1–5, don’t wait passively for it to clear on its own.VACUUM DELETE ONLY <schema.table, identifier, no quotes> — reclaims the space held by ghost rows; high ghost row count, no sort. A DELETE (or the delete half of an UPDATE) removes nothing physically: it marks the rows deleted, and they keep occupying their blocks until VACUUM rewrites those blocks without them. Keep the two nouns straight — they are different mechanisms with different fixes. Rows are ghost/deleted, and VACUUM DELETE ONLY is what clears them. Blocks are tombstoned — superseded block versions pending release, which arise because Redshift blocks are immutable (a rewrite supersedes the old version) and from uncommitted, in-flight or rolled-back writes; those are freed at COMMIT, not by VACUUM, and a long-running open transaction is what stops them clearing. So a table can carry tombstoned blocks with no DELETE workload at all, and VACUUM is the wrong lever for that case.VACUUM RECLUSTER <schema.table, identifier, no quotes> [BOOST] — sorts only the unsorted portion of the table, leaving the already-sorted region intact; it does not merge into the sorted region and does not reclaim all deleted space. Prefer this over SORT ONLY/FULL for large, continuously-ingested tables — AWS recommends it specifically for large tables with frequent ingestion where queries mostly touch recent data, which is exactly where a full VACUUM times out. Not supported with TO threshold PERCENT, on interleaved-sort-key tables, or on DISTSTYLE ALL tables — fall back to SORT ONLY/FULL or a deep copy for those.VACUUM SORT ONLY <schema.table, identifier, no quotes> — a full, global re-sort with no space reclaim; more expensive than RECLUSTER because it re-sorts the whole table, not just the unsorted portion. Reserve it for smaller tables, or for the table types RECLUSTER doesn’t support.VACUUM <schema.table, identifier, no quotes> (full) / VACUUM REINDEX — full sort + reclaim, or interleaved sort keys.unsorted with vacuum_sort_benefit ≈ 0 burns I/O for zero gain — do nothing.VACUUM BOOST is faster but competes with the workload — off-peak only. Confirm runs in sys_vacuum_history and read its status column — it distinguishes completed runs from Failed and Skipped ones, so filter on status rather than assuming every row is a success (stl_vacuum for legacy per-run detail on provisioned single-AZ).SELECT now(), SELECT 1), the cause is NOT table hygiene — that points to leader/planning/transaction overhead instead, often Serializable-isolation memory overhead under heavy concurrent transactions (e.g. a DMS full-load + CDC migration); the fix is ALTER DATABASE <db, identifier, no quotes> ISOLATION LEVEL SNAPSHOT, not ANALYZE or a deep copy. (2) If VACUUM keeps running but reclaims ~0 rows despite real ghost-row/unsorted evidence, suspect a stale Automated Materialized View pinning the auto-vacuum cutoff (its refresh keeps aborting so it’s never marked stale, which blocks vacuum progress indefinitely) — a service-side defect to escalate, not something ANALYZE/deep copy fixes. On a single-node cluster, some VACUUM CPU is expected (leader and compute share the same hardware) — only escalate to table-hygiene remediation if top queries actually show a performance impact, not from the CPU number alone.wlm_query_slot_count is a manual-WLM lever only — it has no effect under Auto WLM (the default) and does not exist on Serverless. Under manual WLM, raise it for that session so the vacuum gets several slots’ worth of memory. Under Auto WLM or Serverless, the equivalent levers are query priority, reducing competing concurrency, and running off-peak so more memory is granted — not a slot count, and not rewriting the table.statement_timeout, a dropped client/session, or the automatic vacuum yielding to workload. Confirm the aborted run in sys_vacuum_history by its status (Failed), then corroborate the cause from the client/driver error, stl_error, the QMR/statement_timeout configuration, and stl_vacuum on provisioned single-AZ — the history tells you it failed, not why. Treat it as an environment/config cause before escalating it as a service defect.sys_vacuum_history records the skip itself — rows with status = 'Skipped' tell you auto-vacuum considered the table and passed on it — but not the reason (below threshold, deferred under WLM occupancy, yielded to workload). If a table needs deterministic maintenance, schedule an explicit VACUUM rather than relying on autonomics.ALTER TABLE ... ALTER SORTKEY on an AUTO-managed table can appear to succeed without taking effect — while the table is SORTKEY AUTO/DISTSTYLE AUTO the Automatic Table Optimizer owns that property and may re-evaluate or revert it. Setting an explicit sort key is what removes it from AUTO management; verify it actually applied via svv_table_info (sortkey1) and svv_alter_table_recommendations rather than trusting the statement’s success.TRUNCATE or ALTER TABLE APPEND, stv_tbl_perm.rows/svv_table_info.tbl_rows can still show stale values, so confirm with SELECT COUNT(*) before concluding data was lost or retained.size_gb × (ghost_rows / tbl_rows) from the anchor query (which aliases size/1024.0 AS size_gb) is the space a VACUUM DELETE ONLY would recover; rank tables by it and sum the top ones. It is an estimate — ghost_rows is an estimated row count and the ratio assumes even row size — so cross-check it against real reclaim: SYS_VACUUM_HISTORY reports what past runs actually freed, and stl_vacuum shows blocks before/after on provisioned single-AZ. Then the decision that matters: compare the estimate against roughly one day of existing auto-vacuum reclaim (SYS_VACUUM_HISTORY over the last 24h, summed). If auto-vacuum is already returning that much or more, a manual pass buys little and the customer’s “is vacuum missing?” is answered no — the space is not vacuum debt. Only when the estimate clearly exceeds the daily auto-vacuum return is a targeted manual pass worth the I/O.VACUUM ... RECLUSTER on a very large table, check whether the cluster itself is undersized for the operation (rough check: table size vs. node count × per-node RAM/cache). A hugely undersized cluster forces constant S3-backed re-reads during the rebuild, so even a deep copy can fail to finish within a session/connection timeout. If one already failed to complete, don’t just retry it as-is: elastic-resize UP temporarily for headroom and cache, run the rebuild from a persistent connection (not a query-editor session, which times out sooner), then resize back down once done.We generally do not recommend a deep copy for ongoing table maintenance. Even though it can be faster than VACUUM when the sort fits in memory, it does not build or maintain the table’s sort partitions, so a deep-copied table degrades again over time and needs re-copying rather than self-maintaining. A deep copy is also all-or-nothing, which is the opposite of VACUUM: it does NOT persist intermediate state, so if it fails partway — session/connection timeout, disk full, a dropped client — every hour of work is lost and a retry restarts from zero. VACUUM persists each completed increment and resumes (see the bullet above), so on a very large table that durability asymmetry usually outweighs the raw speed comparison — say so when a customer is choosing between them, and treat it as a further reason to run any deep copy from a persistent connection with headroom already in place. Prefer VACUUM ... RECLUSTER for large, actively-growing tables (see above). Reach for a deep copy only when RECLUSTER isn’t supported (interleaved sort keys, DISTSTYLE ALL) or the table genuinely needs a one-off full rebuild (e.g. the sort/dist key changed, or the table has never been maintained and is now fully unsorted). For append-mostly or time-partitioned data, propose a time-series table design instead of either option — date-partitioned tables behind a UNION ALL view let an old partition be DROPped to reclaim space immediately, with no VACUUM and no rebuild. That removes the maintenance problem rather than rescheduling it, and it is the durable answer for the ingest-heavy tables that make VACUUM time out in the first place (see S17).
Rebuild a table clean in one pass (fully re-sorted, ghost rows gone):
-- FIRST: capture the grants on the original — CREATE TABLE (LIKE ...) copies the column
-- definitions, distribution and sort keys, and NOT NULL, but NOT the privileges or the owner.
SHOW GRANTS ON TABLE <schema.orig, identifier, no quotes>; -- object form is ON, not FOR
-- (FOR takes a user/role); or query
-- svv_relation_privileges
CREATE TABLE <schema.new, identifier, no quotes> (LIKE <schema.orig, identifier, no quotes>);
INSERT INTO <schema.new, identifier, no quotes> SELECT * FROM <schema.orig, identifier, no quotes>;
-- validate row counts, swap names, RE-APPLY the grants + ALTER TABLE OWNER TO the original owner,
-- then DROP the old table (confirm first)Dropping the original silently drops its privileges with it, so re-granting is a required step, not a nicety — a deep copy that skips it looks successful and then breaks every downstream consumer. A deep copy needs free space — at ~99% full use UNLOAD → DROP → COPY instead (UNLOAD to an encrypted bucket — add ENCRYPTED KMS_KEY_ID '<arn>' per SKILL.md’s security defaults; the transfer itself is TLS). Temp/spill: CREATE TEMP TABLE frees on disconnect; DROP staging tables at end of ETL; a runaway spilling query is a WLM/broadcast-join fix, not a VACUUM.
VACUUM itself needs free space to work, so the correct sequence is not the same at 92% and at 100%. Decide the regime first, then follow that row exactly:
| Regime | Correct order |
|---|---|
| ~90-95%, VACUUM still completes | Free space first: cancel top spillers → TRUNCATE/DROP temp/staging tables → VACUUM DELETE ONLY on high-ghost tables → resize only if still tight. |
| ~99-100%, or VACUUM fails / never finishes / undo is stuck | Add capacity FIRST — elastic resize / add nodes to create headroom — and only THEN VACUUM DELETE ONLY, ordered smallest-reclaimable-first, retrying the largest table last. At zero headroom VACUUM cannot reclaim, so running it first is the wrong move. |
Keep ~25% headroom steady-state; alarm at 70% and 85%. When sizing a resize (up or down), also check local NVMe cache overcommit (fewer/smaller nodes push more of the working set out to S3-backed re-fetch) and whether existing queries already spill heavily from memory pressure — persistent spill at the current size is a signal that downsizing will make it worse, and a genuine capacity/memory shortfall (not just skew) may need more nodes or a larger node type rather than a query-level fix.
Never recommend an elastic resize without checking eligibility first. Elastic resize is not offered for every source→target combination (node type, node count, and available storage all constrain it), so confirm the target shape is actually valid before proposing it:
aws redshift describe-node-configuration-options --action-type resize-cluster \
--cluster-identifier <cluster-id> --region <region>Check the returned NodeType/NumberOfNodes/EstimatedDiskUtilizationPercent options and pick a target with real headroom.
If the target isn’t offered, don’t jump straight to classic resize — rule out three more-specific causes first:
Chain multiple elastic-resize steps. The governing constraint is the cluster’s data-slice count, which is fixed when the cluster is created (or at its last classic resize) — elastic resize redistributes those slices across nodes, it never changes how many exist. The reachable node counts are therefore the ones that fixed slice population can be spread over, so counting slices tells you more about a specific cluster than any published ratio does: SELECT node, COUNT(*) AS slices FROM stv_slices GROUP BY node ORDER BY node; (S9 covers reading the imbalance, including which column actually sizes it). The node-type ratios below are the documented approximation of that constraint — a useful first cut, not authoritative for a given cluster, and not a per-step cap:
| Node type | Legal one-step targets from n nodes |
|---|---|
dc2.large | exactly n×2 or n÷2 — discrete, nothing in between |
dc2.8xlarge, ra3.large | any count from n÷2 to n×2 |
ra3.xlplus | any count from n÷4 to n×2 |
ra3.4xlarge, ra3.16xlarge | any count from n÷4 to n×4 |
Two consequences worth stating to a customer, shown at n=4 but recompute for their n: on ra3.4xlarge the window is 1–16, so 4→2 is legal in one step and no chaining is needed; on dc2.large the only legal down target is 2, and 3 is not reachable at all — a smaller count is not automatically a legal one. describe-node-configuration-options is the authority for any specific case — it covers node types absent from this table (including rg.*), and it reflects the account’s applied quota, which the ratio alone cannot tell you. The advice that survives the correction: a target not offered in one step can still be reachable in two, so chain steps rather than jumping to classic resize.
A same-type node-count change first, then an elastic resize to the new type, when the target isn’t just a smaller/larger count but a different node type/family (e.g. RA3→RG) — going directly from a single-node cluster to a different multi-node type isn’t supported; first modify to 2+ nodes at the SAME type, then elastic-resize to the target type. Check the slice count before committing to that two-step path, because it can dead-end after step one. Single-node ra3.large, ra3.xlplus, rg.large and rg.xlarge each carry two slices, so moving one to 2 nodes leaves a single slice per node with no spare data slices left to redistribute — no further elastic resize is then possible and the remaining hop needs a classic resize. Confirm slices per node up front rather than stranding the customer midway.
A node-count/service-quota limit, not an eligibility gap — if the console dropdown caps at a specific count with no disk/shape explanation (or the account data mentions a node limit/quota), request a service-quota increase; classic resize will not lift an account-level cap. Only use classic resize once these three are ruled out and the shape is genuinely unreachable via any elastic path — say so rather than proposing an elastic resize that will be rejected or roll back (see S18).
A resize that silently does nothing — no error, no node-count change, cluster otherwise healthy — can be a transient internal resize failure, not disk space or eligibility. These are usually a one-off internal contention and frequently succeed on a retry later the same day. Don’t commit to a disk space (S18) or eligibility diagnosis when there’s no error message and no other symptom; suggest retrying before escalating.
aws redshift describe-resize --cluster-identifier <cluster-id> returns the status, elapsed time, and estimated time remaining of an in-progress or most-recent resize; pair it with describe-events (above) for the surrounding workflow.DISTSTYLE EVEN; stage 2 then converts the key-distributed tables back to their KEY distribution in the background. The cluster returns to service far sooner than “hours to days” suggests, while distribution-sensitive query performance only fully recovers once stage 2 completes — so set expectations on both stages, not just availability. See Accelerate resizing of Amazon Redshift clusters with enhancements to classic resize (opens in a new tab).RestoreTableFromClusterSnapshot restores a single table from a snapshot, but always into a new table name (it cannot overwrite an existing table), so restore then swap. Not every table is eligible, so confirm before promising a customer their table is recoverable this way: the table has to be present in that specific snapshot, the operation does not carry the original table’s privileges, and there are documented restrictions on supported table shapes and source/target cluster combinations. Verify against the current API documentation rather than assuming any table can be pulled back individually.describe-node-configuration-options (the console picker reflects the applied quota).dc2.large fleet usually consolidates into fewer, larger RA3 nodes — but dc2.8xlarge is already a large node, so the same migration can need the same or more nodes depending on the target RA3/RG shape. Validate the actual mapping with describe-node-configuration-options instead of predicting it.--availability-zone-relocation, RA3): it moves the cluster to another Availability Zone without changing node shape, and can itself be blocked by capacity in the target AZ. Don’t conflate “move the cluster” with “resize the cluster.”modifying right after that change is usually just the configuration applying — confirm via describe-events before treating it as a stuck workflow.Each heading is a signal signature, not a fixed case — match on the pattern, not on specific percentages, sizes or node counts, and recompute any ratio against the cluster in front of you. Thresholds that are load-bearing decisions (the ~90-95% vs ~99-100% disk-full regimes above, the ~25% headroom target) are called out as such; everything else is illustrative.
unsorted but only a tiny fraction of rows change per day; VACUUM (±BOOST) times out; “extra compute” doesn’t help: svv_table_info.unsorted alone is misleading — vacuum_sort_benefit drives the decision. 100% unsorted can be harmless because automatic table sort is query-driven (sorts the ranges queries scan by sort key), not percentage-targeted. For tables that truly need it (high benefit) this is the textbook VACUUM ... RECLUSTER case — large, continuously-loaded tables where a full VACUUM times out; RECLUSTER sorts only the unsorted portion instead of re-sorting the whole table. Expect svv_table_info‘s sorted/unsorted figures NOT to move after a RECLUSTER — that is normal, not a failed vacuum, and it is usually why a customer reports “sortedness didn’t change”. Those columns describe the sorted region, which RECLUSTER and automatic table sort do not rewrite; they sort rows within the unsorted region. Track vacuum_sort_benefit instead (it measures the rows in unsorted partitions, so it is the figure that responds), and expect query performance to improve even while unsorted stays put. Only SORT ONLY / FULL actually raise the sorted row count in svv_table_info — a global re-sort, far more expensive, and the reason “get this table to 100% sorted” is the wrong goal for a large continuously-ingested table: answer the question the customer means (are scans efficient) rather than the metric they quoted. Fall back to a deep copy only if the table uses interleaved sort keys or DISTSTYLE ALL (RECLUSTER doesn’t support those). Extra compute for automatic optimization (opens in a new tab) only adds resources; it does not change which tables auto-sort selects. It is useful when autonomics (auto-vacuum, auto-sort, auto-analyze) keep getting deferred because the cluster is busy — without it Redshift suspends them under high load until resources free up — so enable it when SYS_VACUUM_HISTORY shows runs being skipped or never scheduled on a continuously loaded cluster, and note the extra compute is billable.stv_slices/svv_table_info. Reason at the slice level: skew 1.0 rules out distkey skew. An elastic resize to a node count that does not divide the existing slice count evenly round-robins the fixed slices unevenly (slices per node / slice-to-node imbalance) — e.g. 8 slices over 5 nodes land as 2/2/2/1/1 — so the nodes holding an extra slice carry roughly twice the data of the others on identical per-node capacity, which is why their fill runs at about double. Gauge severity by counting slices per node. Start with the documented columns (node, slice) — an uneven count is already the defect, and this is the query to show a customer: SELECT node, COUNT(*) AS slices FROM stv_slices GROUP BY node ORDER BY node;. To size the imbalance and judge whether any elastic-resize range is left, count the data slices specifically, because only those hold data and only their spare capacity makes a further elastic resize possible: SELECT node, COUNT(*) AS data_slices FROM stv_slices WHERE type = 'D' GROUP BY node ORDER BY node;. Show the plain per-node count first — it is the simpler figure to explain — and use the type-D count to size the imbalance and to judge how much elastic-resize range remains. Fix: classic resize — slice imbalance can only be corrected by a classic resize, because it is the only operation that rebuilds the slice layout from scratch. A further elastic resize (even to a node count that does divide the slice count) redistributes the existing slices and so does not reliably fix it, and a deep copy / VACUUM REINDEX / distkey or diststyle change cannot fix it either — those move rows between slices, not slices between nodes.stv_node_storage_capacity and the rest of the STL_/STV_ family) are unreachable, so read PercentageDiskSpaceUsed from CloudWatch — it keeps reporting when the cluster is down, which makes it the confirming metric for this scenario; use stv_node_storage_capacity once the cluster is back. Cause: at 100% there is no free space for internal/restart ops (the stuck restart is the symptom). Move: add capacity via elastic resize for headroom AND engage AWS Support / the Redshift service team to recover the stuck workflow (escalate / open a support case — the customer can’t force it). Both are immediate and run in parallel — request the capacity add for headroom at the same time as opening the case; do not demote the resize to a “later, once it’s available” follow-up. Only once the cluster is available can VACUUM/DROP/UNLOAD reduce data. Don't wait, don't VACUUM first.svv_table_info (skew_rows, diststyle) / stv_node_storage_capacity / stv_blocklist: skew_rows ≫ 1 means one key value concentrates rows on a single slice/node — distinct from slice-to-node resize skew (skew ≈ 1 → classic resize). It is a distribution problem, not a VACUUM one. Fix space-safely (near-full, avoid in-place deep copy): change the distribution key to EVEN or a high-cardinality key via ALTER DISTSTYLE, a classic resize, or UNLOAD → DROP → COPY. A deep copy only if free space exists.svv_table_info (stats_off, unsorted) / sys_vacuum_history / svl_query_summary. You can’t disable auto-vacuum per table; its CPU is a symptom. Real drivers: stale stats (stats_off), 100% unsorted, oversized VARCHAR (max_varchar / VARCHAR(65535)) with weak encoding. Fix: ANALYZE, a deep copy to re-sort and right-size the VARCHARs (RAW encoding on the sort key), and schedule heavy VACUUM + ETL off-peak.svv_table_info/svv_alter_table_recommendations, plus stl_vacuum (provisioned single-AZ) for the attempts themselves — the multi-day failed runs appear in sys_vacuum_history with status = 'Failed', so filter on status to see the attempts rather than only the successes. The patch is a red herring — the Automatic Table Optimizer (ATO) changed the sort key, flipping the table to 100% unsorted, so VACUUM now runs a full sort that can’t finish (the ~72-hr timeout is the symptom). Recover: VACUUM DELETE ONLY now, then a deep copy (INSERT...SELECT) or VACUUM BOOST; adopt a time-series rotation design. Prevent recurrence: pin the sort key with ALTER TABLE ... ALTER SORTKEY (or SORTKEY NONE). Don’t escalate a patch bug.stv_node_storage_capacity / PercentageDiskSpaceUsed. The cause is that the target configuration did not have enough disk space to hold the data, so the resize rolled back (the stuck rollback is only the symptom, and the customer cannot force it). Redshift has two unrelated resize failures that read alike, and the value of the answer is saying which one this is: a disk space shortfall on the target (this scenario, fixed by freeing data or choosing a bigger target) versus an EC2 capacity shortfall in the target AZ (“insufficient capacity / please retry later”, above, fixed by retrying or changing AZ). Name the right one, or the customer chases the wrong fix. Recover: engage AWS Support / the Redshift service team to unstick it (escalate / support case). Prevent: free space (drop / UNLOAD, reduce data) or right-size the target before resizing — validate the target with describe-node-configuration-options above; don’t retry the same resize.Don’t reach for STV_PARTITIONS for capacity questions — it’s legacy, local-disk-partition metrics from before managed storage existed, and isn’t the billed footprint on RA3/Serverless. Use STV_NODE_STORAGE_CAPACITY for addressable/used capacity (on RA3/RG this equals the managed-storage quota); size logical data with SVV_TABLE_INFO.size; for a per-node perm/temp/datashared split start with SVV_TABLES_SPACE_USAGE (capacity, local_perm_tables_space_used, local_temp_tables_space_used, datasharing_tables_space_used — superuser; the single fastest answer to “whose blocks are these”), dropping to STV_TBL_PERM/STV_BLOCKLIST when you need it attributed to individual tables; and treat PercentageDiskSpaceUsed as a utilization signal that spill also inflates. Managed storage includes snapshot delta + datashare cache, so storage larger than the source (e.g. Zero-ETL from Aurora) is expected (columnar encoding, replication, per-slice preallocation) — see redshift-storage-utilization.md S16 for the snapshot-billing matrix.
Before telling a customer “space only recovers on restart, that’s expected cache behavior,” confirm a restart actually occurred. An AWS maintenance/patch workflow can roll back without ever restarting the database — check the cluster’s version/event history for the patch outcome and for an actual restart event around that time, rather than assuming every maintenance-window entry means a real restart happened. If the workflow rolled back, there was no restart to reclaim space and no completion event to fire a notification — that’s a maintenance-workflow issue to escalate, not routine cache behavior to tune away.
Snapshot billing is platform-conditional — do not state one blanket rule. Automated snapshots are incremental and free up to 100% of provisioned cluster storage. Manual snapshots (including every snapshot AWS Backup takes) are always billed, and only on Serverless / RG are they metered on the unique data blocks across all active manual snapshots; on RA3 provisioned they are not deduplicated that way, so each retained manual snapshot bills closer to its own full size.
WARN then confirm before VACUUM on large tables, elastic/classic resize, and any DROP/TRUNCATE.