Setting the file. One moment.
Chapter 61 · Amazon Elasticache
Subchapter 61.34
references/monitoring/slot-memory-imbalance-detection.mdMarkdown7 KBView on GitHub
When to use: Uneven memory across cluster nodes (one node near DatabaseMemoryUsagePercentage ceiling while siblings are well below), suspected hash-tag bundling, or planning resharding and want to identify which slots carry the most memory. Valkey 8.0+ cluster mode enabled only. When not needed: Cluster-wide uniform memory pressure (troubleshooting.md Memory Pressure). Per-node CPU imbalance without memory skew (hot-key-detection.md). Serverless (abstracts slot topology).
| Deployment | Applicability |
|---|---|
| Serverless | Does not apply. Slot topology is abstracted. |
| Node-based, cluster mode disabled | Does not apply (single-shard). Use big-key-hunter.md + key-space-distribution-by-prefix.md. |
| Node-based, cluster mode enabled | Full playbook. |
CLUSTER SLOT-STATS with MEMORY-BYTES requires Valkey 8.0+ (this is a Valkey-only feature and is not available on any Redis OSS engine version). Verify:
valkey-cli -h <endpoint> -p 6379 --tls INFO server | grep -E 'redis_version|valkey_version'Pre-Valkey 8.0 degraded workflow: 1) Per-shard CloudWatch triage (Tier A). 2) valkey-cli --bigkeys against the hot shard’s primary. 3) CLUSTER KEYSLOT <key> per big-key candidate to find the heavy slot. 4) MEMORY USAGE <key> on top candidates for byte confirmation. Takes 15-30 minutes on a 10M-key shard vs. subsecond with SLOT-STATS.
cluster-slot-stats-enabled must be yes in the parameter group. This is a Valkey 8.0+ parameter; verify it is available in your cache parameter group before relying on it. Note: This parameter may not appear in ElastiCache parameter groups documentation. It may be enabled by default or not exposed as a configurable parameter. CONFIG GET is restricted on all ElastiCache caches; check via the AWS API:
aws elasticache describe-cache-parameters \
--cache-parameter-group-name <parameter-group-name> \
--query "Parameters[?ParameterName=='cluster-slot-stats-enabled'].ParameterValue" \
--output text --region <region>If missing or no, enable via parameter group modification (zero-downtime on most configs).
| Command | Supported | Notes |
|---|---|---|
| CLUSTER SLOT-STATS ORDERBY MEMORY-BYTES | Valkey 8.0+ cluster mode, NB only. Requires cluster-slot-stats-enabled parameter set to yes. Availability on ElastiCache depends on engine version and parameter group support. | Primary Tier B tool |
| CLUSTER SHARDS | NB (Valkey 7.2+ / Redis OSS 7.0+), SL (virtual shard) | Topology discovery |
| CLUSTER COUNTKEYSINSLOT <slot> | NB all versions | Per-slot key count |
| CLUSTER GETKEYSINSLOT <slot> <count> | NB all versions | Enumerate keys in a slot |
| CLUSTER KEYSLOT <key> | NB all versions | Hash key to slot number |
| MEMORY USAGE <key> | NB only, blocked on SL | Tier C byte confirmation |
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name DatabaseMemoryUsagePercentage \
--dimensions Name=CacheClusterId,Value=<node-id> Name=CacheNodeId,Value=0001 \
--start-time <24h-ago> --end-time <now> \
--period 300 --statistics Maximum --region <region>Run for each primary (discover with CLUSTER SHARDS).
Stop condition: If all primaries are within 10 percentage points, memory is distributed uniformly. No slot-imbalance problem. Route to troubleshooting.md Memory Pressure if overall memory is high.
Proceed if spread is >15 percentage points (e.g., one node at 85%, another at 45%).
Query BytesUsedForCache per node over 24h (node-based, using CacheClusterId and CacheNodeId dimensions). For serverless, use BytesUsedForCache with the ServerlessCacheName dimension. A stable gap over hours is real. A 10-minute transient is not.
valkey-cli -h <endpoint> -p 6379 --tls \
CLUSTER SLOT-STATS ORDERBY MEMORY-BYTES DESC LIMIT 20A single dominant slot (order of magnitude above peers) is a memory hotspot.
Subsecond, negligible impact. Reads stats Valkey already maintains. Safe at any time on production.
Use CLUSTER SHARDS output to map each hot slot to its primary. Hot slot on the already-hot node confirms the story.
valkey-cli -h <endpoint> -p 6379 --tls CLUSTER COUNTKEYSINSLOT <hot-slot>| Pattern | Cause | Next step |
|---|---|---|
| Few keys, large memory | Big-key(s) in this slot | big-key-hunter.md |
| Many keys, large memory | Hash-tag concentration | Continue to Tier C |
Stop condition: If top 20 slots are within 2x of each other, no single hot slot. Consider rebalancing slot assignments rather than application-level remediation.
valkey-cli -h <endpoint> -p 6379 --tls \
CLUSTER GETKEYSINSLOT <hot-slot> 200Look for common {...} hash-tag prefix in returned keys. If most keys are {tenant_42}:session:*, {tenant_42}:cache:*, tenant 42 has concentrated its keys into a single slot.
valkey-cli -h <endpoint> -p 6379 --tls MEMORY USAGE <key>If one key dominates bytes, this is a big-key problem (route to big-key-hunter.md). If memory is evenly distributed, hash-tag concentration is the cause.
Pick based on Tier C classification:
| Cause | Remediation |
|---|---|
| Hash-tag concentration (default) | Redesign hash-tag: {tenant_id}:{shard_id} where shard_id is a random 4-bit prefix. Spreads keys across 16 slots while keeping intra-tenant MULTI/LUA local. Application change required. |
| Oversized tenant | Split to dedicated cache. For tenants using >30% of cluster memory. |
| Big key in hot slot | big-key-hunter.md remediation (decomposition, compression, move to S3). |
| Slot-count skew | Manual slot rebalance via online resharding (modify-replication-group-shard-configuration). |
| Orphan keys | Bulk UNLINK after confirming via key-space-distribution-by-prefix.md TTL audit. |
| Scale up (last resort) | Larger node type. Only masks the symptom. |
hot-key-detection.md Tier B (same SLOT-STATS, ORDERBY CPU-USEC axis)big-key-hunter.mdkey-space-distribution-by-prefix.mdalarm-packs.md