Setting the file. One moment.
Subchapter 43.4
references/fsx-lustre-knowledge.mdMarkdown6 KBView on GitHub
This reference captures information addressing common gotchas and frequently asked questions for Amazon FSx for Lustre to support accurate model responses. It is not a complete specification. The authoritative source for current specifications, limits, quotas, and API behavior is the Amazon FSx for Lustre User Guide, , , and . Retrieve current specifications, limits, and quotas from those pages before citing specifics; cite figures only from those pages.
What it is: Amazon FSx for Lustre is a fully managed service that provides high-performance, cost-effective, and scalable storage powered by Lustre, the world’s most popular high-performance file system. FSx for Lustre provides the fastest storage performance for GPU instances in the cloud, with up to terabytes per second of throughput, millions of IOPS, sub-millisecond latencies, and virtually unlimited storage capacity.
Well-suited for: compute-intensive workloads where storage must keep pace with large fleets of GPU or CPU compute, such as ML training and inference, HPC, genomics, seismic and financial modeling, media rendering, and back-end EDA, with native Amazon S3 integration that makes datasets in S3 transparently accessible as files.
See When to choose Amazon FSx (opens in a new tab) and Amazon FSx for Lustre Features (opens in a new tab).
You MUST retrieve information from the linked documentation in the below table before answering any user question on the related topic.
| Topic | Example service characteristics, features, and actions | Documentation |
|---|---|---|
| Access and protocols | Lustre parallel client on Linux; cross-AZ client access; Elastic Fabric Adapter for high-throughput, low-latency inter-node networking | Accessing file systems (opens in a new tab) |
| Deployment and availability | Persistent 1 and 2; Scratch 2 | Deployment and storage class options (opens in a new tab) |
| Storage classes | SSD; Intelligent-Tiering | Lustre storage classes (opens in a new tab) |
| Data protection and management | S3 data repository association; auto-import and auto-export; writes from multiple locations; file release to free local space; capacity increase; backups | Using data repositories with Amazon FSx for Lustre (opens in a new tab) |
| Performance | aggregate throughput; sub-millisecond latency; high IOPS; metadata IOPS; NVIDIA GPUDirect Storage (GDS); working-set sizing | Amazon FSx for Lustre performance (opens in a new tab) |
| Security | KMS at rest; in-transit automatic from EC2 instances that support it (Nitro-based) and between file-system hosts; instances that do not support it and on-premises clients do not receive in-transit encryption; POSIX permissions; VPC and security groups | Security in Amazon FSx for Lustre (opens in a new tab) |
| Health and monitoring | CloudWatch metrics; alarms; CloudTrail | Monitoring Amazon FSx for Lustre file systems (opens in a new tab) |
| Pricing | storage capacity; throughput capacity; metadata IOPS; backup storage; cross-AZ transfer; Intelligent-Tiering (not exhaustive, review pricing page for the full list of pricing dimensions) | Amazon FSx for Lustre pricing (opens in a new tab) |
You MUST retrieve information from the linked documentation in the below table before answering any user question on the related service.
| Example service | Example characteristics and common workloads | Documentation |
|---|---|---|
| Amazon EKS (FSx for Lustre CSI driver) | dynamic and static Kubernetes volume provisioning; pod access to shared Lustre | Amazon FSx for Lustre CSI driver (opens in a new tab) |
| Amazon SageMaker | high-throughput training-data input; file-system mount for training jobs | Setting up training jobs to access datasets (opens in a new tab) |
You MUST retrieve information from the linked documentation in the below table before providing the user with any guidance on the related area.
| Example area | Example errors | Example fixes | Documentation |
|---|---|---|---|
| Connectivity and networking | mount fails; Lustre client missing | install the client matching the OS and kernel | Installing the Lustre client (opens in a new tab) |
| Connectivity and networking | mount hangs or times out; security group misconfigured | allow the required Lustre ports between the file-system and client security groups (restrict to the specific client security group IDs); verify VPC routing | File system access control with Amazon VPC (opens in a new tab) |
| Data protection and recovery | data repository misconfigured; slow first read; conflicting writes; backups only on Persistent non-S3-linked | restore IAM access; pre-stage hot files; write from a single source; use Persistent (not S3-linked) for backups | Data repository association lifecycle state (opens in a new tab), Linking to an S3 bucket (opens in a new tab), Protecting your data with backups (opens in a new tab) |
| Capacity and scaling | out of space with a large S3 dataset | release infrequently-accessed files; size to the working set | Releasing files (opens in a new tab), Increasing storage capacity (opens in a new tab) |