The Lustre Scratch filesystem, /lustre/scratch, provides an initial allocation of 20TB of disk space and 5Million file and directory entries, i.e. inodes. The Lustre scratch filesystem is designed for high-performance temporary storage of data.
- Lustre is scratch space, it is NOT BACKED UP, a failure of only one component of the Lustre filesystem could result in data loss.
- Lustre Scratch is subject to a 60 DAY PURGE POLICY. Data older than 60 DAYS WILL BE DELETED automatically from the Lustre Scratch filesystem.
- Data results you wish to keep should be post-processed and/or moved to the long-term filesystem, /nfs/long, which is a redundant, backed-up filesystem.
- The long-term filesystem utilizes a redundant 10GbE link to all login systems to make migration of data as fast as possible. Long-term is also housed in a separate, physically segregated, data center with backup power generation to make storage of your data as safe as possible.
- If your scratch data consists of millions of files and directories, we ask that you compress this data using tar when moving it to long-term storage. This reduces your overall file count in the long-term filesystem and also allows for optimal performance of our backup process. Users can tar directly from Lustre Scratch to long-term on their login node, tar without compression is recommended, e.g. tar cf /nfs/long/1/username/data.tar /lustre/scratch/username/scratch_data, replacing “1” and “username” with your login node number and username.
Best Practices
- Avoid listing extended attributes of files and directories when possible. For example, use “ls” or “ls -l file_name” instead of “ls -la”. Listing extended attributes, such as permissions and ownership, requires coordination between the Meta-data Servers and Object Storage Servers components of Lustre which can be resource intensive especially for large directories.
- Avoid creating a large number of files in a single directory. Creating a single directory that contains multiple thousands of files can cause resource contention when attempting to read or list these files and will degrade performance of the filesystem. Instead organize your data into sub-directories of £ 1000 files where possible.
- Use proper striping when creating a large number of small files. When dealing with a large number of small files, i.e. files smaller than 16MB, set your directory stripe count 1 using the “lfs setstripe” command. This will optimize allocation of your data onto the Object Storage Servers. A stripe count of 1 is currently the default stripe size. Files created under this directory will inherit this stripe value.
- lfs setstripe -c 1 directory_name
- Use proper striping when creating large files. When dealing with large files, i.e. files approximately 1GB or larger, set your directory stripe count to 2 or 4 using the “lfs setstripe” command. This will optimize allocation of your data and improve performance by utilizing multiple Object Storage Servers. A stripe count of 1 is currently the default stripe size. Files created under this directory will inherit this stripe value.
- lfs setstripe -c 2 directory_name
- Monitor your usage and quota with the myquota command. You can use the “myquota” command from your login system to view your current usage and quota on the Lustre scratch filesystem.
- Limit usage of any Linux utility that scans your directory, such as GNU find or locate. These utilities can cause resource contention and will degrade performance for yourself and the other users of the filesystem.