Avoid listing extended attributes of files and directories when possible. For example, use “ls” or “ls -l file_name” instead of “ls -la”. Listing extended attributes, such as permissions and ownership, requires coordination between the Meta-data Servers and Object Storage Servers components of Lustre which can be resource intensive especially for large directories.
Avoid creating a large number of files in a single directory. Creating a single directory that contains multiple thousands of files can cause resource contention when attempting to read or list these files and will degrade performance of the filesystem. Instead organize your data into sub-directories of £ 1000 files where possible.
Use proper striping when creating a large number of small files. When dealing with a large number of small files, i.e. files smaller than 16MB, set your directory stripe count 1 using the “lfs setstripe” command. This will optimize allocation of your data onto the Object Storage Servers. A stripe count of 1 is currently the default stripe size. Files created under this directory will inherit this stripe value.
lfs setstripe -c 1 directory_name
Use proper striping when creating large files. When dealing with large files, i.e. files approximately 1GB or larger, set your directory stripe count to 2 or 4 using the “lfs setstripe” command. This will optimize allocation of your data and improve performance by utilizing multiple Object Storage Servers. A stripe count of 1 is currently the default stripe size. Files created under this directory will inherit this stripe value.
lfs setstripe -c 2 directory_name
Monitor your usage and quota with the myquota command. You can use the “myquota” command from your login system to view your current usage and quota on the Lustre scratch filesystem.
Limit usage of any Linux utility that scans your directory, such as GNU find or locate. These utilities can cause resource contention and will degrade performance for yourself and the other users of the filesystem.
