As described in the previous section, Joule 3.0 offers multiple file systems for managing data. This section will provide an overview of each file system, their intended use, and quota/purge policies.
Joule 3.0 File Systems Overview
| Filesystem | Backups | Purge | Mount | Quota (Bytes) | Quota (Inodes) |
| Global | Daily | Delayed | /nfs/global | 4TB / 5TB | 4M / 5M |
| Home | Daily | No | /nfs/home | 4TB / 5TB | 4M / 5M |
| Long-Term | Daily | No | /nfs/long | 5TB / 6TB | 2M / 3M |
| Public | Daily | No | /nfs/home/public | 2TB / 2TB | 2M / 2M |
| Project | Daily | No | /nfs/project | — | — |
| NFS Scratch | No | 60 Days | /nfs/scr/1, /nfs/scr/2 | 4TB / 5TB | 4M / 5M |
Global Directories
The Global directory, /nfs/global/$username, allows users to manage data between the Joule 2.1 and Joule 3.0 Supercomputers. This filesystem will be mounted on both Joule 2.1 and Joule 3.0 login nodes. It cannot be used to run jobs on either system, only to move data between systems. For the purposes of the initial onboarding of users on Joule 3.0, the implementation of purge policy is being delayed. In the future, this purge policy will be the standard length of 60-days. Users will be notified prior to the implementation of this policy.
The global filesystem is backed up daily. Global user directories have a 4 TB soft quota limit and 5 TB hard quota limit. The default inode quota – maximum number of files – is 4 Million soft and 5 Million hard.
Home Directories
Your Home directory, /nfs/home/login#, is the default login directory and contains your session preference files and general user files. If you are on login02, your Home directory is mounted as /nfs/home/2/$username. Home directories can be accessed from any login or compute system on Joule 3.0.
You can use your Home directory to stage or compile code, store input files or other configuration files. You can also launch jobs from your Home directory, however, it is limited in bytes and inodes.
Home directories are backed up daily. User directories have a 4 TB soft quota limit and 5 TB hard quota limit. The default inode quota is 4 Million soft and 5 Million hard.
Long-Term Storage
Long-Term storage is mounted as /nfs/long/login# on your login node. If you are on login02, your long-term directory is mounted as /nfs/long/2/$username. The long-term storage is only mounted on login systems. It is not mounted on compute as it is not intended for active jobs.
The long-term storage utilizes a redundant 10GbE link to all login systems to make migration of data as fast as possible. The Long-term filesystem is also housed in a separate, physically segregated, data center with backup power generation to make storage of your data as safe as possible
The long-term file system is backed up daily. This file system has a 5 TB soft quota limit and 6 TB quota limit. The default inode quota is a 2 Million soft limit and 3 Million hard limit. Quota limits can be increased upon special request.
Public Directory
The public directory, /nfs/home/public, is a safe space to share data with other users of the system. We ask that you be mindful of what you share and who you share it with.
You can create your own directory in this space and use the Linux chown and chmod commands to share with other users.
The public directory is backed up daily. The default byte quota limit is 2TB soft and 2TB hard. The default inode quota is 2 Million soft and 2 Million hard.
Project Directories
Project directories are specialized directories that are only available upon request. These can contain project-sensitive data and access is restricted using access control lists.
NFS Scratch
There are two dedicated NFS Scratch filesystems, /nfs/scr/1 and /nfs/scr/2, that are designed for I/O intensive programs such as OpenFOAM. Each NFS Scratch directory has a default quota of 5TB of disk space and 5 million inodes. Combined, this results in a total of 10TB of disk space and 10 Million inodes per user. NFS Scratch can be accessed from any login or compute system on Joule 3.0.
Users must complete the NFS Scratch Allocation Form before being provided access to the NFS Scratch filesystem.
Scratch Policy and Guidelines
- NFS Scratch is scratch space and it is NOT BACKED UP. A failure of only one component of the scratch filesystems could result in data loss.
- NFS Scratch is subject to a 60 DAY PURGE POLICY. Data older than 60 DAYS WILL BE DELETED automatically from the NFS Scratch filesystems.
- Data results you wish to keep should be post-processed and/or moved to the long-term filesystem, /nfs/long, which is a redundant, backed-up filesystem.
- The long-term filesystem utilizes a redundant high-speed link to all login systems to make migration of data as fast as possible. Long-term is also housed in a separate, physically segregated, data center with backup power generation to make storage of your data as safe as possible.
If your scratch data consists of millions of files and directories, we ask that you compress this data using tar when moving it to long-term storage. This reduces your overall file count in the long-term filesystem and also allows for optimal performance of our backup processes. Users can tar directly from NFS Scratch to long-term on their login node, tar without compression is recommended, e.g. tar -cf /nfs/long/1/username/data.tar /nfs/scr/1/username/scratch_data, replacing “1” and “username” with your login node number and username.