File Systems

  1. /home
    • XFS volume served over Infiniband
    • on backup
    • user quota 15GB
    • persistent file lifetime
  2. /data
    • Lustre volume served over Infiniband
    • no backup
    • initial user quota 1TB (can be extended if additional resources are available)
    • file lifetime of 3 years based on access time (subject to change)
  3. /scratch
    • Lustre volume served over Infiniband
    • no backup
    • initial user quota 2TB (can be extended if additional resources are available)
    • file lifetime of 1 year based on access time (subject to change)

Use check_quota to check your disk quotas routinely

Important

All of the above file-systems are remote for logged in users, hence make sure to avoid causing unnecessary stress for those. Please consider the following notes:

  1. Avoid having more then a few thousand members (files and/or sub-directories) in any directory. Especially for Lustre, directories with large number of members put a large stress on metadata server during access and can lead to data corruption or access timeouts.
  2. When archiving files or directories into tar or zip files, it is much faster and safer to write archive file in local /tmp directory (if it has enough space and is going to be moved out of there promptly) or on a different file-system.
  3. When using rsync to move data to/off the cluster, avoid doing multiple rsyncs at the same time. It does not make it any faster, but puts extra stress on the file-systems.

All of the above are healthy guidelines for any file systems, but remote and especially parallel file systems are more sensitive to such operations.

User's File Locations

Below GROUP is primary group name and $USER is your user name.

  1. /home/GROUP/$USER

    • this is your $HOME directory
    • use it for applications, setups, and scripts.
    • do not not use it for storing any data and running applications inside of this directory
    • the directory has disk-space quotas enabled per /home file-system

  2. /scratch/GROUP/$USER

    • this is your working directory pointed to by $SCRATCHDIR environment
    • use it for working and running jobs
    • the directory will be cleaned routinely from files that were not accessed for some time (see above)
    • the directory has disk-space quota enabled per /scratch file-system

  3. /scratch/GROUP/shared

    • this is the directory for sharing temporary files with other group members
    • the directory has disk-space quota enabled per /scratch file-system

  4. /data/GROUP/$USER

    • this is your archive directory pointed to by $ARCHIVEDIR environment
    • use it for storing more permanent input data and final results
    • this directory has both space and file-count quota for longer-term storage of your important data
    • the directory will be cleaned routinely from files that were not accessed for some time (see above)
    • the directory has disk-space and file-count quota enabled per /data file-system

  5. /data/GROUP/shared

    • this is your directory for sharing your data with other group members
    • the directory has disk-space and file-count quota enabled per /data file-system

  6. temporary directories while executing jobs under torque

    • global $GLOBTMPDIR points to /scratch/tmp/$PBS_JOBID
    • local $LOCTMPDIR points to /tmp/$PBS_JOBID
    • $TMPDIR points to $LOCTMPDIR; i.e., local file system on the node

Routine Cleanups

  • local /tmp on both login and compute nodes: 30 days no-access lifetime
  • /scratch/tmp on Lustre: 30 days no-access lifetime
  • /scratch on Lustre: 1 year no-access lifetime
  • /data on Lustre: 2 years no-access lifetime