Table of Contents
High Performance Computing (HPC) can appear daunting at first. Understanding a few basic fundamentals can go a long ways to empowering you to perform complex computational analysis in order to propel your research forward. The biggest challenge in learning HPC is knowing where to get started.
Most HPC environments require a fairly basic knowledge of Linux/Unix command line and some scripting knowledge. Most HPC environments use Linux for the operating system. Starting out, we will assume that you have some knowledge of command line arguments. A shell scripting language, such as BASH/CSH/KSH is also beneficial and can augment the capabilities of batch submission scripts you create.
If you or your students are using Aeolus cluster (all users with an account or access), you and/or they should be subscribed to the Aeolus mailing list.
This list is for ALL announcements and is required to be signed up to. The Aeolus mailing list distributes emails that cover important developments concerning the cluster resources and operations; you need to know about these things.
Aeolus has its own support website for ALL Aeolus issues and requests:
- If you want to setup an account on Aeolus
- If you have a question or need to report an issue
- If you need help installing a software package
To create an account, follow the guide found here.
Once you have been confirmed an account on the system, you will have access to the cluster and can work through the documentation for examples of using the system and submitting jobs.
Linux provides many terminal environments. Additionally, SSH is a de-facto standard for interfacing remotely with Linux systems and as a result, it is usually installed already. To login, use the following procedure:
Mac has a Terminal application provided by OSX. No additional software is required to connect. To login, use the following procedure:
Windows users will require additional tools to be installed in order to access Aeolus via SSH. Common tools include Putty, which is a windows based SSH program.
For those of you using Putty, you don’t have to live in the misery that is the basic configuration of colors. As a personal favourite, we would recommend the monokai theme, which can be found on GitHub.
Passwords provide one level of security to authenticating into the system. If you need to automate tasks and/or want a more secure and easier way of accessing the system, you should use a SSH key.
For now, please consult this tutorial on how to do just that. Consider the following notes:
- Aeolus is a legacy system that doesn't support the Ed25519 algorithm, you must use RSA algorithm as described in the tutorial.
- Do not leave the file name “id_rsa”, that is too generic. Name it something meaningful, like "rsa4096l_amb_wsu"
- rsa4096 = encryption type
- l = legacy systems
- amb = initials
- wsu = systems that will be connecting
- Definitely provide a passphrase to the key! If you do not, you will still be held responsible for all actions taken on your account, even if your key gets compromised.
- The passphrase has nothing to do with your password on the system you are connecting to. It is used to protect the encrypted key on your system.
- Add it to your ssh-agent, that way you don’t have to unlock the key every time you want to log in (just per your local sessions).
Useful Linux Commands
There are a plethora of commands out there for Linux, which is both a positive and a negative. We want to get you pointed in the right direction with some basic command line interface (CLI) commands:
NOTE: for all of the example commands that we will cover here, consider the following variable nomenclature:
ANY and ALL questions about commands should first refer you to the Linux Manual (man pages), which gives us our first command:
For information about storage options, visit this guide.
Disk Usage (du)
du [/path/to/dir]total disk useage of directory
du -k [/path/to/dir]total disk useage of directory in k bytes
du -sc --si [/path/to/dir]summarize, add up total, in powers of 1024
du -sc --si [/path/to/dir]/* | sort -hsame as above, and sort it by human readable numbers
Report Filesystem Usage (df)
df -h [/path/to/filesystem]print as human readable
Pattern Matching (grep)
grep -i [pattern] [path]search for pattern
grep -v [pattern] [path]ignore pattern
grep -R [pattern] [path]recursively search for pattern
For a more thorough overview on how to utilize rsync, please review the specific Knowledge Base Article
Cron & Crontab
Cron is a Unix utility that allows tasks to be automatically run in the background at regular intervals by the ssh-server cron daemon.
Backup your cron file:
crontab -e and
crontab -r are grossly different commands, yet e & r appear side by side on a standard QWERTY keyboard… thus, learning to back it up first!
crontab -eedit or create your crontab file using your default editor
crontab -ldisplay your crontab file
crontab -vdisplay the last time you edited your crontab file
crontab -rremove your crontab file
For an example base cron file & help:
30 18 * * * rm ~/tmp/*remove all files (that you have access to) from ~/tmp/, each day at 6:30 pm
0 9,18 * * Mon ~/scripts/example.shrun ~/scripts/example.sh every Monday at 9:00 am and 6:00 pm
Don’t just take our word for it, do some research on your own! Always feel free to come back and ask us questions by submitting a ticket.
Here are some extra tutorials to arm yourself for familiarity with Linux:
- 50 Linux commands, with examples
- Command line references
- Bash guide for beginners
- Advanced bash scripting guide
For more detailed information and examples of submitting jobs on Aeolus, please see the Job Submission documentation.