Table of Contents
Batch processing is the execution of a series of jobs in a program on a computer without manual intervention (non-interactive). The series of steps in a batch process are often called a “job” or “batch job”. In computing context - a batch queue is a scheduler managed automatic system which decides when and where your jobs get run when submitted into a queue. On Aeolus, TORQUE is coupled with MAUI to handle batch job management.
- TORQUE Resource Manager controls batch jobs and distributed computing resources. It handles the job submissions, starts, stops, and monitors job & node stats. It is based on an open-source PBS resource management system.
- MAUI is an external scheduler for the Torque resource management system.
To recap, Torque manages the job queue and the compute resources, while Maui queries the PBS server to obtain up to date job and node information. Using this information, Maui directs the PBS server to start or stop jobs in accordance with specified Maui policies, priorities, and/or reservations.
General Rules of Using Aeolus
Aside from the User Policy, these more specific rules apply to use of the system. Accounts found in violation of these rules, may be temporarily banned, require a review of the documentation, and have a discussion with the group/lab PI.
- Do not run computation on the ssh/login node.
- Do not compile software on the ssh/login node.
- Do not write computational output to your home directory (especially true for MPI jobs using parallel I/O).
- You should review the Storage Options knowledge base article and the Investment Policy to fully understand storage options.
- You may not over subscribe or under subscribe to requested resources.
- Over Subscription: the resources requested (cpu & ram) is more than the resources required or used.
- Under Subscription: the resources requested (cpu & ram) is less than the resources required.
Software Packages and Modules
Aeolus uses the module
system to provide version consistent distributed applications to compute nodes. To learn the basics, use module --help
. To learn more, use man module
.
The module system can be used to build an environment unique for your application needs. This can be done before submitting a job or from a script submitted as a job.
Basic command structure looks like:
module avail module help [modulefile name] module list module <add | load> [modulefile name] module <rm | unload> [modulefile name] module <switch | swap> [modulefile name from] [modulefile name to] module clear
Example:
# query, load, and list a module module avail module load python/3.6.1 module list # confirm the module is active in your environment which python env | $LD_LIBRARY_PATH
If you know you will always want to have specific modules loaded, you can call them from your ~/.bashrc
file, the same way you would load them interactively.
Torque PBS Job Submission
Torque manages jobs that users submit to various queues available on Aeolus. Each queue represents a group of resources with attributes to help identify the queue. Commonly used Torque commands include:
qsub
- used to submit both batch or interactive jobs to the clusterqstat
- used to monitor the status of a jobqdel
- used to terminate a job prior to its completionshowq
- list the jobs in the queuepbsnodes
- list nodes, properties, and associated resources
Torque includes numerous directives, which are used to specify resource requirements and other attributes for batch and interactive jobs. Torque directives can appear as header lines (lines that start with #PBS
) in a batch job script or as command-line options to the qsub
command.
For help using Torque to submit and manage jobs, see the Submitting and managing jobs chapter of Adaptive Computing’s Toruqe Guide. For a complete Appendix of Commands for Torque, see the Adaptive Computing website. This is also where you can get a complete Descriptive List of Options for QSUB.
Torque Environment Variables
This is not an exhaustive list, but should provide a good base to work with. Beyond this, there is the previous listed Adaptive Computing documentation.
Variable | Description |
---|---|
PBS_ARRAYID | zero-based value of job array index for this job |
PBS_JOBNAME | user specified job name |
PBS_NODEFILE | file containing line-delimited list of nodes allocated to the job |
PBS_NODENUM | node offset number |
PBS_QUEUE | job queue |
PBS_TASKNUM | number of tasks requested |
PBS_O_HOME | home directory of submitting user |
PBS_O_HOST | host on which job script is currently running |
PBS_O_JOBID | unique PBS jobid |
PBS_O_LANG | language variable for job |
PBS_O_LOGNAME | name of submitting user |
PBS_O_PATH | path variable used to locate executables within job script |
PBS_O_SHELL | script shell |
PBS_O_WORKDIR | job’s submission directory |
Batch Job Scripts
To run a job in batch mode on Aeolus, first prepare a job script that specifies the application you want to run and the resources required to run it, then submit the script to Torque using the qsub
command. Torque passes your job and its requirements to the system’s job scheduler, which then dispatches your job whenever the required resources are available.
A very basic job script might contain just a bash or tcsh shell script. However, Torque job scripts most commonly contain at least one executable command preceded by a list of directives that specify resources and other attributes needed to execute the command (e.g. wall-clock time, number of nodes, number of processors, filenames for output and errors). These directives are listed in header lines (lines with #PBS
), which should precede an executable lines in your job script.
Additionally, your Torque job script (which will be executed under your preferred login shell) should begin with a line that specifies the command interpreter under which it should run.
NOTE:
For some job scripts, you may need to manually load the required module filesSerial Job Example
A serial Torque job script for a serial job might look like this:
#!/bin/bash # -------- -------- -------- -------- # PBS qsub - Scheduler Request # * two octothorpes "#" indicates a comment # -------- -------- -------- -------- ## Export all environment variables in the qsub command's environment to the ## batch job. #PBS -V ## Define a job name #PBS -N dev_serial_01 ## Define compute options #PBS -l nodes=1:amd:ppn=1 ##PBS -l nodes=1:intel:ppn=1 #PBS -l mem=1024mb #PBS -l walltime=00:05:00 #PBS -q batch ## Define path for output & error logs #PBS -k o ##PBS -j oe #PBS -e /fastscratch/[username]/dev_serial_01.e #PBS -o /fastscratch/[username]/dev_serial_01.o ## Define path for reporting #PBS -M [username]@wsu.edu #PBS -m abe # -------- -------- -------- -------- # Actually do something # -------- -------- -------- -------- cd $PBS_O_WORKDIR touch dev_serial_01.test
In the above example, the header lines of the Torque directives include/mean the following:
Torque Directive | Description |
---|---|
#PBS -V | Export your environment variables (see: printenv ) in the qsub command’s environment to the batch job. |
#PBS -N dev_serial_01 | Define a name for the job, which will show up in the queue, email, & logs. |
#PBS -l nodes=1:amd:ppn=1 | Indicates the job requires one node, the amd property, and one processor. |
#PBS -l mem=1024mb | Indicates the job requires 1024 MB of RAM. |
#PBS -l walltime=00:05:00 | Indicates the job requires 05 minutes of wall-clock time. |
#PBS -q batch | Submit job the the queue batch . |
#PBS -k o | Keeps the job output. |
#PBS -j oe | Not used in this instance, would combine the standard output and standard error. |
#PBS -e | Defines output location for standard error. |
#PBS -o | Defines output location for standard out. |
#PBS -M [email] | Sends job-related email to specified email, according to rules listed on the next line. |
#PBS -m abe | Sends email if the job is (a) aborted, when it (b) begins, and when it (e) ends. |
Parallel Job Example
The base job settings are the same for parallel jobs as serial jobs. The main difference(s) which set them apart are the number of processors and/or nodes are requested. This example will cover both shared memory and distributed memory mpi jobs.
The total processor count equals: nodes * ppn
pbsnodes
and lscpu
can provide useful information to your specific node CPU properties.#!/bin/bash ## Export all environment variables in the qsub command's environment to the ## batch job. #PBS -V ## Define a job name #PBS -N dev_parallel_01 ## Define compute options #PBS -l nodes=2:amd:ppn=8 ##PBS -l nodes=2:intel:ppn=8 #PBS -l mem=2gb #PBS -l walltime=00:05:00 #PBS -q batch ## Define path for output & error logs #PBS -k o ##PBS -j oe #PBS -e /fastscratch/[username]/dev_parallel_01.e #PBS -o /fastscratch/[username]/dev_parallel_01.o ## Define path for reporting #PBS -M [username]@wsu.edu #PBS -m abe # -------- -------- -------- -------- # Start the script itself # -------- -------- -------- -------- cd $PBS_O_WORKDIR # Run job on all scheduled threads (processors) # NProc=$(wc -l $PBS_NODEFILE | awk '{print $1}') # mpirun -np $NProc -machinefile $PBS_NODEFILE ~/bin/binaryname # ---- OR ---- # Calculate: physical cores = processors / threads per core NProc=$(wc -l $PBS_NODEFILE | awk '{print $1}') NTpC=$(lscpu | grep Thread | awk '{print $4}') NCore=$(bc <<<"$NProc/$NTpC") # Create a list of cores or threads (needed when not using all threads) cat $PBS_NODEFILE | awk -v tpc="$NTpC" 'NR % tpc == 0' > distributed_cores # Run job on physical cores only (not threads, depends on node type) mpirun -np $NCore -machinefile distributed_cores ~/bin/binaryname
In the above example, the mpiexec
line tells the operating system to use the mpiexec command to execute ~/bin/binaryname on either 4 or 8 processes from the machines listed in $PBS_NODEFILE and evenly spread the load across as many nodes as requested.
Gathering Useful information
Beyond getting a serial or parallel job submitted to the system, it is important that you can arm yourself with information to further your development and research.
After all of the Torque PBS server settings have been set, you can log useful information that will appear in our standard output:
echo "--------- environment ---------" env | grep PBS echo "--------- where am i ---------" echo WORKDIR: ${PBS_O_WORKDIR} echo HOMEDIR: ${PBS_O_HOME} echo "--------- what i do ---------" echo Test scheduler via test_script echo Running time on host `hostname` echo Time is `date` echo Directory is `pwd` echo "--------- end of job ---------" echo ""
Submitting Jobs
To submit your job script(s), use the Torque qsub
command. If the command runs successfully, it will return a job ID to standard output, for example:
qsub job.sh > 123456.mgt2-ib.local
If your job requires attribute values greater than the defaults, less than the maximum allowed, and different than those specified in the job #PBS
options, you can specify those inline with the -l
(lowercase L, for “limit”) option. For example, the following command submits job.sh, using the -l walltime
option to indicate the jobs needs a wall time different than that specified in the script.
qsub -l walltime=02:00:00 job.sh
NOTE:
command-line arguments override Torque directives in your job script
To include multiple options on the command line, use either one -l
flag with several comma-separated options, or multiple -l
flags, each separated by a space. For example, the following two commands are equivalent:
qsub -l ncpus=16,mem=1024mb job.sh qsub -l ncpus=16 -l mem=1024mb job.sh
Other useful qsub options include:
qsub option | Description |
---|---|
-q [queue_name] | Specifies user-selected queue |
-r | Set job to be re-runable |
-a [date_time] | Executes the job only after specific date and time |
-V | Export environment variables in current environment to the job |
-I | Makes job run interactively |
Job Dependencies
It is possible to break down the computational requirements of a large job and build a chain of execution where the next job depends on the previous job finishing successfully.
By utilizing the depend=dependency_list
flag with qsub, this functionality comes to light. For a more complete list of options, please see the official Torque documentation.
Dependency | Description |
---|---|
after:jobid[:jobid…] | Will schedule a job for execution after jobs jobid have started |
afterok:jobid[:jobid…] | Will schedule a job for execution after jobs jobid have terminated with no errors |
before:jobid[:jobid…] | When this job has begun execution, then jobs jobid may begin |
beforeok:jobid[:jobid…] | When this job has terminated with no errors then job jobid may being |
Monitoring Jobs
To monitor the status of a queued or running job, use the qstat
command.
Useful qstat
options include:
qstat option | Description |
---|---|
qstat -q | display all queues |
qstat -a | display all jobs |
qstat -u [user,list] | display jobs for users listed in csv [user,list] |
qstat -r | display running jobs |
qstat -f | display the full listing of job(s) information |
qstat -Qf [queue_id] | display information about queue |
qstat -rn1 | display running job & nodes allocated |
Other useful monitoring commands include:
Command | Description |
---|---|
showq | use Maui to monitor jobs |
showq -i | use Maui to monitor queued jobs |
checkjob [job_id] | show detailed description of the job job_id |
showstart [job_id] | give an estimate of the expected start time of the job job_id |
diagnose -c [LAB/Queue] | give detailed information about a specific queue |
diagnose -j [job_id] | give detailed information about the job |
pbsnodes | versatile PBS node command, when passed nothing will list compute node system status |
Deleting Jobs
To delete queued or running jobs, use the qdel
command:
qdel [job_id]
Should a node become unresponsive to Torque, specify a delay and add the -W option to the command:
qdel -w [job_id]
Compiling and Testing
If you recall the General Rules of Using Aeolus, you should NOT run or compile software on the ssh/login node(s). Well, it is not for a lack of resources, it is because that is not the intended location for the task.
To request resource for compiling software, we suggest you enter into an interactive session. This will allocate appropriate resources for you to compile your software and maintain an environment (on a compute node) for you to interact with the building process. qsub -I
will request interactive mode and qsub -l
will provide properties for the request.
qsub -I -l nodes=1:dev:ppn=1,mem=1024mb,walltime=00:05:00 [~/bin/testapp/build.sh]