Briareo User's Guide

E io a lui: «S'esser puote, io vorrei che de lo smisurato Briareo esperienza avesser li occhi miei».
(Dante, Canto XXXI v98)

Preface

Index

• System Features
• How to get an account
• How to login and change the password
• Login Info and Enviroment Setup
• File Systems and Disk space
• Programming models
• Compilers
  • Serial Compilation
  • Compiling OpenMP Programs
  • Parallel compilation with MPI
• Running codes
• Running serial codes
• Running MPI parallel codes
• Running interactive MPI programs
• Batch Processing
• PBS Options
• Maui Commands
• PBS Environment Variables
• Job Script Template
• Submitting a Job
• Monitoring a Job
• Queue system
• Examples
• Scientific and High Performance libraries
  • FFTW
  • BLAS
  • LAPACK
  • ScaLAPACK
  • BLACS
  • ATLAS
  • MKL
• FAQ
• Further information
• How to get help

System Features

How to get an account

How to login and change the password

Before any login sessions can be initiated using ssh, a working SSH client (version 2) needs to be present in the local machine.

To initiate a ssh connection to a machine, type the following on the local workstation

ssh <login-name>@briareo.sissa.it;

To change the password:

passwd

chage -l <USERNAME>

Login Info and Enviroment Setup

A basic default environment is already set up by means of system login configuration files, this includes variables and paths for the all the compilers and their MPICH implementation of the MPI standard, and OpenPBS batch queuing system with the MAUI scheduler.
Check your environment with the env command. You should be careful modifying the shell customization files (.cshrc .profile .login .bashrc), since they could overwrite the default values altering the behaviour of the compilers and of the batch queuing system.

Plase note that interactive login is only allowed on the master node. Computing nodes are accessed and use only using the PBS queue system.(See later)

File Systems and Disk space

• /home filesystem
  Home directories are on /home filesystem (34 GB) on the master/login node and are mounted via NFS by the compute nodes (node01-node16). Each user will therefore have something like: /home/username .
  The disk space on the home directories is assigned through the quota system. To check your disk space limit use the command quota -v A user's home directory is the place to store files that are routinely used in development and day-to-day work. If the output files from production runs are small, then it is reasonable to store them here.
  Please note that at the moment there is NO BACKUP of the home filesystem. It is left to user the task to save important data on safe disks.
• /scratch filesystem
  This is the real working filesystem of the cluster. It is a GPFS (Global Parallel File System ) of about 140 Gbytes. It is physically formed by 8 disks, one of each computing node but is logicallly seen as a single large filesystem by all the nodes.
  This filesystem should be used for production runs. especially if I/O performance significantly affects program performance. It can also be used to store large files temporarily.
  The filesystem is NFS exported on the master node to facilitate its usage during batch production. Please note that accordingly to GPFS guide "For performance reasons, Linux NFS code caches information at the client and delays updating some state information, such as file size and modification times. Information reported by an NFS client may therefore differ from the GPFS file system on the NFS server node."
• local_scratch filesystem
  Each compute node has an additional local scratch space of about 11 GB mounted on /scratch_local. This local scratch filesystem should be used for parallel application with parallel I/O already implemented.
  This filesystem is not exported on the master node and therefore is not available during interactive sessions.

Programming models

There is another programming model (shared memory programming) that uses either a standard like OpenMP or explicit threaded implementation such as Pthreads.

With the state of the art operating systems, compilers and tools it should be better to use openMP on SMP based nodes with MPI across nodes in hybrid programming enviroment. However there is no sufficient evidence at this time to suggest that using OpenMp with MPI on cluster architecture would be of great benefit from the performance point of view.

Compilers

The following table show the name of the compilers:

source /usr/local/intel/intel_fc_81/bin/ifortvars.csh   (tcsh users)
source /usr/local/intel/intel_fc_81/bin/ifortvars.sh    (bash users)

Serial Compilation

Compilation of serial programs runs as usual:

C	icc|gcc|pgcc -o prog [options] prog.cc [linker options]
Fortran	ifc|g77|pgf90 -o prog [options] prog.f90 [linker options]

"options" denote additional compiler non-default options to be added during the compilation. "linker options" denote represents zero or more linker loader options. The option compiler options take precedence over the link loader options. See later for a list of common option to be used with the compilers.

Compiling OpenMP Programs

Since the cluster nodes are two-way, each of the nodes can technically also act like Symmetric MultiProcessors using the Shared Memory Programming (SMP) paradigm. However because of the limited number of processors in each node, significant performance benefit using the node as an SMP over a node in message-passing model in a distributed memory computer with two processors, is not likely hence use of it in this manner is not recommended at this time. However users can always use the SMP model for program development or port an existing application using OpenMP and use the IA32 node in that manner. It is then straightforward to do so, by just adding appropriate compiler options to the chosen compiler ( Both Intel and PGi compiler support OpenMP standard)

Parallel compilation with MPI

The cluster support compilation of parallel programming by means of the MPICH Message Passing libraries, which is an open source implementation of the popular MPI library. Compiler scripts are provided to compile the MPI code and automatically link startup and message passing libraries to the resulting binary executable. The following table lists the compiler "wrappers" for each language:



Suite	MPI C compiler	MPI C++ compiler	MPI F77 compiler	MPI F90/95 compiler
GNU	mpi_gcc	mpi_gcc	mpi_g77	N/A
Portland Group(3.2.4)	mpi_pgcc	mpi_pgCC	mpi_pgf77	mpi_pgf90
Intel	mpi_icc	mpi_icc	mpi_ifc/mpi_ifort	mpi_ifc/mpi_ifort




Important notes

1. Be careful that object files compiled with PGI/INTEL/GNU compilers have different naming conventions.
   (for instance to link an object compiled with gcc to a code compiled with pgi you should use compatibility flag -Msecond_underscore)



2. Users can always use serial compiler with appropriate include and linking directories where to find mpi include files and mpi libraries. In this case paralllel compilation is achieved in the following way:
   1. fortran77/90
      pgif90|ifc|g77 [option] file.f90 -I$MPICH/include -L$MPICH/lib -lfmpich -lmpich -L/usr/gm/lib -lgm
      
   2. C/C++
      pgcc [option] file.c -I$MPICH/include -L$MPICH/lib -lmpich -L/usr/gm/lib -lgm
      
   Where MPICH environment variable is different for each kind of suite:
   MPICH=/usr/local/mpich/1.2.4..8a/gm-1.6.3_Linux/smp/gnu/ssh for GNU
   MPICH=/usr/local/mpich/1.2.4..8a/gm-1.6.3_Linux/smp/pgi/ssh for PGI
   MPICH=/usr/local/mpich/1.2.4..8a/gm-1.6.3_Linux/smp/intel/ssh for INTEL
   We strongly suggest to use mpi wrappers unless you have clear what you are doing.



3. We strongly discourage use of PGI compilers due to the old version we currently have at SISSA.

Running codes

• Running serial codes

On the master node no production is allowed and any serial execution program lasting more than 5 minutes is automatically deleted. Execution of serial application on computational nodes can be only done the through the queuing system, even for interactive runs.
We remember however that this system is not really indicated to run program: it is better to find out other computational resources for this kind of production (see hokule cluster for instance).

• Running MPI parallel codes

To run MPI parallel program users have to use the command mpiexec: fundamental options is: -n number_of_cpus to specify number of cpus. If not specified, mpiexec will run the code on the available cpu (i.e. the ones requested through PBS).



• Running interactive MPI programs

Interactive use of mpiexec has been turned off so that production code will run in dedicated mode through PBS, the Portable Batch System. PBS does provide "interactive" capability (stdin, stdout, and stderr are connected to the tty).

Suppose for instance you want to run your a test.x interactively on four processors then you could use the following sequence of commands:

qsub -l nodes=2:ppn=2,walltime=0:30:00 -I

at this point (if there are free resources) you will enter in the batch interactive session, and you could run your test with:

cd testdir
mpiexec -n 4 -no-shmem test.x

Note that each node has two processors, hence to run on 2N processors you have to ask only N nodes. Remember that when you are in the batch environment the only processors (resources in general) you could use, are those allocated and displayed at the beginning of the session. You will stay in a sort of cluster partition of your own, no other could enter there.

Example of an interactive execution:

qsub -l nodes=2:ppn=2,walltime=0:30:00 -I
cd testdir
mpiexec -n 4 test.x

• Batch Processing

The Portable Batch System, PBS, is a workload management system for Linux clusters. It supplies command to submit, monitor, and delete jobs. It has the following components.

• Job Server - also called pbs_server provides the basic batch services such as receiving/creating a batch job, modifying the job, protecting the job against system crashes, and running the job.
• Job Executor - a daemon (pbs_mom) that actually places the job into execution when it receives a copy of the job from the Job Server. Mom creates a new session as identical to a user login session as is possible and returns the job's output to the user.
• Job Scheduler - a daemon that contains the site's policy controlling which job is run and where and when it is run. PBS allows each site to create its own Scheduler. We are using the Maui Scheduler. The Maui Scheduler can communicate with various Moms to learn about the state of a system's resources and with the Server to learn about the availability of jobs to execute.

Below are the steps needed to run production code:

1. Create a job script containing the following PBS options:
   • request the resources that will be needed (i.e. number of processors, wall-clock time, etc.) and
   • use commands to prepare for execution of the executable (i.e. cd to working directory, etc.).
2. Submit the job script file to PBS.
3. Monitor the job.

• PBS Options

Below are some of the commonly used PBS options in a job script file. The options start with "#PBS."

Option	Description
#PBS -N myJob	Assigns a job name. The default is the name of PBS job script.
#PBS -l nodes=4:ppn=2	The number of nodes and processors per node.
#PBS -l walltime=01:00:00	The maximum wall-clock time during which this job can run.
#PBS -o mypath/my.out	The path and file name for standard output.
#PBS -e mypath/my.err	The path and file name for standard error.
#PBS -j oe	Join option that merges the standard error stream with the standard output stream of the job.
#PBS -W stagein=file_list	Copies the file onto the execution host before the job starts. (*)
#PBS -W stageout=file_list	Copies the file from the execution host after the job completes. (*)
#PBS -m b	Sends mail to the user when the job begins.
#PBS -m e	Sends mail to the user when the job ends.
#PBS -m a	Sends mail to the user when job aborts (with an error).
#PBS -m ba	Allows a user to have more than one command with the same flag by grouping the messages together on one line, else only the last command gets executed.
#PBS -r n	Indicates that a job should not rerun if it fails.
#PBS -V	Exports all environment variables to the job.

(*) File staging can specify which files should be copied onto the execution host before the job starts and which files should be copied off the execution host when it completes. The file_list regardless of the direction of copy, is of the following form, where the name local_file is the name of the file on the system where the job executes, and the remote_file is the destination name on the host specified by hostname: local_file@hostname:remote_file.
stagein=my.input@frontend-0:/home/login_name/my.input
stageout=my.output@frontend-0:/home/login_name/my.output

• Maui Commands

There are some quite useful Maui commands:

Command	Description
showq	Show a detailed list of submitted jobs
showbf	Show the free resources (time and processors available) at the moment
checkjob job.ID	show a detailed description of the job job.ID
showstart job.ID	gives an estimate of the expected started time of the job job.ID

• PBS Environment Variables

There are a number of predefined environment variables. These include the following:

• Variables defined on the execution host;
• Variables exported from the submission host to the execution host; and
• Variables defined by PBS.

The following environment variables relate to the submission machine:

Option	Description
PBS_O_HOST	The host machine on which the qsub command was run.
PBS_O_LOGNAME	The login name on the machine on which the qsub was run.
PBS_O_HOME	The home directory from which the qsub was run.
PBS_O_WORKDIR	The working directory from which the qsub was run.

The following variables relate to the environment where the job is executing:

Option	Description
PBS_ENVIRONMENT	This is set to PBS_BATCH for batch jobs and to PBS_INTERACTIVE for interactive jobs.
PBS_O_QUEUE	The original queue to which the job was submitted.
PBS_JOBID	The identifier that PBS assigns to the job.
PBS_JOBNAME	The name of the job.
PBS_NODEFILE	The file containing the list of nodes assigned to a parallel job.

• Job Script Template

The following job script template should be modified for the need of the job.

#!/bin/csh #PBS -N MY_JOB #PBS -l nodes=4:ppn=2 #PBS -l walltime=01:00:00 #PBS -j oe echo "MASTER HOST: $PBS_O_HOST is the master host" echo "NODEFILE: $PBS_NODEFILE is the nodefile" echo "PBS_O_WORKDIR: $PBS_O_WORKDIR" cd $PBS_O_WORKDIR mpiexec -n 8 ./a.out

• Submitting a Job

Use the qsub command to submit the job.

qsub jobA

PBS assigns a job a unique job identifier once it is submitted (e.g. 123.briareo). After a job has been queued, it is selected for execution based on the time it has been in the queue, wall-clock time limit, and number of processors.

• Monitoring a Job

Below are commands for monitoring a job:

Command	Function
qstat -a	check status of jobs, queues, and the PBS server
qstat -f	get all the information about a job, i.e. resources requested, resource limits, owner, source, destination, queue, etc.
qdel job.ID	delete a job from the queue
qhold job.ID	hold a job if it is in the queue
qrls job.ID	release a job from hold

• Queue system

At present the batch queues are defined as follows: 

QUEUE	N. CPU MAX	Time Limit per CPU (dd+hh:mm)	Total Time Limit (dd+hh:mm)
dque	64	0+>10:00	-

NOTE:this configuration can change accordinly to future needs and requirement

• Examples:

Suppose you want tu run the program hello.x on 16 processor for 1 hour, then if you want to specify the requests on the qsub command line you should write the jobscript file as follows:

#!/bin/sh
cd workdir
mpiexec -n 16 hello.x



and you can submit it to the queuin system with the command:

qsub -l nodes=8:ppn=2,walltime=1:00:00 jobscript



If you prefer to include the requests in the jobscript, then the jobscript should be:

#!/bin/sh
#PBS -l nodes=8:ppn=2,walltime=1:00:00
cd workdir
mpiexec -n 16 hello.x


and you can submit it to the queuin system with the command:

qsub jobscript

Scientific and High Performance libraries

lib<COMPILER><LIBNAME>

• FFTW 2.1.3

High performance routines for real and complex Discrete Fourier Transforms. Includes serial, shared memory (pthreads), and distributed memory (MPI) versions of the libraries, in both single and double precision.

It has been compiled with MPI support and by default it uses the pgi fortran compiler in its fortran interface. To use them

pgf77 -O3 foo.f -L/usr/local/lib -lpgifftw

mpi_pgf77 -O3 foo.f -L/usr/local/lib -lpgifftw_mpi (parallel version)

• BLAS

Basic Linear Algebra Subroutines.
There is a precompiled version for pgi compiler automatically loaded using the -l<COMPILER>blas flag. However users are strongly encouraged to use ATLAS version of this library.

LAPACK

ScaLAPACK

A subset of LAPACK designed for use on distributed memory parallel architectures. Includes the PBLAS (Parallel BLAS) library.

• MPI BLACS 1.1

Basic Linear Algebra Communications Subprograms for MPI, used by ScaLAPACK and other mathematical software. Built for use with MPICH

• ATLAS 3.4.1

High-performance BLAS and LAPACK routines and for C and Fortran. Each routinne is individually tuned for the underlying hardware. The library has been compiled using gcc compiler (the suggested one) and with fortran wrapper for all the fortran compiler.
To use them:

pgf77 foo.f -L/usr/local/lib -lpgiblas -lpgiatlas (PGI Fortran compiler)

ifcc foo.f -L/usr/local/lib -lifcblas -lifcatlas (Intel Fortran compiler)

g77 foo.f -L/usr/local/lib -lgnublas -lgnuatlas (GNU Fortran compiler)

mpi_pgf77 -O3 foo.f -L/usr/local/lib -lpgiblas -lpgiatlas

• INTEL MKL library

The "Math Kernel Library" consists of functions in the following computational areas:

• BLAS (vector-vector, matrix-vector, matrix-matrix operations) and extended BLAS for sparse computations
• LAPACK for linear algebraic equations solutions and eigensystem analysis
• Fast Fourier transforms
• Interface for use by both C and Fortran compilers, for all of the above functionality in addition to availability for all precisions and/or types. Some of functionality have improved performance by multi-threaded implementation.
In addition, MKL also offers a set of functions collectively known as VML -- the "Vector Math Library". VML is a set of vectorized transcendental functions which offer both high performance and excellent accuracy compared to the libm functions for most of the processors. For the Pentium 3 processor, the accuracy of these functions approaches that of libm, but these vectorized functions are considerably faster for vectors longer than a few elements.

For running MKL and/or VML, the library directory, name and the include directory needs to be included in your makefile or command line. Below is an example of how to compile and link a program which calls a BLAS function in the MKL. Note that the library is for use in a single node, hence can be used by both serial compilers or by MPI wrapper scripts.

mpicc -O3 -I/usr/local/intel/mkl/include foo.c -L/usr/local/intel/mkl/lib/32 -lmkl_p3

For additional documentation and reference on MKL, both pdf and html-based, please look in the directory /usr/local/intel/mkl/doc.

Further information

• Ganglia  (DEMOCRITOS/SISSA GRID monitor tool)
• Mpiexec
• MKL library (local documentation)
• MAUI Scheduler
• PGI Website
• RedHat Linux
• OpenPBS

How to get help