Hokule User's Guide

Preface

Index

• System Features
• How to get an account
• How to login and change the password
• Login Info and Enviroment Setup
• File Systems and Disk space
• Compilers
• Scientific and High Performance libraries
• Queue System
  • FFTW
  • BLAS
  • LAPACK
  • ATLAS
  • MKL
• FAQ
• Further information
• How to get help

System Features

How to get an account

How to login and change the password

Before any login sessions can be initiated using ssh, a working SSH client (version 2) needs to be present in the local machine.

To initiate a ssh connection to a machine, type the following on the local workstation

ssh [<login-name>@]hokule.sissa.it;

To change the password:

passwd

chage -l <USERNAME>

Login Info and Enviroment Setup

Check your environment with the env command. You should be careful modifying the shell customization files (.cshrc .profile .login .bashrc), since they could overwrite the default values altering the behaviour of the compilers and of the batch queuing system.

Plase note that interactive login is only allowed on the master node. Computing nodes are accessed only through the master node.

File Systems and Disk space

• /home filesystem
  Home directories are on /u filesystem (21 GB) on the master/login node and are mounted via nfs by the compute nodes (node001-node012). Each user will therefore have something like: /u/<groupname>/<username>. Please note that there is NO BACKUP of the home filesystem. It is left to user the task to save important data on safe disks.
  Quota is enabled.
  
  
• /scratch_gpfs cluster filesystem
  This is the real working filesystem of the cluster. It is a GPFS (Global Parallel File System) of about 100 Gbytes. It is physically formed by 7 disks, one of each computing node but is logicallly seen as a single large filesystem by all the nodes.
  This filesystem should be used for production runs. especially if I/O performance significantly affects program performance. It can also be used to store large files temporarily.
  The filesystem is NFS exported on the master node to facilitate its usage during batch production. Please note that accordingly to GPFS guide
  

  "For performance reasons, Linux NFS code caches information at the client and delays updating some state information, such as file size and modification times. Information reported by an NFS client may therefore differ from the GPFS file system on the NFS server node."

  
  
• /scratch_master local filesystem
  The masternode (hokule) has an addictional disk of about 72 GB.
  Quota is enabled.
  

Compilers

The following table show the name of the compilers:

source /net/shared/intel/compiler70/ia32/bin/ifcvars.csh   (tcsh users)
source /net/shared/intel/compiler70/ia32/bin/ifcvars.sh    (bash users)

source /net/shared/intel/intel_fc_8.1/bin/ifortvars.csh   (tcsh users)
source /net/shared/intel/intel_fc_8.1/bin/ifortvars.sh    (bash users)

Queue system

To get started, compile your program into an executable: myprog.exe Then make the following PBS-script and name it myscript.sh:

	#!/bin/sh

	### Specification of which queue to submit to:
	#PBS -q medium

	### Name of program as it will appear in the queue-list
	#PBS -N nameofmyprog

	### Make PBS look in the directory where the command was submitted from:
	cd $PBS_O_WORKDIR

	### execute the program:
	./myprog.exe

Then submit myscript.sh using the command:


qsub myscript.sh

If everything goes well, you should get back the job id which could look something like:

80.hokule.linuxfarm.sissa.it

and your job has been placed in the queue. It will execute when enough resources becomes available.
There are of course many more options you can select in the jobscript.

For details type

man qsub

• To see the status of the queue, issue the command:

  qstat -a

• To remove a job in the queue:

  qdel JOBID

  (the JOBID can be found using qstat -a)
  
  
• To remove a job that is running:

  qkill -s SIGKILL JOBID

# qstat -q

server: hokule

Queue            Memory CPU Time Walltime Node Run Que Lm  State
---------------- ------ -------- -------- ---- --- --- --  -----
verylong           --   72:00:00    --     --    0   0  6   E R
long               --   12:00:00    --     --    0   0 10   E R
medium             --   04:00:00    --     --    0   0 10   E R
small              --   01:00:00    --     --    1   0 10   E R
default            --      --       --     --    0   0 10   E R
                                               --- ---

man qsub / qstat / qdel

Scientific and High Performance libraries

lib<COMPILER><LIBNAME>

Note that ORDER *DOES* MATTER when linking against more libraries.

• FFTW 2.1.3

High performance routines for real and complex Discrete Fourier Transforms.

To use them:

g77 foo.f -L/net/shared/lib -lgnufftw

ifc foo.f -L/net/shared/lib -lifcfftw

pgf77 foo.f -L/net/shared/lib -lpgifftw

• BLAS

Basic Linear Algebra Subroutines.

Original BLAS compiled with ifc and g77:

libgnublas_std.a
libifcblas_std.a
libifortblas_std.a


To use them:

Standard BLAS
g77 foo.f -L/net/shared/lib -lgnublas_std (GNU Fortran compiler)
ifc foo.f -L/net/shared/lib -lifcblas_std (Intel Fortran compiler 7.1)
ifort foo.f -L/net/shared/lib -lifortblas_std (Intel Fortran compiler 8.0)

However users are strongly encouraged to use ATLAS version of this library (see below).



• LAPACK

Linear Algebra PACKage.

Original LAPACK compiled with ifc and g77:

libgnulapack_std.a
libgnutmglib_std.a
libifclapack_std.a
libifctmglib_std.a
libifortlapack_std.a
libiforttmglib_std.a


NOTE: when using LAPACK libraries you have to link the blas libraries too (AFTER lapack, order is important).

Standard LAPACK
g77 foo.f -L/net/shared/lib -lgnulapack_std -lgnublas_std (GNU Fortran compiler)
ifc foo.f -L/net/shared/lib -lifclapack_std -lifcblas_std (Intel Fortran compiler 7.1)
ifc foo.f -L/net/shared/lib -lifortlapack_std -lifortblas_std (Intel Fortran compiler 8.0)

• ATLAS 3.6.0

Automatically Tuned Linear Algebra Software

The ATLAS (Automatically Tuned Linear Algebra Software) project is an ongoing research effort focusing on applying empirical techniques in order to provide portable performance. At present, it provides C and Fortran77 interfaces to a portably efficient BLAS implementation, as well as a few routines from LAPACK.

ATLAS does not provide a full LAPACK library. However, we include in a full LAPACK library the ATLAS faster LAPACK routines.

To use them:

ATLAS+LAPACK
g77 foo.f -L/net/shared/lib -lgnulapack -lgnublas -lgnuatlas (GNU Fortran compiler)
ifc foo.f -L/net/shared/lib -lifclapack -lifcblas -lifcatlas (Intel Fortran compiler 7.1)
ifort foo.f -L/net/shared/lib -lifortlapack -lifortblas -lifortatlas (Intel Fortran compiler 8.0)
pgf77 foo.f -L/net/shared/lib -lpgilapack -lpgiblas -lpgiatlas (PGI Fortran compiler)

ATLAS+BLAS
g77 foo.f -L/net/shared/lib -lgnublas -lgnuatlas (GNU Fortran compiler)
ifc foo.f -L/net/shared/lib -lifcblas -lifcatlas (Intel Fortran compiler 7.1)
ifort foo.f -L/net/shared/lib -lifortblas -lifortatlas (Intel Fortran compiler 8.0)
pgf77 foo.f -L/net/shared/lib -lpgiblas -lpgiatlas (PGI Fortran compiler)

• INTEL MKL library

The "Math Kernel Library" consists of functions in the following computational areas:

• BLAS (vector-vector, matrix-vector, matrix-matrix operations) and extended BLAS for sparse computations
• LAPACK for linear algebraic equations solutions and eigensystem analysis
• Fast Fourier transforms
• Interface for use by both C and Fortran compilers, for all of the above functionality in addition to availability for all precisions and/or types. Some of functionality have improved performance by multi-threaded implementation.

In addition, MKL also offers a set of functions collectively known as VML -- the "Vector Math Library". VML is a set of vectorized transcendental functions which offer both high performance and excellent accuracy compared to the libm functions for most of the processors. For the Pentium 3 processor, the accuracy of these functions approaches that of libm, but these vectorized functions are considerably faster for vectors longer than a few elements.

For running MKL and/or VML, the library directory, name and the include directory needs to be included in your makefile or command line. Below is an example of how to compile and link a program which calls a BLAS function in the MKL.

For additional documentation and reference on MKL, both pdf and html-based, please look in the directory /net/shared/intel/mkl/doc.

To use them:

ifc foo.f -L/net/shared/intel/mkl/lib/32 -lguide -lmkl_lapack -lmkl (Intel Fortran compiler ONLY)

To use LAPACK and BLAS software you must link two libraries: LAPACK and one of the processor specific kernels.
Some possible variants:

ld myprog.o /net/shared/intel/mkl/lib/32/libmkl_lapack.a /net/shared/intel/mkl/lib/32/libmkl_p3.a     32-bit processor static linking, LAPACK library, Pentium III processor kernel.

ld myprog.o /net/shared/intel/mkl/lib/32/libmkl_lapack.a /net/shared/intel/mkl/lib/32/libmkl_p4.a     32-bit processor static linking, LAPACK library, Pentium 4 processor kernel.

ld myprog.o /net/shared/intel/mkl/lib/32/libmkl.so -lguide -lpthread     32-bit processor dynamic linking. DLL dispatcher will load the appropriate dll for the processor dynamic kernel.

Further information

• Ganglia  (DEMOCRITOS/SISSA GRID monitor tool)
• MKL library (local documentation)

How to get help