MULO & SOMARO: The SISSA HPC Linux Cluster

MULO & SOMARO User's Guide

SOMARO	Sorella - Maritan openMosix cluster
MULO	Multi - User Linux openMosix cluster

Preface

The Supermicro IA-32 cluster is a computational resource cofunded by CED - Sorella - Maritan for the computational research at SISSA. This document gives some basic information how to use it at best.

Index

System Features
How to get an account
How to login and change the password
Login Info and Enviroment Setup
File Systems and Disk space
Compilers
Scientific and High Performance libraries
- FFTW
- BLAS
- LAPACK
- ATLAS
- MKL
- GSL
openMosix Section
FAQ
Further information
How to get help

System Features

HARDWARE

SOFTWARE

Model	Supermicro
Front-End Nodes	2 x Supermicro (2 Xeon 2.4 Ghz)
Computing Nodes	5 x Supermicro (2 Xeon 3.06 GHz)
Number of PEs	2 x 2 PIV 2.4Ghz, 2 x 5 PIV 3.06GHz
L2 Cache	512Kb per processor full speed
DRAM	14 Gbytes (2GB/node)
Disk space	2x36 GB + 2x140 GB
Internal Network	2x Intel Gigabit Ethernet card

Operating System	Debian Woody
Nodes Operating System	LFS ramdisk (Warewulf-RH7.3 based)
Kernel Version	2.4.20 smp (patched with openMosix-2.4.20-2)
Cluster Software	-
Filesystem	oMFS, NFS
Available compilers	GNU compilers (v2.95, v3.2) Intel F90 compiler

How to get an account

IMPORTANT:
Mulo and Somaro use Yellow Pages, it means that an account must already exists on the main SISSA cluster. If you are not a registered users please contact our local CED or mail to talpo AT sissa.it

If an account already exists on the main SISSA cluster, please mail to cluster-admin AT democritos.it to enable the login on the Mulo and Somaro cluster.

Username and password are those of the main cluster (e.g. on shannon.sissa.it)

Note that the home IS ON LOCAL filesystem, Somaro and Mulo don't use the home of cm-srv server or the main cluster one.

Be sure to subscribe the official cluster users mailing list.

How to login and change the password

To ensure a secure login session, users must connect to machines using the secure shell, ssh program. Telnet is not allowed because of the security vulnerabilities associated with it. The "r" commands rlogin, rsh, and rcp are also disabled on this machine for similar reasons. These commands are replaced by the more secure alternatives included in SSH --- ssh, scp

Before any login sessions can be initiated using ssh, a working SSH client (version 2) needs to be present in the local machine.

To initiate a ssh connection to a machine, type the following on the local workstation

ssh [<login-name>@]somaro.sissa.it;

ssh [<login-name>@]mulo.sissa.it;

Note that the <login-name> is only needed if the user name on remote machine differs from the user name on the workstation.

To change the password:

yppasswd

Note that you are going to change your password on the main SISSA cluster (and this will affect your main SISSA account).

Login Info and Enviroment Setup

The default shell is the one you have on the main SISSA cluster. At login /etc/motd file is displayed: please take care of reading it, because information about system are usally written there.

Check your environment with the env command. You should be careful modifying the shell customization files (.cshrc .profile .login .bashrc), since they could overwrite the default values altering the behaviour of the compilers.

Plase note that interactive login is only allowed on the master nodes. Computing nodes are accessed only through the master nodes and balanced by openMosix.

SOMARO users
Sorella and Maritan groups only. Please contact Sorella or Micheletti to be allowed to use this machine (and enable your account).
MULO users
Other users. Contact talpo AT sissa.it to enable your account.

File Systems and Disk space

/home filesystem
Home directories are on /u filesystem (140 GB) on the master/login nodes and are mounted via mfs by the compute nodes (node001-node005). Each user will therefore have something like: /u/<groupname>/<username>. All nodes also see each other via mfs.
Please note that at the moment there is NO BACKUP of the home filesystem. It is left to user the task to save important data on safe disks.
Quota is enabled: default quota is 1GB each.
scratch filesystem
Each master node has an additional local scratch space of about 140 GB mounted on /scratch, available via mfs (/mfs/<MASTERNODE>/scratch).
local_scratch filesystem
Each computing node has an additional local scratch space of about 60 GB mounted on /local_scratch, available via mfs (/mfs/<NODE>/scratch).

Compilers

The following compilers are available for compilation for serial application:

Vendor/Source	Compilers	Documentation
Intel	Fortran 77/90/95, C, C++	`/usr/local/intel/compiler70/docs/`[*] `icc -help` `ifc -help` `icpc -help` [Intel Software Download]
GNU	Fortran 77, C, C++	http://gcc.gnu.org/onlinedocs/

The following table show the name and version of the compilers:

Vendor/Source	C compiler	C++ compiler	F77 compiler	F90/95 compiler
GNU v3.3.2	gcc / gcc-3.3	g++ / g++-3.3	g77 / g77-3.3	N/A
GNU v2.95.4	gcc-2.95	g++-2.95	g77-2.95	N/A
GNU v3.0.4	gcc-3.0	g++-3.0	N/A	N/A
GNU v3.2.3	gcc-3.2	g++-3.2	N/A	N/A
Intel v7.1	icc	icc	ifc	ifc

All the compilers listed in the table should be present by default in user's path.

Scientific and High Performance libraries

Some important scientific and High Performance libraries have been installed on the cluster.

It is important to note that each compiler requires its own compilation or at least its own wrapper routines in order to link these libraries correctly.

Note also that each library is available for each compiler available. The syntax is:

lib<COMPILER><LIBNAME>

An example for the fftw library is: libifcfftw, libgnufftw, respectively for ifc, g77 compilers. (You can download a silly Makefile.sample)

Note that ORDER *IS* IMPORTANT when linking against more libraries.

FFTW 2.1.3

High performance routines for real and complex Discrete Fourier Transforms.

To use them:

g77 foo.f -L/usr/local/lib -lgnufftw

ifc foo.f -L/usr/local/lib -lifcfftw

BLAS

libgnublas_std.a
libifcblas_std.a

To use them:

Standard BLAS
g77 foo.f -L/usr/local/lib -lgnublas_std (GNU Fortran compiler)

ifc foo.f -L/usr/local/lib -lifcblas_std (Intel Fortran compiler)

Standard BLAS
`g77 foo.f -L/usr/local/lib -lgnublas_std (GNU Fortran compiler)`
`ifc foo.f -L/usr/local/lib -lifcblas_std (Intel Fortran compiler)`

strongly encouraged

LAPACK

Linear Algebra PACKage.

Original LAPACK compiled with ifc and g77:

libgnulapack_std.a
libgnutmglib_std.a
libifclapack_std.a
libifctmglib_std.a

NOTE: when using LAPACK libraries you have to link the blas libraries too (AFTER lapack, order is important).

Standard LAPACK
g77 foo.f -L/usr/local/lib -lgnulapack_std -lgnublas_std (GNU Fortran compiler)

ifc foo.f -L/usr/local/lib -lifclapack_std -lifcblas_std (Intel Fortran compiler)

ATLAS 3.4.1

Standard LAPACK
`g77 foo.f -L/usr/local/lib -lgnulapack_std -lgnublas_std (GNU Fortran compiler)`
`ifc foo.f -L/usr/local/lib -lifclapack_std -lifcblas_std (Intel Fortran compiler)`

The ATLAS (Automatically Tuned Linear Algebra Software) project is an ongoing research effort focusing on applying empirical techniques in order to provide portable performance. At present, it provides C and Fortran77 interfaces to a portably efficient BLAS implementation, as well as a few routines from LAPACK.

ATLAS does not provide a full LAPACK library. However, we include in a full LAPACK library the ATLAS faster LAPACK routines.

To use them:

ATLAS+LAPACK
g77 foo.f -L/usr/local/lib -lgnulapack -lgnublas -lgnuatlas (GNU Fortran compiler)

ifc foo.f -L/usr/local/lib -lifclapack -lifcblas -lifcatlas (Intel Fortran compiler)

ATLAS+LAPACK
`g77 foo.f -L/usr/local/lib -lgnulapack -lgnublas -lgnuatlas (GNU Fortran compiler)`
`ifc foo.f -L/usr/local/lib -lifclapack -lifcblas -lifcatlas (Intel Fortran compiler)`

ATLAS+BLAS
g77 foo.f -L/usr/local/lib -lgnublas -lgnuatlas (GNU Fortran compiler)

ifc foo.f -L/usr/local/lib -lifcblas -lifcatlas (Intel Fortran compiler)

INTEL MKL library

ATLAS+BLAS
`g77 foo.f -L/usr/local/lib -lgnublas -lgnuatlas (GNU Fortran compiler)`
`ifc foo.f -L/usr/local/lib -lifcblas -lifcatlas (Intel Fortran compiler)`

The "Math Kernel Library" consists of functions in the following computational areas:

BLAS (vector-vector, matrix-vector, matrix-matrix operations) and extended BLAS for sparse computations
LAPACK for linear algebraic equations solutions and eigensystem analysis
Fast Fourier transforms
Interface for use by both C and Fortran compilers, for all of the above functionality in addition to availability for all precisions and/or types. Some of functionality have improved performance by multi-threaded implementation.

"Vector Math Library"

libm

For running MKL and/or VML, the library directory, name and the include directory needs to be included in your makefile or command line. Below is an example of how to compile and link a program which calls a BLAS function in the MKL.

For additional documentation and reference on MKL, both pdf and html-based, please look in the directory /usr/local/intel/mkl/doc.

To use them:

ifc foo.f -L/usr/local/intel/mkl/lib/32 -lguide -lmkl_lapack -lmkl (Intel Fortran compiler ONLY)

To use LAPACK and BLAS software you must link two libraries: LAPACK and one of the processor specific kernels.
Some possible variants:

ld myprog.o /usr/local/intel/mkl/lib/32/libmkl_lapack.a /usr/local/intel/mkl/lib/32/libmkl_p3.a 32-bit processor static linking, LAPACK library, Pentium III processor kernel.

ld myprog.o /usr/local/intel/mkl/lib/32/libmkl_lapack.a /usr/local/intel/mkl/lib/32/libmkl_p4.a 32-bit processor static linking, LAPACK library, Pentium 4 processor kernel.

ld myprog.o /usr/local/intel/mkl/lib/32/libmkl.so -lguide -lpthread 32-bit processor dynamic linking. DLL dispatcher will load the appropriate dll for the processor dynamic kernel.

GSL 1.4

The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers.

To use them:

gcc foo.c -L/usr/local/lib -lgsl -lblas (gcc only)

openMosix Section

Migration Policy

SOMARO
Processes from SOMARO are migrated as soon as possible to the computing nodes, if MULO has free resources the processes can be megrated to MULO too.

MULO
Processes from MULO run on local machine and are migrated only if the computing nodes have free resources. Nodes node001 and node002 CAN NOT accept processes from MULO.

Useful commands

openMosix cmdline tools
mosmon	MOSIX load monitor
mps	report multicomputer process status
mtop	display multicomputer top CPU processes
mosrun -<NODE> <PROGRAM_NAME>	run a command with particular node-allocation preferences
migrate <PID> <NODE>	request migration of a particular process on openMosix
cmdline tools (baro)
ompsinfo	get some openMosix related info about processes (based on minimal ps)
oM_info.sh\|oM_parser.sh <PROCESS_NAME>	get some openMosix related info about a process
oM_check.sh <PROCESS_NAME>	same as above in a 'while true' loop
oM_cat.sh <PID> <oM_FILE>	show content of specified openMosix proc file for a process (given pid)
oM_nmigs.sh <PROCESS_NAME>	check number of migrations for a process
oM_cantmove.sh <PROCESS_NAME>	check if process can be migrated or why not
oM_where.sh <PROCESS_NAME>	check on which node the job is running
GUI oriented tools
openmosixview	graphical cluster management tool
openmosixmigmon	graphical migration monitor tool

Using /mfs, the openMosix filesystem

   #!/bin/bash
   # Usage: sample.sh EXP_NAME N_JOBS
   #        N_NODES <= 10 (2 x N_NODES)

   FORCE_MIG=0

   nodo=0
   for i in `seq 1 $2`
   do

      if [ $2 -gt 5 ]
      then
         let nodo=($i+1)/2
      else
         nodo=$i
      fi

      echo run.$i, nodo $nodo

      CMD="./bin/bonnie++ -d /mfs/$nodo/local_scratch/ -s 2000 -r1000 -b $FLAG"

      if [ "$FORCE_MIG" = "1" ]
      then
         mosrun -$nodo $CMD >out/$EXP_NAME.$i 2>&1 &
      else
         $CMD >out/$EXP_NAME.$i 2>&1 &
      fi
   done

   echo -n "Waiting children..."
   wait
   echo

   #EOF

(full version)

Checkpoint - Restart: chpox

http://www.openmosixview.com/chpox/