General run-time tuning for Open MPI 1.4 and later (Part1)

Taken from 17. How do I tell Open MPI to use processor and/or memory affinity in Open MPI v1.4.x? (How do I use the –by* and –bind-to-* options?)
During the mpirun, you can put in the parameter of the Open MPI 1.4 and above to improve performance

1. –bind-to-none: Do not bind processes (Default)
2. –bind-to-core: Bind each MPI process to a core
3. –bind-to-socket: Bind each MPI process to a processor socket
4. –report bindings: Report how the launches processes are bound by Open MPI

If the hardware has multiple hardware threads like those belonging to Hyperthreading, only the first thread of each core is used with the -bind-to-*. According to the article, it is supposed to be fixed in v1.5
The following options below is to be used with –bind-to-*

1. –byslot: Alias for –bycore
2. –bycore: When laying out processes, put sequential MPI processes on adjacent processor cores. (Default)
3. –bysocket: When laying out processes, put sequential MPI processes on adjacent processor sockets.
4. –bynode: When laying out processes, put sequential MPI processes on adjacent nodes.

Finally you can use the –cpus-per-procs which binds ncpus OS processor IDS to each MPI process. If there is a machine with 4 cores and 4 cores, hence 16 cores in total.

$ mpirun -np 8 --cupus-per-proc 2 my_mpi_process

The command will bind each MPI process to ncpus=2 cores. All cores on the machine will be used.

Unable to remove mpi-selector for OFED 1.5.3 during uninstallation

If  you encounter this error "cannot remove mpi-selector" when uninstalling or even installing OFED package, it is best to exit the package and do a yum remove mpi-selector. Apparently, there there are a number of dependencies to BLAS, Scalapack etc.

# yum remove mpi-selector.

Later you can reinstall the dependencies like BLAS, SCALAPACK etc

Testing the Infiniband Interconnect Performance with Intel MPI Benchmark

This writeup focuses on verifying the performance of the Infiniband Interconnects or RDMA/iWARP Interconnects with Intel MPI Benchmark. For more information, do look at my Linux Cluster Blog

1. Testing the Infiniband Interconnect Performance with Intel MPI Benchmark (Part I)
2. Testing the Infiniband Interconnect Performance with Intel MPI Benchmark (Part II) 

More understanding of ulimit

Adding on to blog entries Checking that the ulimit is correct when login as ssh

The stack is nothing but a memory storage which is used by programs to hold intermediate results and subroutine return information. The default size of stack is 10240 KB. You may check it by executing "ulimit -a" command.

The stack reclaim the memory when a process exits. If you allocate more memory in stack, but the memory is not available then it may crash due to stack overflow.


If the stack size is set to too small value and this is not enough for a program then it will fail.

Interesting Whitepaper - "Achieving MPI Performance with Fabrics Collectives Offload"

This white paper "Achieving MPI Performance with Fabrics Collectives Offload" written by Voltaire scientist namely Avid Cohen, Yiftah Sharhar, Yaron Haviv and Asaf Wachtel is really a good reading for those who are trying to figure out how to improve MPI from a fabric point of view.

Happy reading.

Default ulimit setting in torque overide ulimit setting

Even though you may have setup the ulimit correctly as suggested by Blog entry "Checking that the ulimit is correct when login as ssh", your job may still encounter the error is you are using torque resource manager.
This is because the default setting in /etc/init.d/pbs_mom is set too low and overide the ulimit setting, the default setting at /etc/init.d/pbs_mom should be changed to

ulimit -l unlimited
ulimit -c unlimited
ulimit -n 32768
#ulimit -n 20000
#ulimit -i 20000

Checking that the ulimit is correct when login as ssh

To check that whether your ulimit work properly when you ssh, you should use the below command. To set ulimit, you may want to check on blog entry Encountering Segmentation Fault, Bus Error or No output

$ ssh node00 ulimit -l

To check that ulimits is set correctly for your mpirun

$ mpirun -np 8 bash -c ulimit -l

The 2 commands output should be similar.

Defunct but useful ClusterWorld Magazine

ClusterWorld Magazine, now defunct as it merge with Linux Magazine. The last edition is June 2005. But nonetheless, it has interesting information on MPI which is still applicable today.

MPIRun and " You may set your LD_LIBRARY_PATH to have the location of the shared libraries ...... " issues

The Scenario:
I encountered this error while executing an mpirun. Do a "pbsnodes -l" and everything seems is online. I thought my $LD_LIBRARY_PATH was giving the issues. But after some exhaustive check, I've realise that communication to one of our nodes was having issues. Here's are the steps I took to solve the issue

--------------------------------------------------------------------------
A daemon (pid 16704) died unexpectedly with status 127 while attempting to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the  location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------

The Error seems like it is due to LD_LIBRARY_PATH, but it may or may not.

Step 1: Check whether it is a LD_LIBRARY_PATH Issue for your head and compute node
First thing first, you should try to check whether you have the pathing of your LD_LIBRARY_PATH is blank or filled with the correct information for your head node and compute node.

$ echo $LD_LIBRARY_PATH
$/usr/local/lib:/opt/intel/Compiler/11.1/069/lib/intel64 .....

If everything looks normal. Proceed to step 2


Step 2: Check whether the mpirun can be executed cleanly.

$ mpirun -np 32 -hostfile hostfilename openmpi-with-intel-hello-world

where

1. hostfilename contains all the compute node host name
2. openmpi-with-intel-hello-world is the compiled mpi program

Step 3: If the error still remains.....
Modify the hostfilename and insert 1 compute node at a time and compile the  mpirun. You should be able to quickly identify that the problem is not $LD_LIBRARY_PATH but a problematic compute node

n01
n02
...

. In my situation, my problem was due to a broken ssh-generated-key and despite my torque showing all nodes as healthy