General run-time tuning for Open MPI 1.4 and later (Part1)

Taken from 17. How do I tell Open MPI to use processor and/or memory affinity in Open MPI v1.4.x? (How do I use the –by* and –bind-to-* options?)
During the mpirun, you can put in the parameter of the Open MPI 1.4 and above to improve performance

1. –bind-to-none: Do not bind processes (Default)
2. –bind-to-core: Bind each MPI process to a core
3. –bind-to-socket: Bind each MPI process to a processor socket
4. –report bindings: Report how the launches processes are bound by Open MPI

If the hardware has multiple hardware threads like those belonging to Hyperthreading, only the first thread of each core is used with the -bind-to-*. According to the article, it is supposed to be fixed in v1.5
The following options below is to be used with –bind-to-*

1. –byslot: Alias for –bycore
2. –bycore: When laying out processes, put sequential MPI processes on adjacent processor cores. (Default)
3. –bysocket: When laying out processes, put sequential MPI processes on adjacent processor sockets.
4. –bynode: When laying out processes, put sequential MPI processes on adjacent nodes.

Finally you can use the –cpus-per-procs which binds ncpus OS processor IDS to each MPI process. If there is a machine with 4 cores and 4 cores, hence 16 cores in total.

$ mpirun -np 8 --cupus-per-proc 2 my_mpi_process

The command will bind each MPI process to ncpus=2 cores. All cores on the machine will be used.

Encountering cannot restore segment prot after reloc during mpirun

If you encounter this error "Encountering cannot restore segment prot after reloc: permission denied " during a mpirun, this is due to the presence of SELinux setting is enabled on the server. In CentOS, you can disabled this setting by changing /etc/selinux to

SELINUX=disabled

To ensure the above commands take effect, you need to reboot.


If you do not wish to reboot just yet, You can using this command to prevent the current selinux from running

# /usr/sbin/setenforce 0

Diagnosing stuck OpenMPI jobs with mpirun

I was testing a simple Openmpi jobs and somehow my jobs hangs without any output. In order to diagnose why the mpi-run failed, I've used this cool flag to

 $ mpirun --debug-daemons -np 24 -host c18,c19,c20  hello_world_mpi

..........
[c18.cluster.spms.ntu.edu.sg:09182] [[21720,0],1] orted_cmd: received exit
[c19.cluster.spms.ntu.edu.sg:13800] [[21720,0],2] orted_cmd: received exit
[c19.cluster.spms.ntu.edu.sg:13800] [[21720,0],2] orted: finalizing
[c18.cluster.spms.ntu.edu.sg:09182] [[21720,0],1] orted: finalizing
[c20.cluster.spms.ntu.edu.sg:08614] [[21720,0],3] orted_cmd: received exit
[c20.cluster.spms.ntu.edu.sg:08614] [[21720,0],3] orted: finalizing 
..........

Finally realised that my mpirun "ran away to a "IBM RNDIS/CDC ETHER" which I promptly shutdown with a ifdown usb0

After that it run smoothly

mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting

If you see this error "mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting", it measns that there is a mis-matched kernel modules and userspace libraries.

Solution: Reinstall the OFED kernel modules. For more information, see Installing Voltaire QDR Infiniband Drivers for CentOS 5.4

If you have upgraded your linux kernel, you may want to reinstall the OFED kernel modules to the upgraded kernel or boot back to your original  kernel

More understanding of ulimit

Adding on to blog entries Checking that the ulimit is correct when login as ssh

The stack is nothing but a memory storage which is used by programs to hold intermediate results and subroutine return information. The default size of stack is 10240 KB. You may check it by executing "ulimit -a" command.

The stack reclaim the memory when a process exits. If you allocate more memory in stack, but the memory is not available then it may crash due to stack overflow.


If the stack size is set to too small value and this is not enough for a program then it will fail.

Default ulimit setting in torque overide ulimit setting

Even though you may have setup the ulimit correctly as suggested by Blog entry "Checking that the ulimit is correct when login as ssh", your job may still encounter the error is you are using torque resource manager.
This is because the default setting in /etc/init.d/pbs_mom is set too low and overide the ulimit setting, the default setting at /etc/init.d/pbs_mom should be changed to

ulimit -l unlimited
ulimit -c unlimited
ulimit -n 32768
#ulimit -n 20000
#ulimit -i 20000

Checking that the ulimit is correct when login as ssh

To check that whether your ulimit work properly when you ssh, you should use the below command. To set ulimit, you may want to check on blog entry Encountering Segmentation Fault, Bus Error or No output

$ ssh node00 ulimit -l

To check that ulimits is set correctly for your mpirun

$ mpirun -np 8 bash -c ulimit -l

The 2 commands output should be similar.

MPIRun and " You may set your LD_LIBRARY_PATH to have the location of the shared libraries ...... " issues

The Scenario:
I encountered this error while executing an mpirun. Do a "pbsnodes -l" and everything seems is online. I thought my $LD_LIBRARY_PATH was giving the issues. But after some exhaustive check, I've realise that communication to one of our nodes was having issues. Here's are the steps I took to solve the issue

--------------------------------------------------------------------------
A daemon (pid 16704) died unexpectedly with status 127 while attempting to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the  location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------

The Error seems like it is due to LD_LIBRARY_PATH, but it may or may not.

Step 1: Check whether it is a LD_LIBRARY_PATH Issue for your head and compute node
First thing first, you should try to check whether you have the pathing of your LD_LIBRARY_PATH is blank or filled with the correct information for your head node and compute node.

$ echo $LD_LIBRARY_PATH
$/usr/local/lib:/opt/intel/Compiler/11.1/069/lib/intel64 .....

If everything looks normal. Proceed to step 2


Step 2: Check whether the mpirun can be executed cleanly.

$ mpirun -np 32 -hostfile hostfilename openmpi-with-intel-hello-world

where

1. hostfilename contains all the compute node host name
2. openmpi-with-intel-hello-world is the compiled mpi program

Step 3: If the error still remains.....
Modify the hostfilename and insert 1 compute node at a time and compile the  mpirun. You should be able to quickly identify that the problem is not $LD_LIBRARY_PATH but a problematic compute node

n01
n02
...

. In my situation, my problem was due to a broken ssh-generated-key and despite my torque showing all nodes as healthy