Compiling NAMD-2.11 with Intel 2013-SP1 and iMPI 4.1.3 and FFTW-2.1.5

Step 1: Prepare for Environment Setup

source /usr/local/intel_2013sp1/composerxe/mkl/bin/mklvars.sh intel64
source /usr/local/intel_2013sp1/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2013sp1/impi/4.1.3.048/intel64/bin/mpivars.sh intel64
source /usr/local/intel_2013sp1/composerxe/tbb/bin/tbbvars.sh intel64
source /usr/local/intel_2013sp1/itac/8.1.4.045/intel64/bin/itacvars.sh

export CC=icc
export CXX=icpc
export F77=ifort
export F90=ifort

Step 2: Building FFTW-2.1.5 with Intel

$ wget http://www.fftw.org/fftw-2.1.5.tar.gz
$ tar -zxvf fftw-2.1.5.tar.gz
$ cd fftw-2.1.5
$ ./configure F77=ifort CC=icc CFLAGS=-O3 FFLAGS=-O3 --enable-threads --enable-float --enable-type-prefix --prefix=/usr/local/fftw-2.1.5_intel-4.1.3
$ make -j 16
$ make install

Step 3: Building CHARM-6.7.0

$ tar -zxvf NAMD_2.11_Source.tar.gz
$ export $NAMD_SRC=$PWD/NAMD_2.11_Source
$ cd $NAMD_SRC
$ tar -xvf charm-6.7.0.tar
$ MPICXX=mpiicpc CXX=icpc ./build charm++  mpi-linux-x86_64 mpicxx ifort --with-production --no-shared -O3 -DCMK_OPTIMIZE=1
$ cd $NAMD_SRC

Step 4: Building TCL-8.5.9

$ wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64.tar.gz
$ wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64-threaded.tar.gz
$ tar xzf tcl8.5.9-linux-x86_64.tar.gz
$ tar xzf tcl8.5.9-linux-x86_64-threaded.tar.gz
$ mv tcl8.5.9-linux-x86_64 tcl-8.5.9
$ mv tcl8.5.9-linux-x86_64-threaded tcl-8.5.9-threaded

Step 5: Setup the CHARMBASE in $NAMD_SRC for Make.charm

# Set CHARMBASE to the top level charm directory.
# The config script will override this setting if there is a directory
# called charm-6.7.0 or charm in the NAMD base directory.
CHARMBASE = /home/user1/NAMD/NAMD_2.11_Source/charm-6.7.0

Step 6: Setup the FFTW architecture files in $NAMD_SRC/arch

$ vim $NAMD_SRC/arch/Linux-x86_64.fftw

FFTDIR=/usr/local/fftw-2.1.5_intel-14.0.2
FFTINCL=-I$(FFTDIR)/include
FFTLIB=-L$(FFTDIR)/lib -lsrfftw -lsfftw
FFTFLAGS=-DNAMD_FFTW
FFT=$(FFTINCL) $(FFTFLAGS)

Step 7: Setup the TCL architecture files in $NAMD_SRC/arch

 $ vim $NAMD_SRC/arch/Linux-x86_64.tcl

TCLDIR=/usr/local/tcl8.5.9-threaded
TCLINCL=-I$(TCLDIR)/include
TCLLIB=-L$(TCLDIR)/lib -ltcl8.5 -ldl -lpthread
TCLFLAGS=-DNAMD_TCL
TCL=$(TCLINCL) $(TCLFLAGS)

Step 8: Setup the $NAMD_SRC/Linux-x86_64-ics-2013.arch

$ vim Linux-x86_64-ics-2013.arch

NAMD_ARCH = Linux-x86_64
CHARMARCH = mpi-linux-x86_64-ifort-mpicxx
FLOATOPTS = -O2
CXX = icpc -std=c++11
CXXOPTS = -static-intel -O2 $(FLOATOPTS)
CXXNOALIASOPTS = -O3 -fno-alias $(FLOATOPTS)
CC = icc
COPTS = -static-intel -O2 $(FLOATOPTS)

Step 9: Compile the Code.

$ ./config Linux-x86_64-ics-2013 --charm-base ./charm-6.7.0 --charm-arch mpi-linux-x86_64-ifort-mpicxx
$ cd Linux-x86_64-ics-2013
$ make -j 16

You should see the namd2 executable

 Step 10: MPIRUN

$ mpirun -np 32 -machinefile $MACHINEFILE namd2  something.conf > job$LSB_JOBID.log

References:

1. How to install NAMD 2.9 with Intel Cluster Studio 2013 on Intel Sandy Bridge architecture and IB support
2. Compiling NAMD
3. Namd-old

Installing NAMD 2.9 with Intel Cluster Studio 2013 with IB support

There is a very good article of Installing NAMD 2.9 with Intel Cluster Studio 2013 with IB support. You may want to take a look at

How to install NAMD 2.9 with Intel Cluster Studio 2013 on Intel Sandy Bridge architecture and IB support

List of mkl_solver* libraries are deprecated libraries since version 10.2 Update 2

Taken from mkl_solver* libraries are deprecated libraries since version 10.2 Update 2

Since version 10.2 update 2 of Intel® MKL,
all components of Direct Solver (Pardiso and DSS), Trust-Region (TR) Solver, Iterative Sparse Solver (ISS) and GNU Multiple Precision (GMP) were moved to standard MKL libraries.
So now solver ( e.g: mkl_solver.lib and mkl_solver_sequential.lib for IA32 ) libraries are
empty (for backward compatibility).
The list of deprecated libraries are the following:

Intel® MKL for Linux:

lib/32/libmkl_solver.a
lib/32/libmkl_solver_sequential.a
lib/em64t/libmkl_solver_ilp64.a
lib/em64t/libmkl_solver_ilp64_sequential.a
lib/em64t/libmkl_solver_lp64.a
lib/em64t/libmkl_solver_lp64_sequential.a
lib/ia64/libmkl_solver_ilp64.a
lib/ia64/libmkl_solver_ilp64_sequential.a
lib/ia64/libmkl_solver_lp64.a
lib/ia64/libmkl_solver_lp64_sequential.a

Therefore, the updated linking line will look like:


Linking on Intel®64:
static linking:
ifort pardiso.f -L$MKLPATH -I$MKLINCLUDE \
-Wl,--start-group \
$MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a \
-Wl,--end-group -liomp5 -lpthread
dynamic linking:
ifort pardiso.f -L$MKLPATH -I$MKLINCLUDE \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
where:
In these examples,
MKLPATH=$MKLROOT/lib/em64t
MKLINCLUDE=$MKLROOT/include.

Compiling CPMD-3.17.1 with Intel-13.0.1.117 and OpenMPI-1.8.3

I’m assuming you have compiled OpenMPI with Intel Compiler. If you are not sure, you can look at Blog Entry
Compiling OpenMPI 1.6.5 with Intel 12.1.5 on CentOS 6
To get the source code from CPMD, please go to http://www.cpmd.org/

Step 1: From the CPMD Directory

cd ~/CPMD-3.13.2/SOURCE
./mkconfig.sh IFORT-AMD64-MPI > Makefile

Step 2: I’m using CentOS 6 internal Blas, lapack and atlas. Make sure your configure the one below.

#--------------- Default Configuration for IFORT-AMD64-MPI ---------------
SRC  = .
DEST = .
BIN  = .
FFLAGS = -pc64  -tpp6 -O2 -unroll
#LFLAGS =  -L. -latlas_x86_64
LFLAGS =  -L/usr/lib64/atlas -llapack -lblas
CFLAGS = -O2 -Wall -m64
CPP = /lib/cpp -P -C -traditional
CPPFLAGS = -D__Linux -D__PGI -DFFT_DEFAULT -DPOINTER8 -DLINUX_IFC \
-DPARALLEL
NOOPT_FLAG =
CC = mpicc
FC = mpif77 -c
LD = mpif77 -i-static
AR = ar
#----------------------------------------------------------------------------

Step 3: Compile CPMD

# make

If the compilation succeed, it should generate a cpmd.x executable.

Step 4: Pathing
Make sure your $PATH reflect the path of the executable cpmd.x. It is also important to ensure that you check that the libraries are properly linked to the executable

# ldd cpmd.x

Step 5: Test your executable. You have to go to CPMD Consortium to download the cpmd-test.tar.gz for testing.

I/O Direction for Bash

This is a good article on I/O Redirection. I have always been interest to redirect stderr to stdout. So here is it 2>&1

 2>&1
      # Redirects stderr to stdout.
      # Error messages get sent to same place as standard output.
        >>filename 2>&1
            bad_command >>filename 2>&1
            # Appends both stdout and stderr to the file "filename" ...
        2>&1 | [command(s)]
            bad_command 2>&1 | awk '{print $5}'   # found
            # Sends stderr through a pipe.
            # |& was added to Bash 4 as an abbreviation for 2>&1 |.

Enable Centrify Agent to read UID and GID from Centrify DirectManage Access Manager

We purchased Centrify Standard and setup the DirectManage Access Manager. Next we proceed to install the client agent on the compute node.

After unpacking and installing the agent, when we do a

# getent passwd  |grep kittycool
kittycool:x:1304567321211:1304567321211:kittycool:/home/kittycool:/bin/bash
kittycool:x:10001:10001:kittycool:/home/kittycool:/bin/bash

Apparently, the getent passwd |grep kittycool is pulling both the Active Directory UID and the DirectManage Access and the user UID differs

To resolve this issue, you need to specify the zone which is used by DirectManage Access Manager, so your UID of the user will pick from the DirectManage Access Manager.

# adjoin -z cluster -u OU_Administrator  staff.mycompany.com.sg -c "staff.mycompany.com.sg/HPC/Computers"

To check it is displaying the correct UID and GID,

# getent passwd  |grep kittycool
kittycool:x:10001:10001:kittycool:/home/kittycool:/bin/bash

LSF retained the original Max Locked Memory and not the updated one

The value of “max locked memory” has been modified at the operating system level, but LSF still returns the original value.

Symptoms before updating max locked memory

[user1@cluster-h00 ~]$ bsub -m compute-node1 -I ulimit -a
Job <32400> is submitted to default queue .
<>
<>
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1027790
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1027790
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To resolve this issue,

# badmin hshutdown
# badmin hstartup

[user1@cluster-h00 ~]$ bsub -q gpgpu -m compute-node1 -I ulimit -a
Job <32490> is submitted to queue .
<>
<>
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515133
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 515133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

References:

1. LSF does not recognize that “max locked memory” has been updated

Algorithm negotiation failed for SSH Secure Shell Client

If you are using the dated SSH Secure Shell Client 3.2.9, you may have issue connect to the more updated OpenSSH Server.

If you cannot change the client (which is recommended), you will have to update the OpenSSH Server on Linux. Add this in

# vim /etc/ssh/sshd_config

# Ciphers
Ciphers aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,arcfour
KexAlgorithms diffie-hellman-group1-sha1

*If you are using Centrify-OpenSSH, you have to modify /etc/centrifydc/ssh/sshd_config and do the same

References:

1. Bug 1228013 - Server responded "Algorithm negotiation failed"  

Booting using UEFI for Lenovo Server Hardware

To boot CentOS 7.2 using UEFI. Do the following

Step 1: Choose Boot manager
Step 2 : Scroll down and look for Boot Mode
Step 3: In Boot Mode, choose System Boot mode as Uefi and Legacy Only,
Step 4: Go back to  Boot manager,
Step 5: Choose Boot from file
Step 6: Choose No volume label
Step 7: Choose EFI
Step 8: Choose Boot,
Step 9: Choose bootx64,