ntpdate failing to synchronise clock.

My client timing was ahead of the NTP Server by 15 hours. When I did a

# ntpdate -q 0.centos.pool.ntp.org

The synchronization did not work.

To force the client to synchronise no matter how absurd the time differences, you have to use a "-b" parameter.

# ntpdate -b 0.centos.pool.ntp.org

Compiling LAMMPS-14May16 with Intel-15.0.6 and Intel-MPI-5.0.3

Step 1: Remember to initialise Intel Environment. In your .bashrc

source /usr/local/intel/bin/compilervars.sh intel64

Step 2: Untar LAMMPS

# tar -zxvf lammps-stable.tar.gz

Step 3: Prepare libraries for lammps 3a. lib/reax

# make -f Makefile.gfortran

3b. lib/meam

# make -f Makefile.ifort

3c. lib/poems

# make -f Makefile.icc

Step 4: Compile required the packages. Go to src directory

# cd src

4a. Check which packages are included

# make package-status
Installed YES: package ASPHERE
Installed YES: package BODY
Installed YES: package CLASS2
Installed YES: package COLLOID
Installed YES: package COMPRESS
Installed YES: package CORESHELL
Installed YES: package DIPOLE
Installed YES: package FLD
Installed  NO: package GPU
Installed YES: package GRANULAR
Installed  NO: package KIM
Installed YES: package KOKKOS
  src/pair_lj_sdk_kokkos.cpp does not exist
  src/pair_lj_sdk_kokkos.h does not exist
Installed YES: package KSPACE
Installed YES: package MANYBODY
Installed YES: package MC
Installed YES: package MEAM
Installed YES: package MISC
Installed YES: package MOLECULE
Installed YES: package MPIIO
Installed YES: package OPT
Installed YES: package PERI
Installed YES: package POEMS
Installed YES: package PYTHON
Installed YES: package QEQ
Installed YES: package REAX
Installed YES: package REPLICA
Installed YES: package RIGID
Installed YES: package SHOCK
Installed YES: package SNAP
Installed YES: package SRD
Installed  NO: package VORONOI
Installed YES: package XTC
.........

4b. Choose all the standard

# make yes-standard

4c. Exclude packages that are not required

# make no-voronoi
# make no-kim
# make no-gpu
# make no-kokkos

Step 5: Install User-Contributed Intel Optimised Package (user-intel) and User-Contributed OpenMP Packages (user-omp)

# make yes-user-intel
# make yes-user-omp

Step 6: Compile LAMMPS Check make options. Go to src

# make intel_cpu_intelmpi -j 16

Step 6a: If in the midst of compilation you land yourself in this error ld: unable to to locate -lompstub. This is due to Intel 2015 using depreciating ompstub to ompistub5. Just go to the Intel Directory

# cd /usr/local/intel_2015/composerxe/lib/intel64
# ln -s  libiompstubs5.so libompstub.so

Step 7: Create a /usr/local/lammps-7Dec15 and copy libraries. Go to lammps root directory

# cp -Rv bench /usr/local/lammps-14May16
# cp -Rv doc /usr/local/lammps-14May16
# cp -Rv examples /usr/local/lammps-14May16
# cp -Rv potentials /usr/local/lammps-14May16
# cp README /usr/local/lammps-14May16
# cp -Rv tools /usr/local/lammps-14May16
# cp -Rv lib /usr/local/lammps-14May16
# cp lmp_intel_cpu_intelmpi /usr/local/lammps-14May16/bin

7a. Create a softlinks

# ln -s /usr/local/lammps-7Dec15/bin/lmp_intel_cpu_intelmpi lammps

References:

1. Quick install guide for LAMMPS on Linux cluster
2. LAMMPS Optimized for Intel on Quad Socket Xeon

New features and Enhancements of NetApp OnTap 9

Primary ONTAP Features

1. Leading enterprise-class data services • Unified management across flash, disk, and cloud, as well as SAN and NAS protocols
2. High performance for all-flash solutions • Nondisruptive operations
3. Scaling up to 24-nodes
4. Storage efficiency and inline data reduction 
5. NetApp Integrated Data Protection 
6. NetApp OnCommand® management software

 New Features: ONTAP 9 

1. Simplified AFF deployments in <10 li="" microsoft="" minutes="" racle="" server="" sql="" vdi="" vmware="">
2. Up to 60% more IOPS than with ONTAP 8.3.1 
3. ONTAP Select (for SDS) and ONTAP Cloud 
4. Support for 15TB SSD 
5. Inline data compaction works with compression and deduplication for greater data reduction
6. SnapLock® software for data compliance 
7. RAID-TEC triple-parity protection 
8. Headroom for visibility of performance capacity 
9. MetroCluster enhancements (lower TCO,  8-node NAS configurations) 
10. Onboard key manager 
11. Inline deduplication and compression 
12. MetroCluster 2-node solutions and improved distance (300km), performance, and deployment
13. Storage virtual machine disaster recovery  
14. Foreign LUN import enhancements

References:

1. NetApp ONTAP 9 Datasheet

GPFS Nodes being Expelled by Failed GPFS Clients

According to IBm Developer Wiki Troubleshooting Debug Expel  

• Disk Lease Expiration - GPFS uses a mechanism referred to as a disk lease to prevent file system data corruption by a failing node. A disk lease grants a node the right to submit IO to a file system. File system disk leases are managed by the Cluster Manager of the file system's home cluster. A node must periodically renew it's disk lease with the Cluster Manager to maintain it's right to submit IO to the file system. When a node fails to renew a disk lease with the Cluster Manager, the Cluster Manager marks the node as failed, revokes the node's right to submit IO to the file system, expels the node from the cluster, and initiates recovery processing for the failed node.
• Node Expel Request - GPFS uses a mechanism referred to as a node expel request to prevent file system resource deadlocks. Nodes in the cluster require reliable communication amongst themselves to coordinate sharing of file system resources. If a node fails while owning a file system resource, a deadlock may ensue. If a node in the cluster detects that another node owing a shared file system resource may have failed, the node will send a message to the file system Cluster Manger requesting the failed node to be expelled from the cluster to prevent a shared file system resource deadlock. When the Cluster Manager receives a node expel request, it determines which of the two nodes should be expelled from the cluster and takes similar action as described for the Disk Lease expiration.

But in my case, I have an errant failed GPFS Client node in the network which was within the cluster. All the other legitimiate GPFS Client was trying to expel this failed node, but got expel instead. The errant one remain, while the legitimate was expelled. The only solution was to power off this errant and the entire GPFS File System became operational. Here is an except in the Log File of the NSD Nodes. in fact, a lots of hints are found on the /var/adm/ras/mmfs.log.latest of any NSD Nodes. You should be able to locate it there.
  

Fri May 27 16:34:53.249 2016: Expel 172.16.20.5 (goldsvr1) request from 192.168.104.34 (compute186). Expelling: 192.168.104.34 (compute186)
Fri May 27 16:34:53.259 2016: Recovering nodes: 192.168.104.34
Fri May 27 16:34:53.311 2016: Recovered 1 nodes for file system gpfs3.
Fri May 27 16:34:55.636 2016: Accepted and connected to 10.0.104.34 compute186 <c0n135>
Fri May 27 16:39:13.333 2016: Expel 172.16.20.5 (goldsvr1) request from 192.168.104.45 (compute197). Expelling: 192.168.104.45 (compute197)
Fri May 27 16:39:13.334 2016: VERBS RDMA closed connection to 192.168.104.45 compute197 on mlx4_0 port 1
Fri May 27 16:39:13.344 2016: Recovering nodes: 192.168.104.45
Fri May 27 16:39:13.393 2016: Recovered 1 nodes for file system gpfs3.
Fri May 27 16:39:15.725 2016: Accepted and connected to 10.0.104.45 compute197 <c0n141>
Fri May 27 16:40:18.570 2016: VERBS RDMA accepted and connected to 192.168.104.45 on mlx4_0 port 1

Checking which kernel driver in use for CUDA.

This is useful if you are diagnosing the driver that is being used by your Nvidia Card. This was performed on CentOS 6.7

List created "NVIDIA devices":

# ls /dev/nv*
/dev/nvram 

Check the supported NVIDIA card:

[root@comp1 ~]# lspci -d "10de:*" -v
84:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40m] (rev a1)
Subsystem: NVIDIA Corporation Device 097e
Flags: bus master, fast devsel, latency 0, IRQ 64
Memory at c9000000 (32-bit, non-prefetchable) [size=16M]
Memory at 3c400000000 (64-bit, prefetchable) [size=16G]
Memory at 3c3fe000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel driver in use: nvidia
Kernel modules: nvidia, nouveau, nvidiafb