Using Intel IMB-MPI1 to check Fabrics and expected performances

In your .bashrc, do source the

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64
MKLROOT=/usr/local/intel_2015/mkl

To simulate 3 workloads pingpong, sendrecv, and exchange with IMB-MPT1

$ mpirun -r ssh -RDMA -n 512 -env I_MPI_DEBUG 5 IMB-MPT1

Using Vmstat for Profiling

Taken from RHEL Performance Tuning Student Booklet The vmstat command, if given no arguments, will print out the averages of various system statistics since boot. The vmstat command accepts two arguments. The first is the delay, and the second is the count. The delay is a value in seconds between output. The count is the number of iterations of statistics to report.

[root@lime ~]# vmstat 4 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0   2832 145216 194528 6772048    0    0     0     1    0    1  0  0 100  0  0
0  0   2832 145200 194528 6772048    0    0     0     1   44   45  0  0 100  0  0
0  0   2832 145208 194528 6772048    0    0     0     8   46   45  0  0 100  0  0
1  0   2832 145208 194528 6772048    0    0     0     0   44   46  0  0 100  0  0
0  0   2832 145208 194528 6772048    0    0     0     3   46   55  0  0 100  0  0

Category	Statistics	Definition
Process related	r	The number of processes waiting for runtime
Process related	b	The number of processes in uninterruptible sleep
memory	swpd	The amount of memory currently used in swap spaces
memory	free	The amount of idle (immediately available) memory
memory	buff	The amount of memory used as buffers
memory	cache	The amount of memory used as cache
swap: paging statistics	si	Pages of memory swapped in per second
swap: paging statistics	so	Pages of memory swapped out per second
io: block I/O statistics	bi	Blocks per second received from block devices
io: block I/O statistics	bo	Blocks per second sent to block devices
system	in	Interrupts raised per second
system	cs	Context switches per second
cpu: how CPU time is used	us	Percentage of time spent running user-space code
cpu: how CPU time is used	sy	Percentage of time spent running kernel code
cpu: how CPU time is used	id	Percentage of time spent idle
cpu: how CPU time is used	wa	Percentage of time spent blocked while waiting for I/O to complete
cpu: how CPU time is used	st	Percentage of time where the CPU had a process ready to run, but CPU time was stolen by the hypervisor supporting this virtual machine (typically because the CPU is being used by another guest virtual machine)

Basic Overview and use of NMON on CentOS 5

nmon for Linux – Nigel’s performance Monitor for Linux is a wonderful Swiss Army Knife for Performance Information.You can display multiple screen on the same windows and get information on CPU, Memory, NFS, Network, Disks, Resource, kernel etc

For more information, do look at Basic Overview and use of NMON on CentOS 5 from Linux Cluster

Avoiding DNS Lookup for Apache 2

If you wish to avoid situations where you do not wish to do DNS lookup for the client machines which will slow Apache performance. To do it is quite quick by setting the HostNameLookups directive to off at /etc/httpd/conf/httpd.conf

HostNameLookups Off

Switching off CPU Throttling on CentOS or Fedora

Under CentOS and Fedora, you can switch off CPU Throttling or "Dynamic Frequency Scaling" to maximise your CPU performance. For more information of CPU Throttling, you can also read Dynamic Frequency Scaling from Wikipedia. Just type the command

# /usr/bin/cpufreq-selector -g performance

For Debian-based hardware, you may want to take a look at
Looking at Xorg High CPU Usage Issue for netbook

Testing the Infiniband Interconnect Performance with Intel MPI Benchmark

This writeup focuses on verifying the performance of the Infiniband Interconnects or RDMA/iWARP Interconnects with Intel MPI Benchmark. For more information, do look at my Linux Cluster Blog

1. Testing the Infiniband Interconnect Performance with Intel MPI Benchmark (Part I)
2. Testing the Infiniband Interconnect Performance with Intel MPI Benchmark (Part II) 

Article on hdparm

An  interesting article Speeding up Linux Using hdparm by Rob Flickenger

GPFS Tuning Parameters

GPFS Tuning Parameters is a good wiki information resource written by IBM for GPFS Tuning. Just parroting some of the useful tips I have learned

To view the configuration parameters that has been changed from the default

mmlsconfig

To view the active value of any of these parameters you can run

mmfsadm dump config

To change any of these parameters use mmchconfig. For example to change the pagepool setting on all nodes.

mmchconfig pagepool=256M

1. Consideration to modify the PagePool

A. Sequential I/O
The default pagepool size may be sufficient for sequential IO workloads, however, a recommended value of 256MB is known to work well in many cases. To change the pagepool size

mmchconfig pagepool=256M [-i]

If the file system blocksize is larger than the default (256K), the pagepool size should be scaled accordingly. For example, if 1M blocksize is used, the default 64M pagepool should be increased by 4 times to 256M. This allows the same number of buffers to be cached.


B. Random I/O
The default pagepool size will likely not be sufficient for Random IO or workloads involving a large number of small files. In some cases allocating 4GB, 8GB or more memory can improve workload performance.

mmchconfig pagepool=4000M

C. Random Direct IO
For database applications that use Direct IO, the pagepool is not used for any user data. It's main purpose in this case is for system metadata and caching the indirect blocks of the database files.



D. NSD Server
Assuming no applications or Filesystem Manager services are running on the NSD servers, the pagepool is only used transiently by the NSD worker threads to gather data from client nodes and write the data to disk. The NSD server does not cache any of the data. Each NSD worker just needs one pagepool buffer per operation, and the buffer can be potentially as large as the largest filesystem blocksize that the disks belong to. With the default NSD configuration, there will be 3 NSD worker threads per LUN (nsdThreadsPerDisk) that the node services. So the amount of memory needed in the pagepool will be 3*#LUNS*maxBlockSize. The target amount of space in the pagepool for NSD workers is controlled by nsdBufSpace which defaults to 30%. So the pagepool should be large enough so that 30% of it has enough buffers.




For more information

1. GPFS Tuning Parameters
2. mmchconfig Command

NFS share on Linux client not immediately visible to other NFS clients

If you are using NFS as the shared file system, you may encounter this issue where NFS share on Linux client not immediately visible to other NFS clients. This is due to caching parameters which you must take note of on the NFS Client side. These are

1. acregmin=n. The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 3 seconds.
2. acregmax=n. The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 60.
3. acdirmin=n. The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
4. acdirmax=n. The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
5. actimeo=n. When you wish to sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.

For more information, see

1. Configuring NFS Client for Performance
2. Why are changes made on an NFS share on my Red Hat Enterprise Linux 5 client not immediately visible to other NFS clients?

Phoronix Test Suite

The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available for the Linux operating system. This software is designed to effectively carry out both qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. The Phoronix Test Suite consists of a lightweight processing core (pts-core) with each benchmark consisting of an XML-based profile with related resource scripts. Some of the key features are

• 120+ Test Profiles
• 50+ Test Suites
• Extensible (XML-based) Testing Architecture
• Optional Linux-based LiveDVD/USB Testing Distribution (PTS Desktop Live)
• Automated Test Installation
• Dependency Management Support
• Module-based Plug-In Architecture
• Integrated Results Viewer
• PNG, JPG, Adobe SWF, SVG Graph Rendering Support
• Autonomous Batch Mode Support
• Global Database For Result Uploads, Benchmark Comparisons
• HTML Documentation Covering Test Profiles, Module Framework
• Installed Software, Hardware Detection
• System Monitoring Support
• GTK2 Graphical User Interface + Command-Line Interface
• Runs On Linux, OpenSolaris, Mac OS X, & BSD Operating Systems

Linux Network Bonding or Trunking on CentOS 5.x

Finally, I have tested and setup a Linux Network Trunking. You will be very surprise it is very straightforward. See my blog "Linux Network Bonding or Trunking on CentOS 5.x"

Network design consideration for NFS

Network Design are an important consideration for NFS. Often we miss the infrastructure design consideration of NFS. Read further at Network design consideration for NFS

Configuring NFS Server for Performance

Finally, a more detailed writeup on NFS configuration which I was hoping to blog. Finally it is done.
Do read Configuring NFS Server for Performance

Configure TCP for faster connections and transfers

On a default Linux Box, the TCP settings may not be optimise for “bigger” available network bandwidth connections and transfer available for 100MB+. Currently, most TCP settings are optimise for 10MB settings. For more information, on how to configure, read my Linux Cluster Blog Configure TCP for faster connections and transfers

Dstat a versatile resource statistics tool

Taken from DAG

Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).

Some of the features includes:

• Combines vmstat, iostat, ifstat, netstat information and more
• Shows stats in exactly the same timeframe
• Enable/order counters as they make most sense during analysis/troubleshooting
• Modular design
• Written in python so easily extendable for the task at hand
• Easy to extend, add your own counters (please contribute those)
• Includes about 10 external plugins to show how easy it is to add counters
• Can summarize grouped block/network devices and give total numbers
• Can show interrupts per device
• Very accurate timeframes, no timeshifts when system is stressed
• Shows exact units and limits conversion mistakes
• Indicate different units with different colors
• Show intermediate results when delay > 1
• Allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs

Make sure you have rpmforge repository installed.

# yum install dstat

Some of the useful commands are as followed:

# dstat --help 

# dstat --full 

# dstat --vmstat 

# dstat --mem 

# dstat --load

Looking at Xorg High CPU Usage Issue for netbook

If you are suffering from excessive slowness from your netbook or your old and slow notebook, you may want to tweak the CPU frequency governor to performance mode instead of the general mode. How do you do that? If you are using Linux Mint, you have to install a utilities called cpufrequtils

Step 1: Install CPU Frequency Utilities
# apt-get install cpufrequtils

Step 2: Query the CPU frequency governors
# cpfreq-info

Step 3: Force the CPU to use performance mode
# cpufreq-set -g performance

If this does not resolve the issues, you really have to look at other possibilities that is impacting your Xorg. Sometimes it could be your video drivers not the CPU governor issues....

Speeding up boot time by Optimising Physical Memory and Swap

If you have a lot of physical memory, you may want to reduce your swappiness to increase performance. The performance is boosted as the kernel can cache data in memory for faster access and reduce the amount of data being swapped in and out of the swap space.

Alternatively, if you have very little memory (For example: 256MB or below), you may want to increase swappiness to boost performance.

To change the swappiness, You have to edit the /etc/sysctl.conf. /etc/sysctl.conf is a file tocontain sysctl which configure kernel parameters at runtime.

# vim /etc/sysctl.conf

At the bottom of the file /etc/sysctl.conf, add the following parameters
---------------------------------
vm.swappiness=5
---------------------------

Reboot and you will notice the difference. You may want to use conky to monitor the memory/swap usages

Speeding up boot time by running boot processes in parallel for Linux Mint

If you are using Linux Mint and have multi-core computer at home and wishes to speed up boot time by running processes in parallel, you can tweaked Grub to take advantage of parallelism.

Step 1: Edit /etc/init.d/rc and change CONCURRENCY from none to shell
# vim /etc/init.d/rc
---------------------------
CONCURRENCY=shell
---------------------------

You should notice a performance improvement in your boot-time.

Remove the timeout during boot time

When you boot, there is a small count down before continuing with the boot process. This was originally introduced to ensure that older hardware load modules in time for the kernel to boot. Nowsaday, it is used for users to select their kernel modules, to dual boot etc.

However, if you have a modern machine and you wish to boot fast, you can edit
# /boot/grub/menu.lst

----------------------------------
timeout 0                                    
----------------------------------
(Change timeout from 5 to 0)

You just save 5 seconds from your boot time

Tuning LINUX Toolbox

Finding Bottlenecks

Vmstat
Columns Information
(r,b) - How many process can be run if a CPU is available and how many are blocked.
(swpd, free, buff, cache) - Show how memory space is used.
(si, so) - Page-In and Page-Out
(io, bi, bo) - Number of Blocks received and sent to block devices
(in, cs) - Number of Interupts and context switches
(us, sy, id, wa) - Indicate percentage of time the CPU(s) has spent in userspace applications, in the kernel, being idle....

# vmstat 5 10 (To run vmstat with ten updates, five seconds )

Articles to consider:

1. Monitoring Virtual Memory with vmstat by LINUX Journal
2. Understanding the Linux Virtual Memory Manager by Mel Gorman
3. Is swap space obsolete? by Martin Pool

Disk Performance
Hdparm is a good tool to determine whether the disks are healthy and configured
# hdparm -tT /dev/sda

Article to consider:

1. Speeding up Linux Using hdparm by O'Reilly

# iostat -x sda 1
(%iowait) - High means CPU is idele and waiting for outstanding disk I/O requests.
(avgqu-sz) - Value should be less than 1
(%util) - pecentage of time the disk has requests.

To install on CentOS, Fedora, "yum install sysstat", iostat will be installed with the packages.

Articles to consider:

1. Sysstat Utilities Home Page