Linux Toolkits

Using qmove to move a batch job for Torque or OpenPBS

2012-01-27T00:56:00.002+08:00

If you wish to remove a job from the queue in which it resides and instantiate the job in another queue, you should use the qmove commands. The commands is as simple as

# qmove queue_name job_id

For example

# qmove pulasan_queue 1234

For more information, remember to look at qmove man pages

Commonly used qstat options

2012-01-24T00:58:00.000+08:00

Commonly used Qstat Options

Options	Description
qstat -i	Display jobs that are non-running in alternative format
qstat -r	Display jobs that are running
qstat -n	In addition to basic information, it also provide information of nodes allocated to the job listed.
qstat -u users(s)	Display jobs of a user or users
qstat -Q	Status of queues
qstat -Q -f	Full status of queues in the alternative format
qstat -q	Status of queues in the alternative format
qstat -B	Batch server status
qstat -B -f	Full batch server status including configuration

This information can also be found at Commonly used qstat options

Using type to check how each command is interpreted

2012-01-23T15:53:00.002+08:00

If you are hoping to check how each command is interpreted, then use the BASH BuiltIn command TYPE

To print all the places that contain an executable name you are looking for, use "type -a", you will bring out the alias and the path of the icc. In this case, there is no aliase so the alias did not surface up in the command line

$ type -a icc
icc is /opt/intel/composerxe-2011.2.137/bin/intel64/icc

To check the type of file the executable is categorised ie alias, keyword, function, builtin, file,you can use the command

$ type -t icc
file

For more usage of type, see the man page.

High Performance Data Transfers on TCP/IP

2012-01-23T00:30:00.000+08:00

This writeup is a summary of the excellent article from Pittburgh Supercomputing Centre “Enabling High Performance Data Transfers“

For more information, see High Performance Data Transfers on TCP/IP

Speeding up multiple ssh connections with ControlMaster

2012-01-21T01:20:00.003+08:00

I'm assuming you are using OpenSSH 4.

If you are make multiple connections to the same server, you can enables the sharing of multiple sessions over a single network connections. In other words, the additional sessions will try to reuse the master instance's connection rather than initiating new ones.

Step 1: Create a config file in your ~/.ssh directory. Make sure the permission is readable and writable by the owner only ie permission of 600

Step 2: Add the following lines

Host *
   ControlMaster auto
   ControlPath ~/.ssh/master-%r@%h:%p

ControlMaster auto Tries to start a master if there is no existing connection or it will use an existing master connection.ControlPath is the location socket for the ssh processes to communicate among themselves. The %r, %h and %p are replaced with your user name, the host to which you're connecting and the port number—only ssh sessions from the same user to the same host on the same port can or should share a TCP connection, so each group of multiplexed ssh processes needs a separate socket.

Step 3a: To test the configuration, start an ssh session and keep it connected, you should see something like

...........
debug1: setting up multiplex master socket
debug1: channel 0: new [client-session]
...........

Step 3b: Launch another ssh connection to a the same server with the same userid

....................
debug1: auto-mux: Trying existing master
...................

Much of the materials come from Speed Up Multiple SSH Connections to the Same Server (Linux Journal).

Tuning TCP for Linux 2.4 and Linux 2.6

2012-01-20T10:03:00.000+08:00

I have read an interesting article on TCP Tuning for the Linux 2.4 and 2.6 from (Advanced Networking, Pitsburgh Supercomputing Centre). From the article

"NB: Recent versions of Linux (version 2.6.17 and later) have fullautotuning with 4 MB maximum buffer sizes. Except in some rare cases, manualtuning is unlikely to substantially improve the performance of these kernelsover most network paths, and is not generally recommended

All system parameters can be read or set by accessing special files in the /proc file system. E.g.:

cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf

If the parameter tcp_moderate_rcvbuf is present and has value 1 then autotuning is in effect. With autotuning, the receiver buffer size (and TCP window size) is dynamically updated (autotuned) for each connection. (Sender side autotuning has been present and unconditionally enabled for many years now)."

Displaying defaults for useradd

2012-01-19T00:07:00.000+08:00

If you are take a look at the defaults that useradd will use for new logins, you can use the command

# useradd -D

Output

GROUP=100
HOME=/home
INACTIVE=-1
EXPIRE=
SHELL=/bin/bash
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes

Installing Adobe Flash Plugin on CentOS 5 and 6

2012-01-18T22:36:00.002+08:00

Step 1: Install Yum Repository from Adobe

(i) For i386

# rpm -ivh http://linuxdownload.adobe.com
/adobe-release/adobe-release-i386-1.0-1.noarch.rpm

# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

(ii) For x86-64

# rpm -ivh http://linuxdownload.adobe.com/
adobe-release/adobe-release-x86_64-1.0-1.noarch.rpm

# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

Step 2: Update the repositories

# yum update

Step 3: Installing Adobe Player -

CentOS 5

#  yum install flash-plugin nspluginwrapper curl

CentOS 6

# yum install flash-plugin nspluginwrapper alsa-plugins-pulseaudio libcurl

Lowest Latency Switch in the World ?

2012-01-17T22:52:00.000+08:00

Gnodal GS-Series is registering an impressive record for ulta-low latency switches. Their enterprise switch GS7200 has the following records. The URL can be found at GS7200 Features and Benefits

High Port Count 72 SFP+ port, 1U top-of-rack Ethernet switch enables network consolidation at minimal cost
Ultra-low Latency Sub 150ns port-to-port with extremely low jitter, ideal for latency sensitive applications
High Throughput 1.5 Tb/s non-blocking throughput and forwarding capacity exceeding 1 Bpps. Ideal for bandwidth hungry, data-intensive applications in High Performance Data Centers
Minimized TCO Typical power consumption of 1.6W per port substantially reduces operating costs. Reliability and serviceability are ensured with hot-swappable, redundant PSUs and fans

Installing Grace (xmgrace) on CentOS 5 and 6

2012-01-14T09:37:00.000+08:00

For further information on what is Grace ( xmgrace ) and some notes during installation. Do read the blog entry

For installing in a nutshell on CentOS 5 and CentOS 6.

./configure --enable-grace-home=/opt/grace \
--with-extra-incpath=/usr/local/include:/opt/include \
--with-extra-ldpath=/usr/local/lib:/opt/lib
--prefix=/usr/local

--enable-grace-home=DIR      define Grace home dir [PREFIX/grace]

--with-extra-incpath=PATH    define extra include path (dir1:dir2:...) [none]

--with-extra-ldpath=PATH     define extra ld path (dir1:dir2:...) [none]

Compiling,

make

Testing

make tests

Installation

make install

Making links

make links

Adaptive Computing Engine Frame and Moab Demo

2012-01-10T10:06:00.003+08:00

Adaptive Computing Engine Frame and Moab Demo (YouTube (unlisted))

BatchJob Submission and Session Management http://www.youtube.com/watch?v=i3Z5_Tz9Llk
Multiple Desktop Request http://www.youtube.com/watch?v=HUW1A7zwOW
Multi-Userand Multi-Session Use http://www.youtube.com/watch?v=glUgAhmhXp8
SessionSharing http://www.youtube.com/watch?v=Cvcpgv71qYo
DirectApp Launch http://www.youtube.com/watch?v=WzKwlXTJ-WI
DCVDemonstration, Multi-User Graphic Assignment http://www.youtube.com/watch?v=c1t9BBPtRoU

Module - Software Environment Management

2012-01-10T09:38:00.000+08:00

For Academic Environment, this tool is especially useful to dynamically modified user environment via modulefiles

A good description of the module tool can be taken from the Sourceforge - Environment Modules Project.

"The Environment Modules package provides for the dynamic modification of a user's environment via modulefiles.

Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as PATH, MANPATH, etc. modulefiles may be shared by many users on a system and users may have their own collection to supplement or replace the shared modulefiles.

Modules can be loaded and unloaded dynamically and atomically, in an clean fashion. All popular shells are supported, including bash, ksh, zsh, sh, csh, tcsh, as well as some scripting languages such as perl.

Modules are useful in managing different versions of applications. Modules can also be bundled into metamodules that will load an entire suite of different applications."
More can be written on this.

The Modules Environment at the ITP Cluster

Understand parameters in /etc/fstab

2012-01-07T11:36:00.000+08:00

Do look at the article form "Understand parameters in /etc/fstab" (linuxcluster.wordpress.com). Just a more indepth understanding of /etc/fstab parameters

Using qmgr to remove a queue attribute

2012-01-06T00:35:00.002+08:00

To clear the queue attribute in qmgr, you should use the command "unset". For example, you wish to clear acl_users attribute

# qmgr -c "unset queue default_queue acl_users"

where -c = execute single command and exit qmgr.

You will notice that the attribute has been removed by verifying with this command

# qmgr -c "p s"

Using shutdown -h command to do a scheduled shutdown

2012-01-03T22:59:00.000+08:00

If you wish to use the Linux command to remind users of scheduled maintenance and halts the system at 10:00am, you can use the command

# shutdown -h 10:00 "Scheduled Maintenance. Expected downtime at 10am"

Alternatively, you may wish to specify a relative downtime. If you wish to shutdown the system 20min when the command is executed

# shutdown -h +20 "We need to power down to do an emergency repair"

How to disable SSLv2 and Weak Cipers and enable SSLv3 on Linux

2011-12-31T01:14:00.005+08:00

In order to be Payment Card Industry Data Security Standard PCI-DSS) Compliance v1.2, we are required to use “use strong cryptography and security protocols such as SSL/TLS or IPSEC to safeguard sensitive cardholder data during transmission over open, public networks.”

Secure Socket Layer (SSL) version 2 is considered weak cryptography in this aspect. To disabled SSLv2 and enable SSLv3. Assuming you already have OpenSSL installed, you can use another remote server to test the https connections

# openssl s_client -ssl2 -connect remote_server:443

If your server does not support SSLv2, you should receive the following error

CONNECTED(00000003)
22255:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:

If your server is enabled to supports SSLv2 connections, the connection will be accepting input

CONNECTED(00000003)

To use SSLv3 and TLSv1, you have to modify the following at SSLCipherSuite directive in the httpd.conf or /etc/httpd/conf.d/ssl.conf file. In the example, you can do the following

#SSLProtocol all -SSLv2
SSLProtocol -ALL +SSLv3 +TLSv1

On my /etc/httpd/conf.d/ssl.conf

SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

For more information,see

How to Disable SSLv2 and Weak Ciphers(PCI Compliance (http://almamunbd.blogspot.com)
How to Disable SSLv2 and Weak Ciphers(PCI Compliance (http://www.srcnix.com)

Important Apache (httpd) security Update

2011-12-30T16:40:00.000+08:00

An important security update for httpd and solution for

Description of the bugs can be found at CVE-2011-3192

The byterange filter in the Apache HTTP Server 1.3.x, 2.0.x through 2.0.64, and 2.2.x through 2.2.19 allows remote attackers to cause a denial of service (memory and CPU consumption) via a Range header that expresses multiple overlapping ranges, as exploited in the wild in August 2011, a different vulnerability than CVE-2007-0086.

Solution:

# yum update httpd

Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down

2011-12-19T08:55:00.000+08:00

If the post Encountering eth0 NIC SerDES Link is Down did not resolve your issue and you are still encountering "eth0 NIC SerDES Link" issues, do upgrade the Broadcom Drivers from your vendor site and it will eliminate your issue immediately. Since my vendor is IBM, so I downloaded the Broadcom BNX2 Drivers
Broadcom BNX2 driver version bnx2-2.0.23b for RHEL 5 - IBM System x and BladeCenter

If you are not sure what is your version of drivers, you can do a

# ethtool -i eth0

The version 2.0.8 and above should resolve the above issue

Oh yes, if you are using IBM Products and the above drivers from IBM, after unpacking the drivers from IBM and ensuring you have the necessary prerequistics, just do a

If you are using the Free Clone of Redhat which includes CentOS or Scientific Linux, you may want to temporarily modify the /etc/redhat-release information to simulate a real RHEL Distribution. Vendor patches often requires RHEL distribution

#CentOS release 5.4 (Final)
Red Hat Enterprise Linux AS release 5

# mkdir brcm
# cd brcm
# tar -zxvf brcm_dd_nic_netxtreme2-2.0.23b_1.62.15_rhel5_32-64.tgz
# ./install.pl --update

 INSTALL_OPTIONS --yes --update


        Drivers will be installed/migrated to 2.6.18-164 version

----------------------------------------------------------------------
Checking kmod-brcm-netxtreme2-6.2.23-1.x86_64.rpm
WARNING: Non Whitelist symbol detected
----------------------------------------------------------------------
kmod-brcm-netxtreme2-6.2.23-1.x86_64.rpm installed successfully
SUCCESS

sys_copy and scp -rpb error captured on pbs_mom logs

2011-12-17T00:31:00.000+08:00

I was encountering an interesting scp error on my log file regarding pbs_mom

.......pbs_mom: LOG_ERROR::sys_copy, 
command '/usr/bin/scp -rpB  2014.Head-Node.OU 
userid@headnode:/home/xxx' failed with status=1, 
giving up after 4 attempts

It seems that the error may be due to default MaxStartups 10 setting in the /etc/ssh/sshd_config which is too low a value and scp may be overwhelm

According to manual page
MaxStartups - Specifies the maximum number of concurrent unauthenticated connections to the sshd daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime
expires for a connection. The default is 10.

Try increasing the MaxStartups to 100 at /etc/ssh/sshd_config

MaxStartups 100

Blade hangs on boot and "FW/BIOS, firmware progress (ABR Status) FW/BIOS ROM corruption

2011-12-16T00:52:00.001+08:00

2 of my Blade got hang on boot and suffered this "FW/BIOS, firmware progress (ABR Status) FW/BIOS ROM corruption". For more information on the resolution, do look at Blade hangs on boot and "FW/BIOS, firmware progress (ABR Status) FW/BIOS ROM corruption" message in AMM - IBM BladeCenter HS22, HS22V

From the site,

Symptom
When booting BladeCenter HS22 or HS22V with Integrated Management Module (IMM) build yuoo84c installed, the blade may hang at the "UEFI Platform Initializing" screen. The hang will be accompanied by the following event in the chassis Advanced Management Module (AMM) log:

FW/BIOS, firmware progress (ABR Status) FW/BIOS ROM corruption

Solution
This behavior is corrected in IMM firmware release yuoo91e and newer.
The file is or will be available by selecting the appropriate machine type on the 'Product View' of IBM Support's Fix Central web page, at the following URL:
http://www.ibm.com/support/fixcentral/systemx/groupView?query.productGroup=ibm%2FBladeCenter

Workaround
This failure may be reduced by disabling Internet Protocol Version 6 (IPv6) support for the IMM. This can be done via the following steps:
1. Boot the blade to the F1 Unified Extensible Firmware Interface (UEFI) setup screen.
2. Select "System Settings" and press Enter
3. Select "Integrated Management Module" and press Enter
4. Select "Network Configuration" and press Enter
5. Change "IP6" setting to "Disable"
Occasionally, the failure can be recovered by restarting the IMM. If this is not successful, then it is necessary to reseat the blade in the chassis to recover. After a reseat, the blade will boot normally.

Unable to edit fstab as it is a read only file during repair

2011-12-15T09:35:00.001+08:00

I unwittingly changed the label for a partition for the /etc/fstab and was was presented with bootup to bash.
When I tried to revert back to the correct label for the partition, the vi just could not save the newly edited settings, instead it will have the error message "Error writing fstab: Read-only file system"

To solve the issue, you have to remount

mount -n -o remount /

which work fine for me.

Or

mount -n -o remount -t ext2 /dev/hda2 /

Checking Torque Queue Attributes

2011-12-14T08:49:00.003+08:00

If you wish to check Queue Attributes fully, you can use the command

qstat -f -Q  queuename

The output will be

Queue: dqueue
    queue_type = Execution
    total_jobs = 0
    state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0
    resources_default.neednodes = starfruit
    mtime = 1323678795
    resources_assigned.nodect = 0
    enabled = True
    started = True

Using vim editor to find and replace effectively

2011-12-13T00:36:00.002+08:00

Taken from this excellent article Vi and Vim Editor: 12 Powerful Find and Replace Examples

Here is a few examples that I love to use. You can see that this entry is a notepad for me.

Scenario 1: Replace all occurrences of a text with another text in the whole file

:%s/old-text/new-text/g

%s - specifies all lines. Specifying the range as ‘%’ means do substitution in the entire file.
g flag– specifies all occurrences in the line. With the ‘g’ flag , you can make the whole line to be substituted. If this ‘g’ flag is not used then only first occurrence in the line only will be substituted.

Scenario 2: Replace of a text with another text within a range of lines

:1,10s/old-text/new-text/gi

1-10 - Do substitution from line 1 to 10
i flag - Make the substitute search text to be case insensitive.

Scenario 3: Replacing of a text with another text for a the 1st X number of lines
From the current position of the cursor, the command will replace according to the number of count. For example, do substitution in 10 lines from the current line.

:s/old-text/new-text/g 10

Scenario 4: Substitute only the whole word and not partial match
If you wish to change the whole word "text" to "new-text"

Original Text: old to text
:s/\<text\>/new-text/
Translated Text: old to new-text

How to associate compute nodes with a queue name with Torque

2011-12-11T15:39:00.000+08:00

If you wish to use a queue that is locked to a selected group of nodes and wish to allow certain users to run, you may want to take a look at one of the contributor to a Rock-Discussion
[Rocks-Discuss] [Torque roll] How to associate 10 compute nodeswith a queue name ?

In his write-up

========
qmgr -c "create queue vision queue_type=execution"
qmgr -c "set queue vision resources_default.neednodes = vision"
qmgr -c "set queue vision acl_hosts=c2-0-20+c2-0-21+c2-0-22+c2-0-27+c2-0-28+c2-0-29"
qmgr -c "set queue vision acl_host_enable = false"
qmgr -c "set queue vision acl_users=user1"
qmgr -c "set queue vision acl_users+=user2"
qmgr -c "set queue vision acl_users+=user3"
qmgr -c "set queue vision acl_user_enable=true"
qmgr -c "set queue vision enabled = True"
qmgr -c "set queue vision started = True"

qmgr -c "set queue default resources_default.neednodes = general"

for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 23 24 25 26 30 31 32 33; 
do 
        /opt/torque/bin/qmgr -c "set node c2-0-${i} properties = general"; 
done

for i in 20 21 22 27 28 29 ; 
do 
        /opt/torque/bin/qmgr -c "set node c2-0-${i} properties = vision"; 
done
========

For more information, do also read up on

4.1 Queue Configuration (From Cluster Resources)
Cluster Node-Locking with Torque and Maui (Wednesday, October 22, 2008)

Configuration error when compiling octave with BLAS and LAPACK libraries

2011-12-08T00:34:00.001+08:00

Do take a look at the Compiling Octave from Source on CentOS 5. However you make face an error such as
" configure: error: You are required to have BLAS and LAPACK libraries ".

This is due to the missing link. For more information on this error, you may want to take some hints from
Cannot find -llapack when doing /usr/bin/ld on CentOS 5

In other words, just go to /usr/lib64 and do a softlink for the lapack library

ln -s /usr/lib64/liblapack.so.3 /usr/lib64/liblapack.so

Encountering eth0 NIC SerDES Link is Down

2011-12-02T00:26:00.000+08:00

I was noticing this error on my HS22 Blade log files occasionally and on one occasion the NFS which was relying on the ethernet connection got disconnected and hang when the load is exceedingly high. The problem is that it is very hard to reproduce the problem as it is quite random

My Server is using the Broadcom chipset bnx2 and my version of my CentOS is 5.4 or kernel version is 2.6.18-164.el5

After a bit of searching, this particular Red Hat Bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=520888) reflects the problem and workaround very well. I encourage you to take a closer look. If you are not planning to upgrade your RHEL or CentOS to 5.6 ( http://rhn.redhat.com/errata/RHSA-2011-0017.html ) and above yet, you may want to consider the workaround as mentioned in the bugzilla

From Comments 14

"Configuring IRQ SMP affinity has no effect on some devices that use messagesignalled interrupts (MSI) with no MSI per-vector masking capability. Examplesof such devices include Broadcom NetXtreme Ethernet devices that use the bnx2driver.

If you need to configure IRQ affinity for such a device, disable MSI bycreating a file in /etc/modprobe.d/ containing the following line:

options bnx2 disable_msi=1

Alternatively, you can disable MSI completely using the kernel boot parameterpci=nomsi. (BZ#432451)

"http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Technical_Notes/Known_Issues-kernel.html

To Check whether you are still having issues, you can use the command

# dmesg |grep bnx2

I guess the best way is to update your broadcom drivers. For latest update on this "NIC SerDES Link is Down", see my writeup on Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down

Using stunnel to generate to create a self-signed certificate for SL 6 and CentOS 6

2011-12-01T00:57:00.000+08:00

The stunnel Program allows administrator to create self-signed certification using external OpenSSL Libraries included with RHEL and its clone to provide strong cryptography and protect connection. For more information on the installation and setup, see Using stunnel to generate to create a self-signed certificate for SL 6 and CentOS 6

pbs_mom LOG_ERROR sys_copy, command /usr/bin/scp -rpB

2011-11-30T00:29:00.001+08:00

I encountered 1 of my parallel job failed and this error appeared on the log file for my compute nodes.

pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB...............failed with status=1, 
giving up after 4 attempts

My SSH public/private key authentication is working without a hitch. Similarly, my /etc/hosts and firewall is as what I expected. But I realise my /etc/resolv.conf and /etc/sysconfig/network are incorrect. I got a hint of this possibility when I was reading this forum http://www.mail-archive.com/mauiusers@supercluster.org/msg00998.html . A quick amendment and everything seems ok at least for a while. Will write if this solution is incorrect. :)

Companies Comparison for Storage in the Gartner Ranking

2011-11-29T16:07:00.001+08:00

For information on companies selling storage boxes, do look at the article on
Ball-gazer casts magic runes to heal HP's credibility. Swallowing startups pushes up Gartner ranking.
NetApp and EMC are the leaders with NetApp ahead of EMC on the vision scale.

Installing Pylith using Pylith Installer

2011-11-29T00:19:00.001+08:00

PyLith is a finite element code for the solution of dynamic and quasi-static tectonic deformation problems.

This entry will only focus on the compilation of Pylith from the installer. Most if not all of the information comes from INSTALLER files after you untar the software.

For more information, see Installing Pylith using Pylith Installer

Repository of CentOS 6 and Scientific Linux 6

2011-11-28T16:45:00.001+08:00

Repository of CentOS 6 and Scientific Linux 6

EPEL (Extra Packages for Enterprise Linux)

$ wget http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
$ wget http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm

RPMForge (Repoforge)

$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm

PUIAS (Princeton University Institute of Advanced Study)

$ wget http://puias.math.ias.edu/data/puias/6/x86_64/os/Packages/puias-core-6-1.puias6.6.noarch.rpm

Cannot find -llapack when doing /usr/bin/ld on CentOS 5

2011-11-24T00:53:00.001+08:00

I encountered an error when one of our researchers did a compilation of a Fortran Program which requires blas and lapack

$ g77 test.f  -L/usr/lib64/ -llapack -lblas
/usr/bin/ld: cannot find -llapack
collect2: ld returned 1 exit status

I was quite puzzled as I have installed lapack and blas. And it seems that lapack is having issues

To check whether you have the libraries, you can use the command

$ ldconfig -p | grep llapack
 libscalapack.so.1 (libc6,x86-64) => /usr/lib64/libscalapack.so.1
 liblapack.so.3 (libc6,x86-64) => /usr/lib64/liblapack.so.3

So it is not an issue of missing lapack libraries. It is there.

           "On systems which support shared libraries, ld may also search for libraries with exten-
           sions other than ".a". Specifically, on ELF and SunOS systems, ld will search a direc-
           tory for a library with an extension of ".so" before searching for one with an exten-
           sion of ".a". By convention, a ".so" extension indicates a shared library.

           The linker will search an archive only once, at the location where it is specified on
           the command line. If the archive defines a symbol which was undefined in some object
           which appeared before the archive on the command line, the linker will include the
           appropriate file(s) from the archive.   However, an undefined symbol in an object
           appearing later on the command line will not cause the linker to search the archive
           again."

So just do a quick soft-links and the problem was solved

$ ln -s /usr/lib64/liblapack.so.3 /usr/lib64/liblapack.so

Unspecified GSS failure from SSH causes slow login

2011-11-23T18:40:00.000+08:00

I SSH into one of my server, But I encounter this error, but instead I encounter the follow error. Eventually, after waiting about 15-20 seconds, I'm able to connect to. This was far too long for a LAN-based machine

$ ssh -v ip_of_remote_server

.....
debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195
.....

I was quite puzzled. Although I am using IP address of the server to ssh and have tweaked "UseDNS = no" at /etc/sshd_config. See Resolving Slow SSH Login, In addition, I'm doing SSH public/private key authentication. See Auto SSH Login without Password

But the resolution for this issue was easier than I thought. I just need to ensure /etc/hosts contains both the servers I am ssh from and to and it became very quick.

If you are using DNS instead of /etc/hosts, do take a look that your DNS settings at /etc/resolv.conf

Other Issues on SSH, you may want to read about

List of Intel Xeon and AMD Microprocessors with pricing

2011-11-22T19:06:00.001+08:00

The listing Intel Xeon Microprocessors with pricing from Wikipedia in $USD is very useful for price comparison and budgeting. See Wikipedia List of Intel Xeon Microprocessors

Similarly the listing of AMD Microprocessors from Wikipedia is very informative. But sadly no price listing
List of AMD Opteron microprocessors

Brief overview of Valgrind usage

2011-11-21T00:50:00.001+08:00

This write-up covers some very basis commands. But I will try to list out some of the other collections of tutorial and reading to complement this lack of information. I'm assuming that you have compiled the program as written in Compiling Valgrind on CentOS 5One of the most commonly used command in Valgrind is

# valgrind --tool=memcheck --leak-check=full ./my_program

Commonly-used Options

S/No	Command Option	Description
1	--leak-check=<no\|summary\|yes\|full> [default: summary]	When enabled, search for memory leaks when the client program finishes. If set to summary, it says how many leaks occurred. If set to full or yes, it also gives details of each individual leak.
2	--show-reachable=<yes\|no> [default: no]	When disabled, the memory leak detector only shows "definitely lost" and "possibly lost" blocks. When enabled, the leak detector also shows "reachable" and "indirectly lost" blocks. (In other words, it shows all blocks, except suppressed ones)

For more information on more details usage of Valgrind of options and how to use,

Removing a node from the Ganglia Web Frontend

2011-11-16T19:10:00.001+08:00

According to the Ganglia_Readme, there is not easy way to remove a single dead node from the list from the ganglia web front-end. To flush the dead node from the record by restarting the the gmetad and gmond processes, you have to add the line at /etc/gmond.conf

globals { 
host_dmax = 3600 
}

The hosts will be removed from host tables when they haven't been heard from in 3600 seconds. See "man gmond.conf" for details.

Compiling Valgrind on CentOS 5

2011-11-13T01:07:00.001+08:00

Valgrind tools automatically detect many memory management and threading bugs, and is able to profile your programs in detail. It runs on the following platforms: X86/Linux, AMD64/Linux, ARM/Linux, PPC32/Linux, PPC64/Linux, S390X/Linux, ARM/Android (2.3.x), X86/Darwin and AMD64/Darwin (Mac OS X 10.6 and 10.7)According to Valgrind, a number of useful tools are supplied as standard.

Memcheck is a memory error detector. It helps you make your programs, particularly those written in C and C++, more correct.
Cachegrind is a cache and branch-prediction profiler. It helps you make your programs run faster.
Callgrind is a call-graph generating cache profiler. It has some overlap with Cachegrind, but also gathers some information that Cachegrind does not.
Helgrind is a thread error detector. It helps you make your multi-threaded programs more correct.
DRD is also a thread error detector. It is similar to Helgrind but uses different analysis techniques and so may find different problems.
Massif is a heap profiler. It helps you make your programs use less memory.
DHAT is a different kind of heap profiler. It helps you understand issues of block lifetimes, block utilisation, and layout inefficiencies.
SGcheck is an experimental tool that can detect overruns of stack and global arrays. Its functionality is complementary to that of Memcheck: SGcheck finds problems that Memcheck can't, and vice versa..
BBV is an experimental SimPoint basic block vector generator. It is useful to people doing computer architecture research and development.

Compilation of ValgrindCompilation is very straightforward......

# tar -xvjpf valgrind-3.7.0.tar.bz2
# cd valgrind-3.7.0
# ./configure --prefix=/usr/local/valgrind-3.7.0
# make; make install

Testing Valgrind

# /usr/local/valgrind-3.7.0/bin/valgrind ls -l

Either this works, or it bombs out with some complaint.

For more information, see Compiling Valgrind on CentOS 5

Compiling adaptive Poisson-Boltzmann Solver (APBS) on CentOS 5

2011-11-12T01:54:00.001+08:00

Adaptive Poisson-Boltzmann Solver (APBS) is a software package for modeling biomolecular solvation through solution of the Poisson-Boltzmann equation (PBE), one of the most popular continuum models for describing electrostatic interactions between molecular solutes in salty, aqueous media......

Installation is very simple. There are many binaries there and you can use the binaries directly. Do note that the latest binaries (apbs-1.3) uses will require glibc 2.7 and greater. If you are using CentOS 5, you may want to use apbs-1.21 binaries or below.

For details on Compiling adaptive Poisson-Boltzmann Solver (APBS) on CentOS 5 on Linux Cluster

Using strace as a troubleshooting tool

2011-11-08T00:24:00.003+08:00

Taken from Using strace as a troubleshooting tool (linuxcluster.wordpress.com) Strace, when runs in conjunction with a program do output all the calls made to the kernel by the program.

One of quick way to found out what is going on in your program is to do

$ strace -c ./my_hello_world_program

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
74.80    0.002998        1499         2           wait4
21.91    0.000878           4       221           read
0.95    0.000038           0       237         2 mmap
0.77    0.000031          10         3         1 mkdir
0.67    0.000027           0       566       361 open
0.35    0.000014           0        81           mprotect
0.30    0.000012           0        62        37 stat
0.25    0.000010           0       225           close
0.00    0.000000           0        37         1 write
0.00    0.000000           0       132           fstat
0.00    0.000000           0         8           poll
0.00    0.000000           0         2           lseek
0.00    0.000000           0       120           munmap
0.00    0.000000           0        15           brk
0.00    0.000000           0        16           rt_sigaction
................

................

------ ----------- ----------- --------- --------- ----------------
100.00    0.004008                  1990       411 total

If you wish to do a tracing, just do a, you can easily find out the error if there was....

$ strace ./my_hello_world_program

............

............

open("/tmp/openmpi-sessions-root@starfruit-h00.cluster.spms.ntu.edu.sg_0/25979/1/0",
O_RDONLY|O_NONBLOCK|O_DIRECTORY) = -1 ENOENT (No such file or directory)
munmap(0x2b46e05ef000, 2111200)         = 0
munmap(0x2b46dffe5000, 2102312)         = 0
munmap(0x2b46dfdde000, 2123264)         = 0
munmap(0x2b46e103f000, 2106960)         = 0
munmap(0x2b46e1242000, 2104560)         = 0
munmap(0x2b46e269d000, 2114912)         = 0
munmap(0x2b46e41c9000, 2145008)         = 0
munmap(0x2b46e43d5000, 2162608)         = 0

If you wish the output of strace to a file instead, do use the argument -o

$ strace -o strace_output_file ./my_hello_world_program

If you wish to trace system call, process,network, you can use the "-e trace=file", "-e trace=process", "-e trace=network",

$ strace -e trace=open,close,read,write ./my_hello_w0rld_program

$ strace -e trace=stat,chmod,unlink ./my_hello_world_program

Further Information:

Solutions for tracing UNIX applications (IBM DeveloperWorks)
strace - A very powerful troubleshooting tool for all Linux users (linuxhelp.blogspot.com)
Ten commands every linux developer should know (Linux Journal)

A*Star Computational Resource Centre Software Listings

2011-11-04T10:56:00.002+08:00

A*Star which is the main Government funded research organisation in Singapore has a highly effective A*STAR Computational Resource Centre (A*CRC) which provides high performance computational (HPC) resources to the entire A*STAR research community.

They have an interesting software listing which includes,

Biology and Bioinformatics
Chemistry and Molecular Modeling
Physics and Material Science
Mathematical, Statistical and Other Utilities
Software Development
System Software

NetApp posts world-record SPEC SFS2008 NFS benchmark result

2011-11-03T20:32:00.000+08:00

NetApp achieved over 1.5 million SPEC SFS2008 NFS operations per second with a 24-node cluster based on FAS6240 boxes running ONTAP 8 in Cluster Mode......For more information, see NetApp posts world-record SPEC SFS2008 NFS benchmark result

Basic Overview and use of NMON on CentOS 5

2011-11-02T00:43:00.000+08:00

nmon for Linux – Nigel’s performance Monitor for Linux is a wonderful Swiss Army Knife for Performance Information.You can display multiple screen on the same windows and get information on CPU, Memory, NFS, Network, Disks, Resource, kernel etc

For more information, do look at Basic Overview and use of NMON on CentOS 5 from Linux Cluster

Installing ALPS 2.0 from source on CentOS 5

2011-11-01T01:17:00.000+08:00

What is ALPS Project?

The ALPS project (Algorithms and Libraries for Physics Simulations) is an open source effort aiming at providing high-end simulation codes for strongly correlated quantum mechanical systems as well as C++ libraries for simplifying the development of such code. ALPS strives to increase software reuse in the physics community.Good information on installing ALPS can be found on ALPS Wiki's Download and install ALPS for Ubuntu 9.10, Ubuntu 10.04, Ubuntu 10.10, Debian and MacOS

Installing ALPS with Boost

# wget http://alps.comp-phys.org/static/software/releases/alps-2.0.2-r5790-src-with-boost.tar.gz

You will need either gfortran or Intel Fortran Compiler. If you are installing using gfortan

# yum install gcc-c++ gcc-gfortran

If you want to use the evaluation tools, you will need to install a newer version of Python than the provided 2.4. You can install from source or use an unofficial repository for binary RPMs. This is not required if you just want to run your compiled simulations (c++ applications), but make sure you still have python headers (specify -DALPS_BUILD_PYTHON=OFF when invoking cmake):

# yum install python-devel

BLAS/LAPACK is necessary. Make sure you have EPEL repository ready. For more information,Red Hat Enterprise Linux / CentOS Linux Enable EPEL (Extra Packages for Enterprise Linux) Repository

# yum install blas-devel lapack-devel

CMake 2.8.0 and HDF5 1.8 need to be installed. There is a wonderful scripts that comes with ALPS that help to compile CMAKE 2.8 and HDF5.1.8 with CentOS 5

$ $HOME/src/alps2/script/cmake.sh $HOME/opt $HOME/tmp
$ $HOME/src/alps2/script/hdf5.sh $HOME/opt $HOME/tmp

Build ALPS

Create a build directory (anywhere you have write access) and execute cmake giving the path to the alps and to the boost directory:

# cmake -D Boost_ROOT_DIR:PATH=/path/to/boost/directory /path/to/alps/directory

For example if the alps precompiled directory is in /root/alps-2.0.2# cmake -D Boost_ROOT_DIR:PATH=/root/alps-2.0.2/boost /root/alps-2.0.2/alpsTo install in another directory, set set the variable CMAKE_INSTALL_PREFIX

# cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/directory /path/to/alps/directory

For example:

# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/alps-2.0.2 /root/alps-2.0.2/alps

Build and test ALPS

$ make -j 8
$ make test
$ make install

* HDF5.1.8 binaries and libraries are very useful not only for compiling ALPS but other applications require HDF5.1.8. You may want to consider to move its binaries and libraries to the /usr/local/ directories

Set higher MTU for vSwitch and Virtual Distributed Switch

2011-10-31T17:09:00.000+08:00

Refer to KB: iSCSI and Jumbo Frames configuration on ESX 3.x and ESX 4.x for more details

**Any packet larger than 1500 MTU is a Jumbo Frame. ESX supports frames up to 9Kb (9000 Bytes).

To set the MTU size for the vSwitch, run the command:

# esxcfg-vswitch -m (MTU_Number) (Vswitch)

where MTU_Number = 9000, Vswitch = Name of the Vswitch

This command sets the MTU for all uplinks on that vSwitch. Set the MTU size to the largest MTU size among all the virtual network adapters connected to the vSwitch.

Refer to KB: Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch

Run this command to change the MTU size for the individual port group:

# esxcfg-vmknic -m 9000 -v (port number) -s (dvs Switch name)

For Example:

# esxcfg-vmknic -m 9000 -v 115 -s "NewLAN-DVS"

To enable Jumbo Frames on a VMkernel port from vCenter Server:

1) Click Home > Hosts and Clusters > Host > Configuration > Networking.
2) Navigate to the vSphere Distributed Switch tab.
3) Click the VMkernel port (eg: vmk1)
4) Click Manage Virtual Adapters.
5) Select the vmk interface and click Edit.
6) Under the NIC settings, change the MTU value to 9000.
7) Click OK.

Error in Compiling GotoBLAS2 in Westmere Chipsets

2011-10-29T01:44:00.001+08:00

GotoBLAS2 uses new algorithms and memory techniques for optimal performance of the BLAS routines.

When I was tried compiling the GotoBLAS2 on my Westmere chipsets, I followed the "02QuickInstall.txt", I got this error

../kernel/x86_64/gemm_ncopy_4.S: Assembler messages:
../kernel/x86_64/gemm_ncopy_4.S:192: Error: undefined symbol `RPREFETCHSIZE' in                                        operation
...........
...........
...........

gcc -O2 -Wall -m64 -DF_INTERFACE_INTEL -fPIC  -DSMP_SERVER -DMAX_CPU_NUMBER=8 -D                                       ASMNAME=strmm_kernel_RN -DASMFNAME=strmm_kernel_RN_ -DNAME=strmm_kernel_RN_ -DCN                                       AME=strmm_kernel_RN -DCHAR_NAME=\"strmm_kernel_RN_\" -DCHAR_CNAME=\"strmm_kernel                                       _RN\" -I.. -UDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -UDOUBLE -UCOMPLEX -ULEFT -UTRANS                                       A ../kernel/x86_64/gemm_kernel_8x4_sse3.S -o strmm_kernel_RN.o
make[1]: *** [sgemm_oncopy.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/root/GotoBLAS2/kernel'

I was quite puzzled to why the compilation did not work. I googled and found a wonderful answer Trouble compiling GotoBLAS2 on newer CPU. Basically, you will need to

gmake clean
gmake TARGET=NEHALEM

Eventually yo will get something like

 GotoBLAS build complete.

  OS               ... Linux
  Architecture     ... x86_64
  BINARY           ... 64bit
  C compiler       ... GCC  (command line : gcc)
  Fortran compiler ... INTEL  (command line : ifort)
  Library Name     ... libgoto2_nehalemp-r1.13.a (Multi threaded; Max num-threads is 8)

According to Trouble compiling GotoBLAS2 on newer CPU, the problem appears to be that newer CPUs (Intel X5650 in my case) arenot detected properly by the CPU ID routine in GotoBlas2.

The problem with gemm_ncopy_4.S arises because it definesRPRETCHSIZE and WPREFETCHSIZE using #ifdef statements dependingon CPU type. There is an entry for #ifdef GENERIC, but that was notset for me in config.h.

IBM DS4000 (FastT) Storage Manager Client on Windows

2011-10-28T21:13:00.000+08:00

Boy, this blog "How to Install IBM DS4000 (FastT) Storage Manager Client on Windows" is really helpful in getting information and the software on IBM DS4000 (FastT) Storage Manager Client. Even easier than finding on IBM Website.

......
1) Download the software.
For Windows XP, 2000, 2003, or 2008, on a 32-bit platform, click here.
For Windows Vista , 2003, 2008, on a 64-bit platform, click here
For Windows Vista 32-bit, click here.

If these links don’t work for you, try navigating IBM’s site:
www.ibm.com > support & downloads > fixes, updates, and drivers
Category > (under SYSTEMS) system storage
Product Family > disk systems, Product > DS4800 or whichever DS4000
Select Operating System and Click GO
Click Downloads on the Support and Downloads box.
.............

For more information do go to How to Install IBM DS4000 (FastT) Storage Manager Client on Windows

Comparison of File System

2011-10-26T23:00:00.000+08:00

Wikipedia did a wonderful job in providing a comparison of various file system
Comparison of file systems. For those who are assessing various file system offerings, do read it.

Another shorter but not as comprehensive is File System Primer from Novell

Another comparison between Lustre File System and Panasas ActiveScale Performance comparison of the Cluster File Systems at the Intel CRT-DC

Looking for solution.... CPU soft lockup detected for CentOS 5 and IBM BladeCentre

2011-10-24T00:34:00.001+08:00

One of my IBM BladeCentre node hangs and the log messages generated
"BUG: soft lockup - CPU#1 stuck for 10s! [rpciod/1:3646]"

I have been looking around, but found a reference but not a solution though. I have seen other forum that
RHEL 5.X CPU soft lockup detected in PAGE_LOCK_ANON_VMA - IBM BladeCenter and System x

Symptom
While running the Red Hat Enterprise Linux 5.x (RHEL5) family of products, the kernel may report the following error:

BUG: soft lockup - CPU#XX stuck for 10s!

Where XX can be the number of any processor in the system.

The associated stack backtrace points to page_lock_anon_vma as the code running at the time of the soft lockup detection.

Affected configurations
The system is configured with at least one of the following:

Red Hat Enterprise Linux 5, any update

This tip is not system specific.
This tip is not option specific.
Note: This does not imply that the network operating system will work under all combinations of hardware and software. Please see the compatibility page for more information: http://www.ibm.com/servers/eserver/serverproven/compat/us/

Solution
This behavior will be corrected in subsequent families of Red Hat products. For more information, contact Red Hat at the following URL:

http://www.redhat.com/about/contact/

What is nearline SAS Hard Disk?

2011-10-23T01:18:00.002+08:00

"Nearline" refers to the lower rotational speed hard disk which is usually refer to the high-capacity SATA hdd. On the other hands, "SAS is a enterprise-class drive which supposedly has a more robust mechanical specification and a controller/firmware optimized for high-volume I/O, manageability, and better error detection and correction".

So inline SAS really means standard consumer-class SATA drives wtih SAS interface.

Large capacity with enterprise interface.....Not too bad.

For more inforamtion on Nearline, see http://en.wikipedia.org/wiki/Nearline_storage

Myricom DBL 2.0 Achieves Lowest UDP and TCP Latency for High Frequency Trading

2011-10-22T12:25:00.000+08:00

For the full article, see Myricom DBL 2.0 Achieves Lowest UDP and TCP Latency for High Frequency Trading

Myricom DBL 2.0 software has benchmarked application-to-application UDP latency of under 3.5 microseconds and transparent sockets TCP latency of 4.0 microseconds. For HFT applications, DBL enables unmatched networking performance for UDP multicast and TCP order execution, all over industry-standard 10-Gigabit Ethernet
.......
DBL reduces latency by microseconds for existing applications running on standard TCP/UDP Ethernet networks. With the DBL solution, end-users can achieve extreme performance without rewriting their applications or resorting to specialty networks such as Infiniband. DBL provides transparent acceleration in both Linux and Windows environments.
......
In addition to extremely low UDP and TCP communication latency, DBL 2.0 delivers repeatable low latency, rather than unpredictable and variable latency performance found with competing solutions. Repeatable low latency performance is critical, as packet delay or loss in mission-critical trading and order environments can be devastating to the traders' bottom line.

What is SCSI RDMA Protocol (SRP)?

2011-10-19T13:16:00.000+08:00

What is SCSI RDMA Protocol (SRP)?

"The SCSI RDMA Protocol (SRP) is an emerging industry standard protocol for utilizing block storage devices over an InfiniBand™ fabric. The use of RDMA makes higher throughput and lower latency possible than what is possible through e.g. the TCP/IP communication protocol. RDMA is only possible with network adapters that support RDMA in hardware."

Here is the diagram. With this, you can use Infiniband as an alternative interconnect instead of relying on Fibre Channel. The advantages of Infiniband is obvious. It has tremendous high throughput and low latency which is important for High Read -Write.

Using dd to test and analyse read and write performance

2011-10-19T01:07:00.000+08:00

According to Wikipedia, dd is a common UNIX program whose primary purpose is the low-level copying of raw data. There are many usage of dd, but for this blog we will use dd to test and analyse read and write performance of file system.

# dd if=/dev/zero of=/home/myaccount/outfile bs=4M count=4096

4096+0 records in
4096+0 records out
17179869184 bytes (17 GB) copied, 433.088 seconds, 39.7 MB/s

if = input file
of = output
bs = block size
count = file size in kb

Malformed database image issue with yum

2011-10-18T00:09:00.000+08:00

Today I was doing a yum install after updating my LVM and I suffered a "malformed database image issue". This error can be easily rectify. Just do a

# yum clean dbcache

Then do a

# yum check

Extend LVM on Vmware Linux Guest

2011-10-17T23:56:00.000+08:00

One of my mirror ran out of space today. I've come across an excellent article on How to extend LVM on Vmware Guest running Linux by Edward's Blog. Tried his tutorial and it work without a hitch.

Which File System Blocksize is suitable for my system?

2011-10-15T01:23:00.000+08:00

Taken from IBM Developer Network "File System Blocksize"

Although the article has referenced to General Parallel File System (GPFS), but there are many good pointers System Administrators can take note of.

Here are some excerpts from the article........

This is one question that many system administrator asked before we start preparing the system. How do choose a blocksize for your file system? IBM Developer Network (File System Blocksize) recommends the following block size for various type of application.

IO Type	Application Examples	Blocksize
Large Sequential IO	Scientific Computing, Digital Media	1MB to 4MB
Relational Database	DB2, Oracle	512kb
Small I/O Sequential	General File Service, File based Analytics,Email, Web Applications	256kb
Special*	Special	16KB-64KB

What if I do not know my application IO profile?
Often you do not have good information on the nature of the IO profile or the applications are so diverse it is difficult to optimize for one or the other. There are generally two approaches to designing for this type of situation separation or compromise.

Separation
In this model you create two file systems, one with a large file system blocksize for sequential applications and one with a smaller block size for small file applications. You can gain benefits from having file systems of two different block sizes even on a single type of storage. Or you can use different types of storage for each file system to further optimize to the workload. In either case the idea is that you provide two file systems to your end users, for scratch space on a compute cluster for example. Then the end users can run tests themselves by pointing the application to one file system or another to and determining by direct testing which is best for their workload. In this situation you may have one file system optimized for sequential IO with a 1MB blocksize and one for more random workloads at 256KB block size.

Compromise
In this situation you either do not have sufficient information on workloads (i.e. end users won't think about IO performance) or enough storage for multiple file systems. In this case it is generally recommended to go with a blocksize of 256KB or 512KB depending on the general workloads and storage model used. With a 256KB block size you will still get good sequential performance (though not necessarily peak marketing numbers) and you will get good performance and space utilization with small files (256KB has minimum allocation of 8KB to a file). This is a good configuration for multi-purpose research workloads where the application developers are focusing on their algorithms more than IO optimization.

How to check FileSystem Block Size on Linux

2011-10-14T01:42:00.000+08:00

In case you wish to find out what Block Size your system is using in using, you can use the following commands to check

# tune2fs -l /dev/sda1 | grep -i 'block size'
Block size:1024

# blockdev --getbsz /dev/sda1
1024

Tuning rsize and wsize on NFS for a 10GbE network

2011-10-13T00:58:00.000+08:00

Taken from Myricom Site "Do you have recommendations for tuning NFS on a 10GbE network"

Use Recent Linux Kernel 2.6.19 or later. CentOS 6 will be a good candidate to implement.
On /etc/fstab, you can set rsize=1048576,wsize=1048576
You can use the above buffers on NFSv3
Do note that for Linux Kernel 2.18 and below, the rsize and wsize is 32KB.

Using mpstat to display SMP CPU statistics

2011-10-10T00:34:00.001+08:00

mpstat is a command-line utilities to report CPU related statistics.For CentOS, to install mpstat, you have to install the sysstat package (http://sebastien.godard.pagesperso-orange.fr/)

# yum install sysstat

1. mpstat is very straigtforward. Use the command below. On my 32-core machine,

# mpstat -P ALL

11:10:11 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
11:10:13 PM  all   40.75    0.00    0.03    0.00    0.00    0.00    0.00   59.22   1027.50
11:10:13 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   1000.50
11:10:13 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM    2  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM    4  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM    5  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM    7  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00     16.50
11:10:13 PM    9    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   10    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   11    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   12  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00     10.50
11:10:13 PM   13    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   14   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   15  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   16    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   17    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   18    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   19  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   20  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   21    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   22    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   23    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   24  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   25  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   26    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
11:10:13 PM   27  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
11:10:13 PM   28    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00

whereCPU - Processor number. The keyword all indicates that statistics are calculated as averages among all processors.
%user - Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice - Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%sys - Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing interrupts or softirqs.
%iowait - Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%irq - Show the percentage of time spent by the CPU or CPUs to service interrupts.
%soft - Show the percentage of time spent by the CPU or CPUs to service softirqs. A softirq (software interrupt) is one of up to 32 enumerated software interrupts which can run on multiple CPUs at once.
%steal - Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%idle - Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
intr/s - Show the total number of interrupts received per second by the CPU or CPUs.

2. Getting average from mpstatTo get an average you have to invoke the interval and count argument. In the example, interval is 2 second for 5 count

# mpstat -P ALL 2 5

At the end of the statistics report, you will see an average

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all   40.76    0.00    0.03    0.00    0.00    0.00    0.00   59.21   1047.50
Average:       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   1000.60
Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:       2  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:       4   99.90    0.00    0.10    0.00    0.00    0.00    0.00    0.00      0.00
Average:       5  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:       7  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:       8    0.00    0.00    0.10    0.00    0.00    0.00    0.00   99.90     17.30
Average:       9    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      10    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      11    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      12   99.90    0.00    0.00    0.00    0.00    0.10    0.00    0.00     29.70
Average:      13    0.00    0.00    0.10    0.00    0.00    0.00    0.00   99.90      0.00
Average:      14   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00      0.00
Average:      15  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      16    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      17    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      18    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      19  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      20  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      21    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      22    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      23    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      24  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      25   99.90    0.00    0.10    0.00    0.00    0.00    0.00    0.00      0.00
Average:      26    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      27  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      28    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      29    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
Average:      30  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Average:      31    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00

Outline of File Hierarchy Systems in RHEL, CentOS and SL

2011-10-09T00:58:00.000+08:00

The location of the files and directories in RHEL or its clone system are based on the Filesystem Hierarchy System (FHS) guidelines. For more information on the Filesystem Hierarchy System (FHS), do read the Filesystem Hierarchy Standard

/bin/ (Essential Commands for admins and users)
/usr/bin/ (Common commands for admins and users)
/sbin/ (Essential commands for admins)
/usr/sbin (Common commands for admins)
/tmp/ (Temporary files for all users)
/usr/local/ (Location for locally-installed software indepndent of operating systems updates)
/usr/share/man (Manual Pages)
/usr/src (source code)
/var/ (variable files such as spool and log files)
/var/log/ (Log files)
/etc/ (Configuration files)
/proc/ (Kernel virtual file system)
/dev/ (Device file)

Much of these information is derived from "Red Hat Enteprise Linux Administration Unleased"

Avoiding DNS Lookup for Apache 2

2011-10-08T11:57:00.001+08:00

If you wish to avoid situations where you do not wish to do DNS lookup for the client machines which will slow Apache performance. To do it is quite quick by setting the HostNameLookups directive to off at /etc/httpd/conf/httpd.conf

HostNameLookups Off

A good read - Dissecting shared libraries

2011-10-06T00:56:00.004+08:00

This article "Dissecting shared libraries" from IBM DeveloperWorks is a good read if you wish to have a deeper understanding on shared libraries.

Troubleshooting Blade Management Module connectivity issues

2011-10-03T13:00:00.002+08:00

This article is a sub-set of the full document from IBM "Troubleshooting Management Module connectivity issues"

Solution

The Management Module (MM) and the Advanced Management Module (AMM) are the central points of management for the IBM BladeCenter chassis. As such, when the MM is not responsive, the ability to perform normal management on the chassis is significantly compromised. This document covers four different symptoms related to MM connectivity failures: (1) cannot login to the web or telnet interface because of USERID and/or PASSWORD failures. (2) cannot get any network response from the MM, and (3) the MM responds to network pings, but either the web interface or telnet interface does not respond. (4) MM failover does not work.

Throughout this document, "MM" will be used to mean either the MM or AMM. The term AMM will only be used to point out any differences between the two.

When troubleshooting MM connectivity problems, there are a few common procedures that are used in several situations.

Reset the IPaddress of the MM (this procedure does not work on the AMM)

When the MM is restored to its default TCPIP configuration, the Ethernet port on the MM will attempt to get a DHCP address. Disconnect the Ethernet cable if this is not wanted. With the Ethernet cable disconnected, the MM will search for a DHCP server for five minutes, then timeout and take the address 192.168.70.125/255.255.255.0.

Before resetting the MM to its default configuration, have a laptop local to the chassis that can connect to the MM with a cross-over cable (the AMM supports either cable type). Make sure that the laptop is configured with the IPaddress 192.168.70.100/255.255.255.0 so it will not conflict with any address on the chassis. To reset the TCPIP address on the MM, insert a paper-clip into the hole on the back of the MM labeled "IPreset" until it depressed the button inside. Hold it there for just under three seconds, then remove the paper clip. That resets the MM's Ethernet interface to its default configuration.

Reset the IPaddress of the AMM using the serial cable

The AMM has a port for ethernet and serial connectivity. The serial port is at the top of the AMM, just above the video connection. To connect to the serial port, insert one end of a straight-through ethernet cable in the AMM serial port. Attach the other end of the cable to the serial dongle whose pinouts are described in the AMM Installation Guide ("Serial connection," near the end of Chapter 3).

The default serial settings for the AMM are 57k, 8 data bits, No parity, 1 stop bit, flow control off. Once connected to the serial console, login as usual. Create a basic config for the external interface with the following commands (system: x is either system:mm 1 for the AMM in slot 1 or system:mm 2 for the AMM in slot 2).

use static ip: ifconfig -eth0 -c static -T system:mm x

IPaddress: ifconfig -eth0 -i ip-address -T system:mm x

subnet mask ifconfig -eth0 -s subnet mask -T

system:mm x

gateway: ifconfig -eth0 -g IPaddress of gateway -T

system:mm x

They can be combined into one long command as follows:

ifconfig -eth0 -i ip_address -s subnet mask -g IPof gateway -c static -T system:mm x

Reset the MM to its default configuration

One should remember that resetting the MM to defaults turns off the external ports for all four I/O modules, which will cut off all network and fibre connectivity. Therefore, this operation should only be done when the chassis is in a maintenance window and can be off-line for a short period of time. Also, when the MM is restored to its default configuration, it will attempt to get a DHCP address. Disconnect the Ethernet cable if a DHCP address is not wanted. The MM will search for a DHCP server for five minutes, then timeout and take the address 192.168.70.125/255.255.255.0. Before resetting the MM to its default configuration, have a laptop local to the chassis that can connect to the MM with a cross-over cable (the AMM supports either cable type). Make sure that the laptop is configured with the IPaddress 192.168.70.100/255.255.255.0 so it will not conflict with any default address on the chassis.

If the MM is accepting web logins, the default configuration can be restored in the web GUI at:

Select (MM) MM Control, click Restore Defaults, and then click Restore Defaults

Select (AMM) MM Control, click Configuration Mgmt, then click Restore Defaults or click Restore Defaults Preserve Logs

If neither login service is working, the default configuration can be restored by accessing the back of the MM. On the back of the MM, there is a pin hole that is large enough for a paper clip. It is labeled "IPreset." In addition to resetting the IPaddress, pushing a paper clip in for the right amount of time resets the entire MM configuration back to its defaults. To reset the Management Module to the default configuration, including the default login name "USERID" and password "PASSW0RD," push a paper clip into the pin hole until it hits the button inside and hold it. The amount of time required to hold the pin in varies as follows:

MM with 82D firmware or earlier = push in for 5 seconds, then release the pin for 5 seconds, then push it in for another 10 seconds. The timing is quite precise, make sure a watch with a second hand is available. When the reset starts, the fans will ramp up to full speed, which is clearly audible.

AMM or MM with 82F firmware or later = push in the pin and hold it for 10 seconds. When the reset starts, the fans will ramp up to full speed, which is clearly audible.

Remove and reinsert the MM

Troubleshooting the MM sometimes requires physically removing it from the slot and re-inserting it. Before removing it, note whether the green Ethernet LED or amber LED are lit. In normal operations with an Ethernet cabled connected, the Ethernet LED will be on, and the ambler LED will be off. The amber LED will come on briefly when the MM is powered on or reset. It is also a good idea to look at the female connectors when the MM is removed and examine the female connectors to confirm they have not been damaged. When both MMs are removed, the fans ramp up to full speed.

This is clearly audible. When re-inserting the MM, listen to hear if the fans return to the previous noise level. If they do, that indicates that the MM has completed its POST process. If they do not, that indicates that there is some other problem with the chassis that the MM is trying to address. For a visual indication that the MM is working correctly, look at the MM directly.

After the MM is re-inserted and an Ethernet cable connected, confirm the status of the green Ethernet LED and the amber LED. If the amber LED stays on, that indicates a fault in the MM.

Symptom 1: Cannot login due to bad userid or password

If a user makes five unsuccessfull login attempts, the MM will stop accepting logins for a period of time. Two minutes is the default lockout time, though this is configurable in the MM interface at MM Control then click Login Profiles.

If login fails through both the web and telnet interface, resetting the MM to the default login of "USERID" and "PASSWORD" can be accomplished by following the procedure "Reset the MM to its default configuration." The default login ID and password are case sensitive, and in "PASSWORD." a zero is used for the letter "O."

If USERID/PASSWORD login problems still exist after resetting the MM to defaults, contact IBM support. If the MM does not have network connectivity after resetting defaults, follow the steps below for the appropriate symptom.

Symptom 2: MM does not respond to any network connection
If the MM does not respond to any remote network connection, troubleshooting will need to be done at the chassis. The first step is to find a laptop that can login to other MMs and connect it to the MM with a cross-over cable (either a cross-over or straight through can be used for the AMM). Verify that the IPconfiguration on the laptop puts it in the same subnet as the MM, and verify that the laptop is not running a local firewall. Try to connect to the MM via a web browser, telnet, and ping. Depending on the results you get, take the following steps:

If the laptop has complete access to the MM when connected locally, then the previous connectivity problems are most likely due to network problems on the customer's LAN, or the other workstation the customer used to access this MM.

If the laptop can ping the MM, but cannot connect via web browser or telnet, go to symptom

If the laptop cannot ping the MM, take the following steps to try and restore connectivity.

Clear the arp cache on the laptop. - If the chassis has a redundant MM, fail over to it and attempt to connect to it.

If the chassis only has one MM, move it to the other slot, following the procedure "Remove and reinsert the MM."

Follow the procedure "Reset the IPaddress of the MM" or "Reset the IPaddress for the AMM using the serial cable."

Follow the procedure "Reset the MM to its default configuration"

If these all fail, contact IBM support for assistance.

Symptom 3: Cannot connect to the MM using the web browser/telnet/ssh, but can ping the MM
The MM runs a few network servers that enable users to login and manage the chassis. If basic connectivity via 'ping' is functioning, but one or more of the login services is not working (for example, web server, telnet server), the problem is due to a configuration error or firmware defects. It is never a hardware failure. When the MM will respond to a ping, but any one of the login services does not respond, take the the following steps:

Ensure that a supported web browser is being used.

If possible, verify whether the MM is running the network servers on their default network ports. If all logins fail, check with the administrator for the BladeCenter. If it is possible to login to the web interface, select MM Control and click Port Assignments. There is no way to get that information in the telnet interface.

Verify whether this workstation can connect to other MMs. If it cannot, the problem is most likely due to a firewall running on the client workstation or the network. Shutdown any firewalls on the client machine and try again. If the client still has problems connecting to multiple MMs, consult the network administrator for the LAN.

Restart the MM. If the MM responds to network logins after it has been restarted, this is most likely an MM firmware defect. Download the changelog for the current MM or AMM code and see if any similar issues have been resolved. If not, contact IBM Support for additional assistance.

At this point. troubleshooting must continue with a laptop or other workstation local to the MM. Find a laptop which can connect to other MMs, and connect it directly to the MM with a crossover cable (both cross-over and straight-though work for the AMM). Verify that the Ethernet link is up and the laptop is configured so it is on the same subnet as the MM. If the laptop can ping the MM, attempt to login to the MM with a supported web browser. If that works, contact the network administrator for assistance troubleshooting the network.

If the MM still does not allow logins over the WEB interface at this point, restore the MM to its defaults with the procedure "Reset the MM to its default configuration." If this does not restore connectivity, contact IBM Support.

Symptom 4: Failover of MM to redundant MM does not work

When there are two MMs in a chassis, one MM is active and the second MM is on standby. When a user initiates a failover from the primary to redundant, the primary sends a message to redundant to become the primary, then reboots itself. On rare occasions, this does not work. When it does not, take the follow steps to resolve it:

Examine the MM Event log and the MM BIST log to see if any errors have been detected.

Physically remove the primary MM and see if the redundant MM boots successfully. If it does not, move it to the other slot and see if it can boot in that slot

Reset the MM to defaults using the procedure "Reset the MM to its default configuration."

Repeat the failover process with both MMs. If it still does not work, contact IBM support

Getting `GLIBCXX_3.4.9 not found' when starting the license manager for MATLAB

2011-09-28T00:28:00.001+08:00

If you are starting the latest version of MATLAB License Server, you may encounter this error`GLIBCXX_3.4.9 not found'

For more information do look at MathWorks "Why do I get an error 'GLIBCXX_3.4.9 not found"

According to the mathwork website,

Solution:

This issue is caused by a missing or outdated libstdc++.so.6 as required by the keycheck application (R2011a) or the MLM vendor daemon (R2011b). Both the R2011a keycheck and R2011b MLM vendor daemon require libstdc++.so.6.0.10. Refer to your operating system documentation for information on how to update or install a missing library.

If the necessary version of the library is not available for your Linux distribution it can be copied and installed from the MATLAB installation files following the instructions below:

NOTE: $MATLAB refers to the MATLAB installation location (ex: /usr/local/MATLAB/R2011b)
NOTE: $ARCH refers to the machine architecture (ex: glnx86 for Linux 32-bit or glnxa64 for Linux 64-bit)

If MATLAB is installed in addition to the FlexNet license manager, skip directly to step 3.

1. Create a subdirectory within the MATLAB installation folder as shown below:

[root@localhost ~] mkdir -p $MATLAB/sys/os/$ARCH

2. Copy the libstdc++.so.6.0.10 library from the MATLAB installation files (either an installation DVD or the extracted downloaded installer archive) into the newly created directory:

[root@localhost ~] cp /media/MATLAB_R2011b/bin/$ARCH/libstdc++.so.6.0.10 $MATLAB/sys/os/$ARCH

3. Run 'ldconfig' to create symbolic links to the new library and update the dynamic linker cache:

[root@localhost ~] ldconfig $MATLAB/sys/os/$ARCH

World's largest Cloud Storage System designed by SDSC

2011-09-28T00:02:00.001+08:00

San Diego Supercomputer Center (or SDSC) has designed the World's largest Cloud Storage System that is specifically targeted for academic and research use. Dubbed the SDSC Cloud, the total cloud capacity will start at raw capacity of 5.5 PBand is scalable to "Hundreds of Petabytes". Rates are starting at US$3.46 a month for 100GB storage.

For more information do look at

Understanding VXLAN Virtual-Physical-Cloud L2/L3 Networks by ARISTA

2011-09-26T23:14:00.002+08:00

Interesting Article....From ARISTA

Understanding VXLAN Virtual-Physical-Cloud L2/L3 Networks by ARISTA (pdf)

Video interview - Vmware CTO Steve Herrod and Arista Founder, CDO, Chairman Andy Bechstolsheim

2011-09-26T23:09:00.002+08:00

Video interview - Vmware CTO Steve Herrod andArista Founder, CDO, Chairman Andy Bechstolsheim

Server has no node list when executing pbsnodes -s

2011-09-26T09:19:00.002+08:00

If you see this error "Server has no node list" when you execute "pbsnodes -s 192.168.1.1" (your torque server DNS or IP), it is due to the missing "nodes" file that was supposed to be at /var/spool/torque/server_priv/

The node file should show something like this

## This is the TORQUE server "nodes" file.
##
## To add a node, enter its hostname, optional processor count (np=),
## and optional feature names.
##
## Example:
##    host01 np=8 featureA featureB
##    host02 np=8 featureA featureB
##
## for more information, please visit:
##
## http://www.clusterresources.com/torquedocs/nodeconfig.shtml

compute-c00     np=12
compute-c01     np=12
compute-c02     np=12
compute-c03     np=12
compute-c04     np=12

Restart the pbs_sched and pbs_server services

# service pbs_server restart
# service pbs_sched restart

Using iperf to measure the bandwidth and quality of network

2011-09-25T00:37:00.002+08:00

According to iperf project site. This writeup is taken from iPerf Tutorial by OpenManiak. For a more detailed and in-depth writeup, do read up on the iPerf Tutorial

Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

Iperf can generate traffic using TCP and UDP Traaffic to perform the following kinds of test

Latency (response time or RTT): can be measured with the Ping utility.
Jitter: can be measured with an Iperf UDP test.
Datagram loss: can again, be measured with an Iperf UDP test.
Bandwidth tests are done using the Iperf TCP tests

A collection of selected iperf usages is written in Using iperf to measure the bandwidth and quality of network from linuxCluster.wordpress.com

Cannot resolve default server host for Torque, check server_name file

2011-09-24T13:29:00.000+08:00

If you are encountering Torque Error something like

Cannot resolve default server host 
'headnode.cluster.com' - check server_name file.
pbsnodes: cannot connect to server headnode.cluster.com, 
error=150010 (Access from host not allowed, or unknown host)

To resolve this issues, you have to look at 3 possible mis-configured areas

Ensure your /etc/sysconfig/network reflect the correct hostname
Ensure your /var/spool/torque/server_name are the same for both head and compute nodes
Ensure the environment variable PBS_DEFAULT is reflecting the correct hostname. For my situation, I have placed the environment variable on /etc/profile.d/torque.sh

You should be able to eliminate the issue.

/.../libfftw3f.a: could not read symbols: Bad value when compiling Gromacs

2011-09-20T13:26:00.003+08:00

If you encounter this error like " /.../libfftw3f.a: could not read symbols: Bad value when compiling Gromacs ", it is likely due to that the compilation of FFTW did not enable shared libraries "--enable-shared" . GROMACS seems to require shared libraries extension from FFTW. So do follow the steps listed in

you should have a clean compilation

How to enable Directory listing on Apache

2011-09-19T14:24:00.000+08:00

To enable Directory Listing for a particular folder at Apache, you need to set this option on at /etc/httpd/conf/httpd.conf or put your new settings at /etc/httpd/conf.d/yourfile.conf. Either way is find

<Directory /home/mysite/public_html>
        Options Indexes FollowSymLinks
        AllowOverride None
</Directory>

/usr/bin/ld cannot find -lliblapack.so

2011-09-13T00:38:00.000+08:00

We were linking our c program to gcc which require the lapack and blas libraries. I did not compile my own libraries but uses instead the CentOS lapack and blas libraries. For more information on installing lapack and blas Installing lapack, blas and atlas on CentOS 5.4

We will compiling our codes

$ gcc exact.c -L/usr/lib64 -lliblapack.so.3 -llibblas.so.3

The result error

/usr/bin/ld: cannot find -lliblapack.so.3
collect2: ld returned 1 exit status

I checked my /etc/ld.so.conf.d/ to ensure that I have /usr/lib64. Of course remember to do a ldconfig if you have made any changes

If the -L and -I does not allow linking process to scan correctly, do a direct complete pathing of the library like the one below and it compile nicely

$ gcc exact.c /usr/lib64/liblapack.so.3 /usr/lib64/libblas.so.3

In case you do not know where to locate your libraries, you can do a

$ locate liblapack

For similar notes on the linking challenges, you may want to explore this forum thread
problems in linking lapack to g77

Recommended sshd_config for OpenSSH

2011-09-11T00:09:00.001+08:00

There are a few settings at /etc/ssh/sshd_config we can set to improve security, performance and user experience. Many of this information comes from SSH The Secure Shell, 2nd Edition from O'Reilly

1. Using SSH-2 Protocol and disable SSH-1 protocol altogether

Protocol 2

2. Ensure that the HostKey and PidFile are located on a machine's local disk and not over the NFS mount. The default setting should be in the machine local file like those below

HostKey /etc/ssh/ssh_host_key
PidFile /var/run/sshd.pid

3. File and directory permissions
The StrictModes value requires users to protect their SSH-related files and directories or else they will not authenticate.The default values is yes

StrictModes yes

4. Enable KeepAlive messages
Keepalive messages are enabled so that the connections to clients that have crashed or unreachable will terminate rather than be an orphaned process which require manual intervention by sysadmin to eliminate it.

Port 22 
ListenAddress 0.0.0.0
TcpKeepAlive yes

5. Disable Reverse DNS lookup

UseDNS no

6. Select a shorter grace login time
The default grace login is 2 minute which you might want to change. The value here is 30 seconds

LoginGraceTime 30

7. Authentication
The default setting are fine unless you wish to use Public-Key Authentication and wish to disabled Kerberos, Interactive and GSSAPIAuthentication

PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
RSAAuthentication yes
RhostsRSAAuthentication no
HostbasedAuthentication no
KerberosAuthentication no
ChallengeResponseAuthentication yes
GSSAPIAuthentication no
IgnoreRhosts yes

8. Access Control
If you wish to allow only selected users or groups to use ssh, you would like to use

AllowGroups users
AllowUsers me_only
DenyGroups black_list
DenyUsers hacker_id

For more information, see How do I permit specific users SSH access?

9. Securing TCP port forwarding and X forwarding

AllowTcpForwarding yes
X11Forwarding yes

rpm needed for blas, blacs, scalapack for CentOS

2011-09-10T01:50:00.001+08:00

If you are installing blas, blacs, scalapack in CentOS manually, you will need these packages.

mpi-selector-1.0.2-1.el5.noarch.rpm
libgfortran-4.1.2-50.el5.x86_64.rpm
lapack-3.0-37.el5.x86_64.rpm
lam-7.1.2-14.el5.x86_64.rpm
lam-libs-7.1.2-14.el5.x86_64.rpm
blacs-1.1-24.el5.1.x86_64.rpm           [EPEL]
blacs-devel-1.1-24.el5.1.x86_64.rpm [EPEL]
blas-3.0-37.el5.x86_64.rpm
blas-devel-3.0-37.el5.x86_64.rpm
scalapack-1.7.5-1.el5.x86_64.rpm    [EPEL]

Of course if you use yum install, that will be more efficient. You will need EPEL and the standard repository from CentOS

Hitachi Data Systems Corporation (HDS) Acquires BlueArc Corporation

2011-09-08T17:01:00.000+08:00

Hitachi Data Systems Corporation(HDS), a wholly owned subsidiary of Hitachi, Ltd. (NYSE: HIT / TSE: 6501), on 7 Sep 2011, announced asignificant milestone in its strategy to give customers seamless access to alldata, content and information with the acquisition of BlueArc Corporation, aleader in scalable, high performance network storage. Building upon asuccessful 5 year OEM partnership, HDsand BlueArc will give customers theunmatched combination of Hitachi enterprise-class quality, reliability andsupport with innovative, highly scalable, high performance BlueArc networkattached storage (NAS).

Hitachi Data Systems Announces Acquisition of BlueArc

Resolving Slow SSH Login

2011-09-07T00:01:00.000+08:00

If you are facing slow login times, it might be due to reverse DNS is not responding quick enough. This system can show up on your log file

# tail -50 /var/log/secure

You will notice that there is a time lag from accepting the key to opening a session

Sep  6 10:15:42 santol-h00 sshd[4268]: 
Accepted password for root from 192.168.1.191 port 51109 ssh2

Sep  6 10:15:52 santol-h00 sshd[4268]: pam_unix(sshd:session): 
session opened for user root by (uid=0)

To fix the issue, you should modify the /etc/ssh/sshd_config file

# vim /etc/ssh/sshd_config

At /etc/ssh/sshd_config, change UseDNS no

#ShowPatchLevel no
UseDNS no
#PidFile /var/run/sshd.pid

Restart the ssh service

# service sshd restart

Feel the login speed :)

Interesting Writeup on the Four Pillars of Software-Defined Cloud Networking

2011-09-06T00:39:00.002+08:00

Do read this interesting article.

The Four Pillars of Software-Defined Cloud Networking by Arista Network. Here the summary in a table format

Spirit of Startup. Security of Hewlett Packard

2011-09-03T00:59:00.003+08:00

I think most of us have heard about Hewlett Packard spinning of its PC Business. Read the official response from Hewlett Packard what they will do with their PC Business. A new Startup?

Here is the message from their Executive Vice President, Personal Systems Group, Todd Bradley, taken from Hewlett Packard "Spirit of Startup. Security of Hewlett Packard"

HP is the #1 PC maker on the planet, and that won't change. I can assure you our future is brighter than ever.

Spirit of a Startup

Our preferred course to harness our vision of the future is to build a separate, more agile company. It's time to think like a startup again. It's time to be nimble and revolutionary. It's time again for world-changing innovation. And so, it's time we realized we're at a crossroads in an evolving HP.

But don't misunderstand: We-the same great
folks who make HP PCs today-will make them tomorrow. We will continue to build on our legacy creating reliable, stylish, and high-performance PCs to improve your personal and professional life.

This is the future we are passionate about, and we hope this site will answer all your questions and leave you feeling as inspired and excited as we are for what lies ahead.

Security of HP

We became #1 by focusing on our customers' needs.
On its own, our PC business would be the 60th largest Fortune 500 company.
We sell two PCs every second.
We provide personal computing products, services, and support for customers in over
170 countries.
We have been dedicated to our customers for over 70 years, and we look forward to many,
many more.

Apache released 2nd workaround for Devastating' Apache bug

2011-09-01T00:18:00.000+08:00

Apache had release 2nd update but no solution yet for the 'Devastating' Apache bug leaves servers exposed . For more information on the 2nd update, see
http://mail-archives.apache.org/mod_mbox/httpd-announce/201108.mbox/%3C20110826103531.998348F82@minotaur.apache.org%3E

Finally a solution is provided. See Important Apache (httpd) security Update

Tweaking the Linux Kernel to manage memory and swap usage

2011-08-30T01:47:00.000+08:00

This writeup is assuming you are tweaking to the minimise swap and maximise physical memory. This tweaking should be considered especially for High Performance MPI applications where good low latency parallelism between nodes is very essential.

In addition, this writeup also help you to “kill” runaway memory applications. For more information, see Tweaking the Linux Kernel to manage memory and swap (Linux Cluster)

Network File System (NFS) in High Performance Network

2011-08-29T00:05:00.001+08:00

This article "High Performance (NFS) in High Performance" by Carnegic Mellon is very interesting article about NFS Performance. Do take a look. Here is a summary of their fundings

For point-to-point throughput, IP over InfiniBand (Connected Mode) is comparable to a native InfiniBand.
When a disk is a bottleneck, NFS can benefit from neither IPoIB nor RMDA
When a disk is not a bottleneck, NFS benefits significantly from both IPoIB and RDMA. RDMA is better than IPoIB by ~20%
As the number of concurrent read operations increases, aggregate throughputs achieved for both IPoIB and RDMA significantly improve with no disadvantage for IPoIB

High NFS Load causing echo 0 > /proc/sys/kernel/hung_task_timeout_secs

2011-08-28T00:20:00.001+08:00

Do note that simultaneous numerous write by the NFS Clients on the NFS Server will cause tremendous performance penalty and system lock-out as describe below. You will notice if you use "top" utilities, the load can be extremely high as numerous system locks are queued.

One of my researcher was running a intense load on the NFS Server that cause an eventual "echo 0 > /proc/sys/kernel/hung_task_timeout_secs". Before that, I saw on the log file "rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket"

To solve the problem, you have to lighten the load of the NFS or improve the setting. You may want to take a look at the Configuring NFS Server for Performance. A longer term solution will be to move to parallel file system.

'Devastating' Apache bug leaves servers exposed

2011-08-27T00:52:00.000+08:00

'Devastating' Apache bug leaves servers exposed

Devs race to fix weakness disclosed in 2007

By Dan Goodin in San Francisco

Posted in Security, 24th August 2011 18:05 GMT

Maintainers of the Apache webserver are racing to patch a severe weakness that allows an attacker to use a single PC to completely crash a system and was first diagnosed 54 months ago.

Attack code dubbed “Apache Killer” that exploits the vulnerability in the way Apache handles HTTP-based range requests was published Friday on the Full-disclosure mailing list. By sending servers running versions 1.3 and 2 of Apache multiple GET requests containing overlapping byte ranges, an attacker can consume all memory on a target system.

“The behaviour when compressing the streams is devastating and can end up in rendering the underlying operating system unusable when the requests are sent parallely,” Kingcope, the researcher credited with writing and publishing the proof-of-concept attack code wrote Wednesday on Apache's Bugzilla discussion list. “Symptoms are swapping to disk and killing of processes including but solely httpd processes.”

The denial-of-service attack works by abusing the routine web clients use to download only certain parts, or byte ranges, of an HTTP document from an Apache server. By stacking an HTTP header with multiple ranges, an attacker can easily cause a system to malfunction. On Wednesday morning, Apache developers said they expect to release a patch in the next 96 hours.

The Apache advisory contains several workarounds that admins can deploy in the meantime.

The susceptibility of Apache's range handling to crippling DoS attacks was disclosed in January 2007 Michal Zalewski, a security researcher who has since taken a job with Google. He said at the time that both Apache and Microsoft's competing IIS webserver were vulnerable to crippling DoS attacks because of the programs' “bizarro implementation” of range header functionality based on the HTTP/1.1 standard.

“Combined with the functionality of window scaling (as per RFC 1323)), it is my impression that a lone, short request can be used to trick the server into firing gigabytes of bogus data into the void, regardless of the server file size, connection count, or keep-alive request number limits implemented by the administrator,” Zalewski wrote. “Whoops?”

In an email to The Register on Wednesday, Zalewski wrote: “Not sure why they haven't done something about it back then, probably just haven't noticed in absence of an exploit.”

The episode challenges the conventional wisdom repeated by many proponents of open-source software that flaws in freely available software get fixed faster than in proprietary code because everyday users are free to inspect the source code and report any vulnerabilities they find. Assuming that claim is true, the four-year weakness in Apache's range-handling feature would appear to be an obvious exception.

About 235 million websites use Apache, making it the most widely used webserver with about 66 percent of the entire internet, according to figures released last month by Netcraft. IIS ranked second with more than 60 million sites, or about 17 percent.

In a statement issued several hours after this article was published, Microsoft spokesman Jerry Bryant said: "IIS 6.0 and later versions are not susceptible to this type of denial-of-service due to built in restrictions." ®

Update

Trustwave's SpiderLabs has provided a detailed technical analysis here along with instructions for mitigating attacks using the open-source ModSecurity firewall.

Infiniband HOWTo

2011-08-26T00:47:00.000+08:00

Stumble on this short but wonderful tutorial on Infiniband HOWTo by Guy Coates. Although the article is for Debian, but you can apply to CentPS

IBM HPC Management Suite for Cloud

2011-08-25T00:56:00.000+08:00

Do look at IBM Management Suite for Cloud . According to the site, the list of comprehensive tools include

Provisions bare metal high performance compute clusters for technical computing and analysis workloads.
Consolidates the infrastructure for efficient sharing of HPC resources.
Accesses the HPC infrastructure through an on-demand, self-service web portal optimized for HPC users and administrators.
Achieves rapid image deployment and resource management using diskless provisioning.
Centralizes user and energy management, usage metering and accounting.

Strange /etc/sysconfig/network-scripts/ifcfg-eth*.bak problems on CentOS

2011-08-24T01:33:00.002+08:00

I think there is a bug with Kudzu that causes sometimes to change network setting from /etc/sysconfig/network-scripts/ifcfg-eth* to /etc/sysconfig/network-scripts/ifcfg-eth*.bak causing loss of network connectivity after there is a reboot. The original setting was replaced with one with DHCP. In other words, the original ifcfg-eth0 becomes ifcfg-eth0.bak and a new one ifcfg-eth0 is a config scripts with DHCP settings.

It seems that the cause of this issue seems to be coming from the Kudzu Daemon. Once I stop the service, my network configuration remain as it is during bootup.

# chkconfig --levels 2345 kudzu off

# service kudzu stop

For more information, do look at CentOS Discussion Thread http://www.centos.org/modules/newbb/viewtopic.php?topic_id=8376

How to check whether crontab is working for CentOS

2011-08-21T01:05:00.000+08:00

To check whether Crontab is working, there are a few pointers you may wish to take note:

1. Check the crond service whether it is working

# service crond status

crond (pid  2873) is running...

Or if you prefer to use ps

# ps -ef | grep cron

2. Check your log file.

# tail -20 /var/log/cron

You should see something like this

........... 

Aug 22 00:41:01 yoursever crond[29827]: (root) CMD ("/root/yourscripts.sh" 2> /usr/local/yourscript_rsync/yourscript_rsync.errors > /dev/null)

..........

Stuck at " running /sbin/loader " on CentOS 5 using PS/2 keyboard

2011-08-17T02:30:00.003+08:00

I was installing CentOS 5.5 on a IBM x3550 M3. But somehow the installation procedure was stuck and stops at "running /sbin/loader" I waited for a while after 15 min, but it has not moved away from the "running /sbin/loader".

I suspect that my PS/2 keyboard with my PS/2 connector to usb converter for my keyboards was an issue after reading some forums that have raised such an issue.

I changed the keyboard to a usb-keyboard and the issue was gone.

Green Memory and SSDs

2011-08-16T10:31:00.002+08:00

Interesting Information on Green Memory and SSDs from Samsung.

From Samsung

"In a server, distribution of power is directly related with its memory density and as such the need to reduce memory power consumption is becoming increasingly critical. When memory density increases among different servers, its contribution to total system power increases in other words, more memory requires a server to allocate more power for memory usage....."

SGI Acquires OpenCFD Ltd., the Leader In Open Source Computational Fluid Dynamics (CFD) Software

2011-08-16T08:53:00.002+08:00

Interesting news....More information, read more.....

SGI Acquires OpenCFD Ltd., the Leader In Open Source Computational Fluid Dynamics (CFD) Software

RAMSpeed cache and memory benchmarking tool

2011-08-15T00:05:00.000+08:00

RAMspeed is a free open source command line utility to measure cache and memory performance of computer systems. For more information on the algorithm and other information, do read the RAMSpeed site.

RAMspeed is more accurate than many other benchmarking tools, more customisable, open source, compact, and gives you much more information to analyse. Some people may say that the lack of some graphical interface is a large drawback, but it may be considered as an advantage as well.

Now for the download information.....taken from RAMSpeed site.

RAMspeed (UNIX) v2.6.0 (August, 2009) — for uniprocessor machines running UNIX-like operating systems. The source code is available for download (76Kb).

RAMspeed/SMP (UNIX) v3.5.0 (August, 2009) — for multiprocessor machines running UNIX-like operating systems and supporting System V IPC extensions. The source code is available for download (78Kb).

RAMspeed (DOS) v2.5.0 (August, 2009) — for DOS as well as 32-bit Windows operating systems (95 to 2003; i386 only). Both the source code and a pre-compiled executable are available for download (109Kb).

RAMspeed (Win32) v1.1.1 (August, 2009) — for 32-bit as well as 64-bit Windows operating systems (95 to Vista; i386 or amd64). Both the source code and a pre-compiled executable are available for download (71Kb).

Memory Management - Preventing the kernel from dishing out more memory than required

2011-08-12T01:00:00.003+08:00

I think for us who have been running computational jobs have seen the memory got eaten up by some buggy or stray applications. Hopefully the kernel kills it. But somehow you must have seen that the kernel may not have kill the culprit and the server go to a linbo.

Let's say if we wish to ensure that the kernel only gives out memory to processes equal to the physical memory, then we have to do the following at /etc/sysctl.conf or /etc/sysctl.d/myapp.conf

My assumption is that you have 10GB of swap and 20GB of memory and you wish the kernel to stop handling processes at 18GB RAM, then the calculation should be (swap size + 0.4 * RAM size)

So at /etc/sysctlf.conf, the configuration will be

vm.overcommit_memory = 2

vm.overcommit_ratio = 40

Note: The ratio is (40/100). For explanation of vm.overcommit_memory =2. Do look at Tweaking Linux Kernel Overcommit Behaviour for memory

Once the memory hits 18GB, the so-called OOM killer of the Linux kernel will kick in.

Another calculation example is that your RAM size and SWAP size are the same and you wish exactly the physical memory to be used only. then

vm.overcommit_memory = 2

vm.overcommit_ratio = 0

For more information, do read

Tweaking Linux Kernel Overcommit Behaviour for memory

2011-08-11T01:04:00.000+08:00

You can change the behaviour of the Linux Kernel in regardings to its Overcommit Behaviour for memory. You can change the behaviour at /etc/sysctl.conf/ or /etc/sysctl.d/myconfig.conf

You will write something like

vm.overcommit_memory = (a integer from 0 to 2)

where
0 - means that the kernel will use predefined heuristics when deciding whether to allow such an overcommit. This is the default.
1 -always overcommits (Very dangerous)
2 - Prevents overcommits from exceeding a certain value. Within this mode, the total commit can not exceed the swap space(s) size + overcommit_ratio percent * RAM size. By default, the overcommit ratio is 50.

For example, do look at Memory Management - Preventing the kernel from dishing out more memory than required

Selecting default OS or kernel in GRUB boot loader

2011-08-10T00:01:00.002+08:00

If you are wishing to change the default OS or kernel in GRUB boot loader, you can easily make the changes

# vim /boot/grub/grub.conf

Change the default=0 to 1 of other numeric if that is the kernel or OS you wish to use.

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You do not have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /, eg.

#          root (hd0,0)

#          kernel /boot/vmlinuz-version ro root=/dev/sda1

#          initrd /boot/initrd-version.img

#boot=/dev/sda

default=1

timeout=5

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.18-238.19.1.el5.centos.plus)

        root (hd0,0)

        kernel /boot/vmlinuz-2.6.18-238.19.1.el5.centos.plus ro root=LABEL=/ rhgb quiet

        initrd /boot/initrd-2.6.18-238.19.1.el5.centos.plus.img

title CentOS (2.6.18-238.19.1.el5)

        root (hd0,0)

        kernel /boot/vmlinuz-2.6.18-238.19.1.el5 ro root=LABEL=/ rhgb quiet

        initrd /boot/initrd-2.6.18-238.19.1.el5.img

Encountering cannot restore segment prot after reloc during mpirun

2011-08-09T02:00:00.005+08:00

If you encounter this error "Encountering cannot restore segment prot after reloc: permission denied " during a mpirun, this is due to the presence of SELinux setting is enabled on the server. In CentOS, you can disabled this setting by changing /etc/selinux to

SELINUX=disabled

To ensure the above commands take effect, you need to reboot.

If you do not wish to reboot just yet, You can using this command to prevent the current selinux from running

# /usr/sbin/setenforce 0

Diagnosing stuck OpenMPI jobs with mpirun

2011-08-08T11:22:00.000+08:00

I was testing a simple Openmpi jobs and somehow my jobs hangs without any output. In order to diagnose why the mpi-run failed, I've used this cool flag to

 $ mpirun --debug-daemons -np 24 -host c18,c19,c20  hello_world_mpi

..........

[c18.cluster.spms.ntu.edu.sg:09182] [[21720,0],1] orted_cmd: received exit

[c19.cluster.spms.ntu.edu.sg:13800] [[21720,0],2] orted_cmd: received exit

[c19.cluster.spms.ntu.edu.sg:13800] [[21720,0],2] orted: finalizing

[c18.cluster.spms.ntu.edu.sg:09182] [[21720,0],1] orted: finalizing

[c20.cluster.spms.ntu.edu.sg:08614] [[21720,0],3] orted_cmd: received exit

[c20.cluster.spms.ntu.edu.sg:08614] [[21720,0],3] orted: finalizing 

..........

Finally realised that my mpirun "ran away to a "IBM RNDIS/CDC ETHER" which I promptly shutdown with a ifdown usb0

After that it run smoothly

Infiniband versus Ethernet myths and misconceptions

2011-08-07T23:50:00.002+08:00

I have written my thoughts on the Infiniband versus Ethernet myths and misconceptions. For more information, see Infiniband versus Ethernet myths and misconceptions from Linux Cluster Blog

Only 3 critical myths are discussed.

Opinion 1: Infiniband is lower latency than Ethernet
Opinion 2: QDR‐IB has higher bandwidth than 10GbE
Opinion 3: IB Switch scale better than 10GbE

Most of my materials are taken from Chelsio White Paper - Eight myths about InfiniBand WP 09-10

Server Nodes File Configuration for Torque

2011-08-06T01:30:00.000+08:00

As you may be having different server hardware specification for Torque, it is quite useful to specify in the Torque $TORQUEHOME/server_priv/nodes so that you can use these features to request specific nodes when submitting jobs such as

c001 np=8     64G     
c002 np=12   126G

For more information, do look at Torque Server Node File Configuration

mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting

2011-08-05T13:56:00.001+08:00

If you see this error "mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting", it measns that there is a mis-matched kernel modules and userspace libraries.

Solution: Reinstall the OFED kernel modules. For more information, see Installing Voltaire QDR Infiniband Drivers for CentOS 5.4

If you have upgraded your linux kernel, you may want to reinstall the OFED kernel modules to the upgraded kernel or boot back to your original kernel

Understanding memory usage with top

2011-08-03T00:42:00.000+08:00

If you type "top" on the console, you will see similar information

Mem:  12299804k total,  1505328k used, 10794476k free,   196024k buffers

How do you interpret the information especially the buffer? The above server has 12GB of RAM

About 200Mbytes of memory is in the buffer. This buffering is to speed up disk access by caching written data in memory and reading in more than what we need in anticipation of the next week. If the memory is for more important requirements such as applications, it will be released immediately.