Adding and Specifying Compute Resources at Torque

This blog entry is the follow-up of Installing Torque 2.5 on CentOS 6 with xCAT tool.

After installing of Torque on the Head Node and Compute Node, the next things to do is to configure the Torque Server. In this blog entry, I will focus on the Configuring the Compute Resources at Torque Server

 Step 1: Adding Nodes to the Torque Server

# qmgr -c "create node node01"

Step 2: Configure Auto-Detect Nodes CPU Detection. Setting auto_node_np to TRUE overwrites the value of np set in $TORQUEHOME/server_priv/nodes

# qmgr -c "set server auto_node_np = True"

Step 3: Start the pbs_mom of the compute nodes, the torque server will detect the nodes automatically

# service pbs_mom start

BUG soft lockup - CPU#3 stuck for 10s! on CentOS 5

I noticed that the under intense CPU load on CentOS 5.4, I've got this bug BUG soft lockup - CPU#3 stuck for 10s! on CentOS 5.

This particular bug have been mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=484590

There are mentioned solution such as upgrading the kernel and changing the softlockup_thresh and softlockup_panic parameters at /proc/sys/kernel/softlockup_thresh and /proc/sys/kernel/softlockup_panic

I increase the /proc/sys/kernel/softlockup_thresh from 10 to 60. The original figure may be too aggressive.

# echo 30 > /proc/sys/kernel/softlockup_thresh

Check that the  /proc/sys/kernel/softlockup_panic = 0
# less  /proc/sys/kernel/softlockup_panic

Add this line to /etc/sysctl.conf (takes effect on next reboot):

kernel.softlockup_thresh=30

For more information, see

1. Why do I see "cpu soft lockup" messages in Red Hat Enterprise Linux on a Unisys E7600 or NEC 5800 Express with 96 cores?

Ganglia Node unable to update Gmetad Node

I was using CentOS 5.5 for the Ganglia Head Node and CentOS 5.7 for the Ganglia node. I followed the blog entry Installing and configuring Ganglia on CentOS 5.x , but somehow the Ganglia Head Node did not register the compute node and there is no error when I used the command

# service gmond restart

Even when I turn on the Ganglia debugging as seen in Gmond dead but subsys locked for ganglia monitoring daemon

The only hint of an issue is to go to the /usr/sbin directory and use the command

# /usr/sbin/gmond --debug=1

You will get this output

Unable to create tcp_accept_channel. Exiting

Solution

Actually I realise the version of the Ganglia package and the Ganglia package are not of the same version. My Ganglia  Head Node is v3.0.7. But my Compute Node is v3.1.7, I have to downgrade my Ganglia Node to v3.0.7

First remove the newer version

# yum remove ganglia-gmond

Install the older version

# yum install ganglia-gmond-3.0.7

Using netstat to diagnose network

Netstat is one good "swiss army knife" to look deeper into the workings linux networking.

I thought I just quickly put some netstat commands which I often used and find it helpgul in resolving networking issue.

1. Checking of networking interfaces

netstat -i

Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 95453656      0      0      0   177764      0      0      0 BMRU
lo        16436   0       70      0      0      0       70      0      0      0 LRU

 

2. Show Kernel Routing Table Information

# netstat -r 

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  1.1.57.28     *               255.255.255.128 U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default             1.1.1.125   0.0.0.0         UG        0 0          0 eth0

3. Show active listening ports ( -l) with the specific process (-p) associated with listening port

# netstat -ap

....
unix  2      [ ACC ]     STREAM     LISTENING     21138  2474/gnome-session  @/tmp/.ICE-unix/2474
unix  2      [ ACC ]     STREAM     LISTENING     23166  2674/pulseaudio     /tmp/.esd-0/socket
....

3a. To show a specific process ie ssh

# netstat -ap |grep ssh

tcp        0      0 *:ssh                       *:*                         LISTEN      1771/sshd
tcp        0     52 1.1.57.28:ssh            172.21.4.129:50591          ESTABLISHED 7837/sshd
tcp        0      0 *:ssh                       *:*                         LISTEN      1771/sshd
unix       2      [ ACC ]     STREAM     LISTENING     21646  2464/gnome-keyring- /tmp/keyring-i1zxcd/socket.ssh
unix       2      [ ]         DGRAM                    8766783 7837/sshd

4. View operational statistics for network protocol

# netstat -s

Ip:
    12311840 total packets received
    1801583 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    10510256 incoming packets delivered
    174002 requests sent out
Icmp:
    300 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 18
        echo requests: 282
    555 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 125
        echo request: 148
        echo replies: 282
...

A good resource can be found at

1.  UNIX / Linux: 10 Netstat Command Examples

Adding and removing nodes on Torque in real-time

In order to add new nodes on Torque in real0-time, the command is quite simple,

# qmgr -c "create node node"

To delete the  nodes, see Deleting Queue and Nodes in Torque in real-time