Linux Toolkits

Tuesday, June 25, 2013

Platform LSF – Submitting and Controlling jobs

I thought I list out some useful commands that can be used for for viewing a cluster using an LSF Cluster. Please read the manual for more in-depth information. Taken from Platform LSF 8.3 Quick References.

**Submitting and Controlling jobs**
bbot	Moves a pending job relative to the last job in the queue
bchkpnt	Checkpoints a checkpointable job
bkill	Send a signal to a job
bmig	Migrates a checkpointable or rerunnable job
bmod	Modifies job submission options
brequeue	Kills and requeues a job
bresize	Releases slots and cancels pending job resize allocation requests
brestart	Restarts a checkpointed job
bresume	Resumes a suspended job
bstop	Suspends a job
bsub	Submits a job
bswitch	Moves unfinished jobs from one queue to another
btop	Moves a pending job relative to the first job in the queue

Other References:

Friday, June 21, 2013

Error on Torque .... received SISTER_EOF attempting to communicate with sister MOM's

Taken from https://nesccdocs.rdhpcs.noaa.gov/wiki/index.php/Frequently_Asked_Questions

When you experience the error below. According to the FAQ,

My job crashed with PBS: job killed: node 1 ($NODENAME) requested job terminate
, 'EOF' (code 1099) - received SISTER_EOF attempting to communicate with 
sister MOM's

This could be caused by a failed node.

But recently I'm seeing this error from a code run and the issue as that the code took took too much memory and the result was a node hang. The above result ensues.

Thursday, June 20, 2013

Checking status of nodes for Torque

pbsnodes is a very useful command to check status of nodes. One of my favourite usage is check list the status of the nodes

# pbsnodes -l free
node-c00        free
node-c01        free

Other option includes "active", "all", "busy", "down", "free", "offline", "unknown", and "up"

# pbsnodes -a
 node-c00
     state = free
     np = 8
     properties = pulasan
     ntype = cluster
     jobs = 0/31516.starfruit-h00
     status = rectime=1371836955,varattr=,jobs=31516.node-c00,state=free,
netload=1810185981631,gres=,loadave=1.00,ncpus=8,physmem=16436612kb,
availmem=27307052kb,totmem=28626300kb,idletime=5540229,nusers=1,nsessions=1,
sessions=18708,uname=Linux node-c00.linuxtoolkit.blogspot.com 2.6.18-164.el5 
#1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64,opsys=linux

All attributes of a node or all nodes are listed. This is the default if no flag is given

Clear OFFLINE from listed nodes

# pbsnodes -c node-c00

Add the OFFLINE state

# pbsnodes -o node-c00

Saturday, June 15, 2013

Enabling SRIOV in BIOS for IBM Servers and Blade Servers

Step 1: Power on the system, and press F1 to enter the Setup utility.

Step 2: Select System Settings and then Network.

Step 3: Under the Network Device List, select the device to be configured and press Enter to see all the Network Device options (Figure 1).

Step 4: Select the device’s description and press Enter to configure the device (Figure 2)

Step 5: From the selection menu, select Advanced Mode and press Enter to change the value (Figure 3).

Step 6: Choose Enable and press Enter.

Step 7: On the same selection menu, select Controller Configuration and press Enter to enter the configuration menu.

Step 8: Select Configure SRIOV and hit Enter.

Step 9: On the Configure SRIOV page, press Enter to toggle the values

Step 10: Select Enable and press Enter

Step 11: Select Save Current Configurations and press Enter.

Step 12: Press Esc to exit the menu. Then, click Save to save the configuration.

Step 13: Reboot the system.

Thursday, June 13, 2013

Platform LSF – Monitoring jobs and tasks

**Monitoring jobs and tasks**
bacct	Reports accounting statistics on completed LSF jobs
bapp	Displays information about jobs attached to application profiles
bhist	Displays historical information about jobs
bjobs	Displays information about jobs
bpeek	Displays stdout and stderr of unfinished jobs
bsla	Displays information about service class configuration for goal-oriented service-level agreement (SLA) scheduling
bstatus	Reads or sets external job status messages and data files