Platform LSF – Submitting and Controlling jobs

I thought I list out some useful commands that can be used for for viewing a cluster using an LSF Cluster. Please read the manual for more in-depth information. Taken from Platform LSF 8.3 Quick References.

Submitting and Controlling jobs

bbot	Moves a pending job relative to the last job in the queue
bchkpnt	Checkpoints a checkpointable job
bkill	Send a signal to a job
bmig	Migrates a checkpointable or rerunnable job
bmod	Modifies job submission options
brequeue	Kills and requeues a job
bresize	Releases slots and cancels pending job resize allocation requests
brestart	Restarts a checkpointed job
bresume	Resumes a suspended job
bstop	Suspends a job
bsub	Submits a job
bswitch	Moves unfinished jobs from one queue to another
btop	Moves a pending job relative to the first job in the queue

Other References:

1. Platform LSF – Monitoring jobs and tasks
2. Platform LSF – Administration and Accounting Commands
3. Platform LSF - View information about cluster
4. Platform LSF - Submitting and Controlling jobs

Error on Torque .... received SISTER_EOF attempting to communicate with sister MOM's

Taken from https://nesccdocs.rdhpcs.noaa.gov/wiki/index.php/Frequently_Asked_Questions

When you experience the error below. According to the FAQ,

My job crashed with PBS: job killed: node 1 ($NODENAME) requested job terminate
, 'EOF' (code 1099) - received SISTER_EOF attempting to communicate with 
sister MOM's

This could be caused by a failed node.

But recently I'm seeing this error from a code run and the issue as that the code took took too much memory and the result was a node hang. The above result ensues.

Checking status of nodes for Torque

pbsnodes is a very useful command to check status of nodes. One of my favourite usage is check list the status of the nodes

# pbsnodes -l free
node-c00        free
node-c01        free

Other option includes "active", "all", "busy", "down", "free", "offline", "unknown", and "up"

# pbsnodes -a
 node-c00
     state = free
     np = 8
     properties = pulasan
     ntype = cluster
     jobs = 0/31516.starfruit-h00
     status = rectime=1371836955,varattr=,jobs=31516.node-c00,state=free,
netload=1810185981631,gres=,loadave=1.00,ncpus=8,physmem=16436612kb,
availmem=27307052kb,totmem=28626300kb,idletime=5540229,nusers=1,nsessions=1,
sessions=18708,uname=Linux node-c00.linuxtoolkit.blogspot.com 2.6.18-164.el5 
#1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64,opsys=linux

All attributes of a node or all nodes are listed. This is the default if no flag is given

Clear OFFLINE from listed nodes

# pbsnodes -c node-c00

Add the OFFLINE state

# pbsnodes -o node-c00

Enabling SRIOV in BIOS for IBM Servers and Blade Servers

Step 1: Power on the system, and press F1 to enter the Setup utility.

Step 2: Select System Settings and then Network.

Step 3: Under the Network Device List, select the device to be configured and press Enter to see all the Network Device options (Figure 1).


Step 4: Select the device’s description and press Enter to configure the device (Figure 2)


Step 5: From the selection menu, select Advanced Mode and press Enter to change the value (Figure 3).


Step 6: Choose Enable and press Enter.

Step 7: On the same selection menu, select Controller Configuration and press Enter to enter the configuration menu.

Step 8: Select Configure SRIOV and hit Enter.

Step 9: On the Configure SRIOV page, press Enter to toggle the values

Step 10: Select Enable and press Enter

Step 11: Select Save Current Configurations and press Enter.

Step 12: Press Esc to exit the menu. Then, click Save to save the configuration.

Step 13: Reboot the system.

Platform LSF – Monitoring jobs and tasks

I thought I list out some useful commands that can be used for for viewing a cluster using an LSF Cluster. Please read the manual for more in-depth information. Taken from Platform LSF 8.3 Quick References.

Monitoring jobs and tasks

bacct	Reports accounting statistics on completed LSF jobs
bapp	Displays information about jobs attached to application profiles
bhist	Displays historical information about jobs
bjobs	Displays information about jobs
bpeek	Displays stdout and stderr of unfinished jobs
bsla	Displays information about service class configuration for goal-oriented service-level agreement (SLA) scheduling
bstatus	Reads or sets external job status messages and data files

Other References:

1. Platform LSF – Monitoring jobs and tasks
2. Platform LSF – Administration and Accounting Commands
3. Platform LSF - View information about cluster
4. Platform LSF - Submitting and Controlling jobs 

MOAB Job Control - mjobctl

Taken from Adaptive Computing: mjobctl

The mjobctl command controls various aspects of jobs. It is used to submit, cancel, execute, and checkpoint jobs. It can also display diagnostic information about each job. The mjobctl command enables the Moab administrator to control almost all aspects of job behavior. 

Selected examples of using mjobctl. For more information, see


1. Cancel Job

# mjobctl -c job_id

Cancel Job according to credentials or job attributes
Example A, Cancels all jobs that currently have a USERHOLD on them.

# mjobctl -c -w state=USERHOLD

Example B,  Cancels all jobs assigned to user1 or acct1.

# mjobctl -c -w user=user1 -w acct=acct1

2. Set or Release a Job Hold

Example A, Set a user hold on job job1045

# mjobctl -h user job1045

Example B, Unset all holds on job job1045

# mjobctl -u all job1045

3. Unhold a Job

Release user and system holds on job job1045.

# mjobctl -u user,system job1045

4. Execute a Job

The -w option allows flags to be set for the job. Allowable flags are, ignorepolicies, ignorenodestate, and ignorersv.
ignorepolicies - Ignore Policies
ignorenodestate - Ignore Node State
ignorersv - Ignore RSV

Example A, Execute the job1045

# mjobctl -x job1045

Example B, Execute job job1046 and ignore policies, such as MaxJobPerUser

# mjobctl -x -w flags=ignorepolicies job1046

Example C, Resume a Job

# mjobctl -r job1045

Example D, Requeue a Job

# mjobctl -R job1045

Example E, Suspend a job

# mjobctl -s job1045 

Example F, Submit a Job

# mjobctl -S job1045

5. Modify a Job's Priority

Example A, Modify a job's System Priority.

# mjobctl -p +1000 job1045

Example B, Adds 1000 points to what the priority of the job would be from normal calculation.

# mjobctl -p 1000 job1045 --flags=relative

6. Query a Job

Example A, Query job job1045.

# mjobctl -q diag job1045

Example B, Query starttime of job job1045

# mjobctl -q starttime job1045

Adding and Removing the 2nd Mellanox Ethernet Port as an uplink to an Existing Vswitch using the CLI

At VSphere 5.1 Client. I was able to see the Dual-Port Network Adapter (vmnic22.p1, vmnic22.p2) after I install the Vmware Installing Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server.

But somehow I am not able to use the 2nd port of the Mellanox ConnectX 10G on the VSphere Client > Configuration > Networking. It will not be visible. However at the VSphere Client > Configuration > Networking > Add Networking, I not able to see the 2nd Port being available.

I found the document from Mellanox (MellanoxMLX4_ENDriverforVMwareESXi-5.xREADME) which is useful to resolve the issue.

At Page 10, Adding the Device as an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permission

Step 2: Add an uplink to a vswitch, run:

# esxcli network vswitch standard uplink add –u <uplink_name> -v <vswitch_name>

* Uplink_name refer to the name used by ESX for the network Adapter. For example, vmnic22.p2 is the uplink name

Step 3: Check that uplink was added successfully. Run:

# esxcli network vswitch standard list -v <vswitch_name>

Removing the Device an an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permissions Step 2: Remove an uplink from a vswitch, run:

# esxcli network vswitch standard uplink remove -u <uplink_name> -v <vswitch_name>

Using mdiag -j to provides detailed information about the state of jobs

The mdiag -j command provides detailed information about the state of jobs Moab is currently tracking. This command also performs a large number of sanity and state checks.

Display user1 jobs

# mdiag -j -w user=user1

Display Idle and Running Jobs

# mdiag -j -w state=Idle,Running

Display the subjobs of an array.

# mdiag -j --xml job_id 

In the XML, the master job element contains a child element called ArraySubJobs that contains the subjobs in the array.

Display the subjobs in the array and completed sub-joys as well

# mdiag -j -v --xml job_id

For more information, read

1. MOAB Commands

Helping users to SSH without password into the Compute Nodes manually

There are occasionally in a cluster environment that users accidentally delete their head node SSH keys and later cannot submit their jobs to the queue or their MPI jobs cannot scale beyond 1 node. The system you will see when you turn on the verbose method

To conduct a quick test,

# ssh -v remote-host

you will see an errors similar to  such as those below:

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

OR

debug1: Miscellaneous failure
No credentials cache found

To reinstate the password-less access to compute nodes, you have to do the following. First thing first, please do backup files at your ~/.ssh/

Step 1: Regenerate the SSH keys
Auto SSH Login without Password

Step 2: Append the public keys ~/.ssh/id_rsa.pub and put into the ~/.ssh/authorized_keys

# cd ~/.ssh/

# cat id_rsa.pub >> authorized_keys

# chmod 400 /home/myuser/.ssh/authorized_keys

Step 3: Try ssh into the compute nodes. It should be clear password-less access to all nodes.

Upgrading Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server

Do read the Blog Entry Installing Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server

Step 1: At VMware ESX 5.x Hypervisor,

• Click the F2 button
• Select "Troubleshoot Options"
• "Enable ESXi Shell" and "Enable SSH"

Step 2: Download the VMware ESXi 5.0 Driver for Mellanox ConnectX Ethernet Adapters

Step 3: Unzip the mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip

Step 4: The upgrade process is similar to a new install, except the command that should be issued is the following:

#   esxcli software vib upgrade -v {VIBFILE}

In the example above, this would be:

#  esxcli software vib upgrade -v 
/tmp/net-mlx4-en-1.6.1.2-1OEM.500.0.0.406165.x86_64.vib