MOAB Job Control - mjobctl

Taken from Adaptive Computing: mjobctl

The mjobctl command controls various aspects of jobs. It is used to submit, cancel, execute, and checkpoint jobs. It can also display diagnostic information about each job. The mjobctl command enables the Moab administrator to control almost all aspects of job behavior. 

Selected examples of using mjobctl. For more information, see


1. Cancel Job

# mjobctl -c job_id

Cancel Job according to credentials or job attributes
Example A, Cancels all jobs that currently have a USERHOLD on them.

# mjobctl -c -w state=USERHOLD

Example B,  Cancels all jobs assigned to user1 or acct1.

# mjobctl -c -w user=user1 -w acct=acct1

2. Set or Release a Job Hold

Example A, Set a user hold on job job1045

# mjobctl -h user job1045

Example B, Unset all holds on job job1045

# mjobctl -u all job1045

3. Unhold a Job

Release user and system holds on job job1045.

# mjobctl -u user,system job1045

4. Execute a Job

The -w option allows flags to be set for the job. Allowable flags are, ignorepolicies, ignorenodestate, and ignorersv.
ignorepolicies - Ignore Policies
ignorenodestate - Ignore Node State
ignorersv - Ignore RSV

Example A, Execute the job1045

# mjobctl -x job1045

Example B, Execute job job1046 and ignore policies, such as MaxJobPerUser

# mjobctl -x -w flags=ignorepolicies job1046

Example C, Resume a Job

# mjobctl -r job1045

Example D, Requeue a Job

# mjobctl -R job1045

Example E, Suspend a job

# mjobctl -s job1045 

Example F, Submit a Job

# mjobctl -S job1045

5. Modify a Job's Priority

Example A, Modify a job's System Priority.

# mjobctl -p +1000 job1045

Example B, Adds 1000 points to what the priority of the job would be from normal calculation.

# mjobctl -p 1000 job1045 --flags=relative

6. Query a Job

Example A, Query job job1045.

# mjobctl -q diag job1045

Example B, Query starttime of job job1045

# mjobctl -q starttime job1045

Adding and Removing the 2nd Mellanox Ethernet Port as an uplink to an Existing Vswitch using the CLI

At VSphere 5.1 Client. I was able to see the Dual-Port Network Adapter (vmnic22.p1, vmnic22.p2) after I install the Vmware Installing Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server.

But somehow I am not able to use the 2nd port of the Mellanox ConnectX 10G on the VSphere Client > Configuration > Networking. It will not be visible. However at the VSphere Client > Configuration > Networking > Add Networking, I not able to see the 2nd Port being available.

I found the document from Mellanox (MellanoxMLX4_ENDriverforVMwareESXi-5.xREADME) which is useful to resolve the issue.

At Page 10, Adding the Device as an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permission

Step 2: Add an uplink to a vswitch, run:

# esxcli network vswitch standard uplink add –u <uplink_name> -v <vswitch_name>

* Uplink_name refer to the name used by ESX for the network Adapter. For example, vmnic22.p2 is the uplink name

Step 3: Check that uplink was added successfully. Run:

# esxcli network vswitch standard list -v <vswitch_name>

Removing the Device an an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permissions Step 2: Remove an uplink from a vswitch, run:

# esxcli network vswitch standard uplink remove -u <uplink_name> -v <vswitch_name>

Using mdiag -j to provides detailed information about the state of jobs

The mdiag -j command provides detailed information about the state of jobs Moab is currently tracking. This command also performs a large number of sanity and state checks.

Display user1 jobs

# mdiag -j -w user=user1

Display Idle and Running Jobs

# mdiag -j -w state=Idle,Running

Display the subjobs of an array.

# mdiag -j --xml job_id 

In the XML, the master job element contains a child element called ArraySubJobs that contains the subjobs in the array.

Display the subjobs in the array and completed sub-joys as well

# mdiag -j -v --xml job_id

For more information, read

1. MOAB Commands

Helping users to SSH without password into the Compute Nodes manually

There are occasionally in a cluster environment that users accidentally delete their head node SSH keys and later cannot submit their jobs to the queue or their MPI jobs cannot scale beyond 1 node. The system you will see when you turn on the verbose method

To conduct a quick test,

# ssh -v remote-host

you will see an errors similar to  such as those below:

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

OR

debug1: Miscellaneous failure
No credentials cache found

To reinstate the password-less access to compute nodes, you have to do the following. First thing first, please do backup files at your ~/.ssh/

Step 1: Regenerate the SSH keys
Auto SSH Login without Password

Step 2: Append the public keys ~/.ssh/id_rsa.pub and put into the ~/.ssh/authorized_keys

# cd ~/.ssh/

# cat id_rsa.pub >> authorized_keys

# chmod 400 /home/myuser/.ssh/authorized_keys

Step 3: Try ssh into the compute nodes. It should be clear password-less access to all nodes.

Upgrading Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server

Do read the Blog Entry Installing Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server

Step 1: At VMware ESX 5.x Hypervisor,

• Click the F2 button
• Select "Troubleshoot Options"
• "Enable ESXi Shell" and "Enable SSH"

Step 2: Download the VMware ESXi 5.0 Driver for Mellanox ConnectX Ethernet Adapters

Step 3: Unzip the mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip

Step 4: The upgrade process is similar to a new install, except the command that should be issued is the following:

#   esxcli software vib upgrade -v {VIBFILE}

In the example above, this would be:

#  esxcli software vib upgrade -v 
/tmp/net-mlx4-en-1.6.1.2-1OEM.500.0.0.406165.x86_64.vib