Offline Nodes in MOAB

Change State of MOAB Clients Nodes

To offline the nodes

# mnodectl -m state=drained node1

To flush the nodes

# mnodectl -m state=flush node1

To reserve the nodes

# mnodectl -m state=reserved node1

To delete nodes

# mnodectl -d node1

MOAB Job Control - mjobctl

Taken from Adaptive Computing: mjobctl

The mjobctl command controls various aspects of jobs. It is used to submit, cancel, execute, and checkpoint jobs. It can also display diagnostic information about each job. The mjobctl command enables the Moab administrator to control almost all aspects of job behavior. 

Selected examples of using mjobctl. For more information, see


1. Cancel Job

# mjobctl -c job_id

Cancel Job according to credentials or job attributes
Example A, Cancels all jobs that currently have a USERHOLD on them.

# mjobctl -c -w state=USERHOLD

Example B,  Cancels all jobs assigned to user1 or acct1.

# mjobctl -c -w user=user1 -w acct=acct1

2. Set or Release a Job Hold

Example A, Set a user hold on job job1045

# mjobctl -h user job1045

Example B, Unset all holds on job job1045

# mjobctl -u all job1045

3. Unhold a Job

Release user and system holds on job job1045.

# mjobctl -u user,system job1045

4. Execute a Job

The -w option allows flags to be set for the job. Allowable flags are, ignorepolicies, ignorenodestate, and ignorersv.
ignorepolicies - Ignore Policies
ignorenodestate - Ignore Node State
ignorersv - Ignore RSV

Example A, Execute the job1045

# mjobctl -x job1045

Example B, Execute job job1046 and ignore policies, such as MaxJobPerUser

# mjobctl -x -w flags=ignorepolicies job1046

Example C, Resume a Job

# mjobctl -r job1045

Example D, Requeue a Job

# mjobctl -R job1045

Example E, Suspend a job

# mjobctl -s job1045 

Example F, Submit a Job

# mjobctl -S job1045

5. Modify a Job's Priority

Example A, Modify a job's System Priority.

# mjobctl -p +1000 job1045

Example B, Adds 1000 points to what the priority of the job would be from normal calculation.

# mjobctl -p 1000 job1045 --flags=relative

6. Query a Job

Example A, Query job job1045.

# mjobctl -q diag job1045

Example B, Query starttime of job job1045

# mjobctl -q starttime job1045

Using mdiag -j to provides detailed information about the state of jobs

The mdiag -j command provides detailed information about the state of jobs Moab is currently tracking. This command also performs a large number of sanity and state checks.

Display user1 jobs

# mdiag -j -w user=user1

Display Idle and Running Jobs

# mdiag -j -w state=Idle,Running

Display the subjobs of an array.

# mdiag -j --xml job_id 

In the XML, the master job element contains a child element called ArraySubJobs that contains the subjobs in the array.

Display the subjobs in the array and completed sub-joys as well

# mdiag -j -v --xml job_id

For more information, read

1. MOAB Commands

Using MOAB mdiag -n to provide state of nodes

mdiag -n command provides detailed information about the state of nodes Moab or MAUi is currently tracking

Name                State  Procs     Memory         Disk          Swap      Speed  Opsys   Arch Par   Load Res Classes                        Network                        Features

Node-c00            Idle   8:8    32161:32161       1:1       62862:64158   2.10  linux [NONE] DEF   0.00 000 [rambutan_8:8][queue_8:8][lemo [DEFAULT]    
.....
.....
Node-c03            Busy   0:8    32161:32161       1:1       62735:64158   2.10  linux [NONE] DEF   8.00 001 [rambutan_8:8][queue_0:8][queue [DEFAULT]
.....
.....

Interesting Columns that I would find especially useful is the STATE, Procs (Available Core), Swap and Load

You can further refine the search

# mdiag -n |grep Busy
# mdiag -n|grep Idle

MOAB and storing of statistics with ODBC

This documentation ias a useful "How-To" to show how to set up and configure Moab to connect to a MySQL database using the MySQL ODBC driver.

For more information, do look at MOAB Workload Manager, 22.2 ODBC

Jobs submitted placed on the queue and not able to run on HPC

Sometimes, when out users submit job via MOAB using Torque as a resource manager, their jobs is just not able to run even though there are resources to accomdoate the run. Of course there are many ways to troubleshoot. Here is how I troubleshoot 1 of them.

Step 1: Use the command checkjob on the process ID of the job that is stuck.
# checkjob 100001
--------------------------------------
Partition List:      xxxxxxxx
Flags:                 RESTARTABLE
Attr:                   checkpoint
StartPriority:       94
rejected for CPU              - (null)
rejected for State              - (null)
NOTE: job req cannot run in partition xxxxx (available procs do not meet requirements : 0 of 32 procs found)
---------------------------------------
Here's the key hint under NOTE. There is no node suitable for this user. But why? Launch another command


# qstat -f 100001
----------------------------------
euser = xxxxx
egroup = xxxxx
queue_rank = xxxxx
queue_type = xxxxx
etime = xxxxxxxx
submit_args = -1 nodes2:ppn=16 ./run.sh
-----------------------------------
Note the key issue is that users submitted a job requesting 2 nodes with 16 cores each. This does not exist in our cluster configuration. Hence it is not able to run....

Using MOAB and Torque Commands together to analyse the problem is useful indeed.

pbs node manipulation Revision

It has been a long time since I used torque. I though I pen down some common commands I will used

• pbsnodes -l (List node names and their state. If no state is specified, only nodes in the DOWN, OFFLINE, or UNKNOWN states are listed. Specifying a state string acts as an output filter. Valid state strings are "active", "all", "busy", "down", "free", "offline", "unknown", and "up". )
  
• pbsnodes -o (Add the OFFLINE state. This is different from being marked DOWN. OFFLINE prevents new jobs from running on the specified nodes. This gives the administrator a tool to hold a node out of service without changing anything else. The OFFLINE state will never be set or cleared automatically by pbs_server; it is purely for the manager or operator.)
  
• pbsnodes -c (Clear OFFLINE from listed nodes)
  
• pbsnodes -a (All attributes of a node or all nodes are listed. This is the default if no flag is given)
  

Learning MOAB - Using showq

showq displays information about queued jobs including active, eligible, blocked, and/or recently completed jobs. Do note that the -c, -i, and -r flags can be used  Administrator only

Note 1: Display blocked jobs only
> showq -b

Note 2: Display details about recently jobs (by MOAB Admin)
# showq -c

Note 3: Display local and full resource manager job ids. If specified with the -i option, will display job reservation time. (If using MOAB
# show -vi

Note 4: Display jobs associated with specified constraint. Valid constraint include user, group, account, class and qos
> showq -w user=john

Note 5: Display extended details about active (running jobs)
# show -r
------------------------------------------------------
S - Job State. Either "R" for Running or "S" for Starting.
PAR - Partition in which job is running.
EFFIC - CPU efficiency of job.
XFACTOR - Current expansion factor of job, where XFactor = (QueueTime + WallClockLimit) / WallClockLimit
Q - Quality Of Service specified for job.
USERNAME - User owning job.
GROUP - Primary group of job owner.
MHOST - Master Host running primary task of job.
PROC  - Number of processors being used by the job.
--------------------------------------------------------

Learning MOAB. What is workload?

When you are doing High Performance Computing or Cloud, the concept of workload is very important. Basically there are 4 kinds of workloads (Taken and summarise from MOAB Admin Guide 5.2)

1. Batch Workload - With a Batch Job, the job is submitted to a job queue, and is run somewhere on the cluster as resources becomes available
2. Interactive Workload - Requestors are interested in immediate response and are generally waiting for interactive request to be executed before going on to other activities. To manage interactive jobs, the focus is usually on setting aside resources to guarantee immediate execution or at least a minimal wait time for interactive jobs
3. Calendar Workload - Calendar workload must be executed at a particular time and possibly in a regular periodic manner. There must be guarantee of resource availability at the needed time to allow calendar jobs to run as required.
4.  Service Workload - Service workload processes externally generated transaction requests while a scheduler or resource allocation mechanism provides the distributed service with needed resources to meet target backlog or response targets to the service. Examples are web farms etc. 

MOAB Adaptive Datacenter and Cloud Solution

If you are considering MOAB solution for Cloud Computing, here 4 sets of online video which might help in your decision

1. Adaptive DataCenter
2. Key Benefits
3. Internal Hosting
4. External Hosting

Happy viewing.....

MOAB and Cloud Computing

For those who are keen on Cloud Computing and how MOAB can make the HPC Cloud-like. Go to Chris's Blog on MOAB. Excellent Resource

Archive for the ‘Moab’ Category

MOAB Installation and Configuration

Important MOAB Documentation:

1. Building and Installing Moab
2. Initial Configuration
3. Credential Overview

MOAB online Documentation (FULL):

1. MOAB online Documentation