MAUI secondary client lost connection to primary MAUI host

If you are using Secondary Submission Node for torque and wish to use MAUI as a scheduler,  you may encounter the error  

ERROR:    lost connection to server
ERROR:    cannot request service (status)

The solution is quite simple, make sure the the time are the same. This is the most common cause of lost connection.

Related Information:

1. Bad UID for job execution MSG=ruserok failed validating user1 from ServerNode while configuring Submission Node in Torque

Summary of Job Management Commands for MAUI

This is a good summary taken from Adaptive Computing 4.3 Job Managament Commands to manage jobs for MAUI.

Command	Flags	Description
canceljob		cancel existing job
checkjob		display job state, resource requirements, environment, constraints, credentials, history, allocated resources, and resource utilization
diagnose	-j	display summarized job information and any unexpected state
releasehold	[-a]	remove job holds or defers
runjob		start job immediately if possible
sethold		set hold on job
setqos		set/modify QoS of existing job
setspri		adjust job/system priority of job

Deleting PBS and MAUI Jobs which cannot be purged

If the Compute Node pbs_mom is lost and cannot be recovered (due to hardware or network failure) and to purge a running job from the qstat output or show 1. Shutdown the pbs_server daemon on the PBS Server

# service pbs_server stop

2. Remove Job Spool Files that holds the hanged JobID (For example 4444)

# rm /var/spool/torque/server_priv/jobs/4444.headnode.SC
# rm /var/spool/torque/server_priv/jobs/4444.headnode.JB

3. Start the pbs_Server Daemon

# service pbs_server start

4. Restart the MAUI Daemon

# service maui restart

References:

1. Deleting PBS/Maui Jobs

Using MOAB mdiag -n to provide state of nodes

mdiag -n command provides detailed information about the state of nodes Moab or MAUi is currently tracking

Name                State  Procs     Memory         Disk          Swap      Speed  Opsys   Arch Par   Load Res Classes                        Network                        Features

Node-c00            Idle   8:8    32161:32161       1:1       62862:64158   2.10  linux [NONE] DEF   0.00 000 [rambutan_8:8][queue_8:8][lemo [DEFAULT]    
.....
.....
Node-c03            Busy   0:8    32161:32161       1:1       62735:64158   2.10  linux [NONE] DEF   8.00 001 [rambutan_8:8][queue_0:8][queue [DEFAULT]
.....
.....

Interesting Columns that I would find especially useful is the STATE, Procs (Available Core), Swap and Load

You can further refine the search

# mdiag -n |grep Busy
# mdiag -n|grep Idle

Issues arising when node has muliple queue with Torque

I noticed that for Torque/MAUI, when the compute nodes belong to different queues, there could be a tendency where Torque/MAUI could conclude that the resource pool does not have sufficient resources.

I'm using Torque 2.5.3 / MAUI 3.3.1 version

Take for example, in /var/spool/torque/server_priv/nodes, if your nodes belong to

node01 np=8 queue1 queue2
node02 np=8 queue1 queue2
node03 np=8 queue2 queue3
node04 np=8 queue2 queue3

If you submit a job to queue2, something like

$ qsub -q queue2 -l nodes=3:ppn=8 openmpi.sh -v file=my_mpi_file

Based on the resources in queue2, there should be enough, but somehow MAUI will see that the resoruce is not enough. One of the best way to identify is to see the issue is to use checkjob. See Using MAUI checkjob command 

It is recommended that compute resource is tagged to one queue to prevent Torque/MAUI miscalculation

Using MAUI checkjob command

The command checkjob is very useful if you are using MAUI or MOAB to identify jobs issues

1. One of my favourite commands to check why a Job is IDLE is

$ checkjob -v 6235

The above commands display reasons why idle job is blocked ignoring node state and current node utilization constraints

node-c00              accepted : 8 tasks supported
node-c02              accepted : 8 tasks supported
node-c03              accepted : 8 tasks supported
node-c04              accepted : 8 tasks supported
node-c05              accepted : 8 tasks supported
node-c06              accepted : 8 tasks supported
node-c07              rejected : CPU
node-c08              rejected : CPU
node-c09              rejected : CPU

Some of the reasons why it is rejected can be due to. The list is not exhausted.

Features - queue or features does not meet requirement 
CPU - CPU does not match requirements
State - Used by other jobs

For more information, see Adaptive Computing checkjob. Do note if you are using MAUI some of the checkjob features are available in the commercial scheduler.

Dealing with stuck jobs and Torque and MAUI

This is a add-on for the blog entry "Manually Deleting Torque amd PBS jobs using MAUI"

1. Force the Torque Server or MOM to send an obituary of the job ID to the server

# qsig -s 0 job_id

2. Using the momctl command on the compute nodes where the job is listed. You can use a tracejob to check which nodes the job has been send to

# momctl -c job_id -h compute_node_1

3i. Setting the qmgr server setting mom_job_sync to True might help prevent jobs from hanging.

# qmgr -c "set server mom_job_sync = True"

3ii. To verify that the setting in 3i is in, you can use trhe command

# qmgr -c "p s"

4. The final option. If all else fail, do a

qdel -p job_id

For more information, see Adaptive Computing Website Section 11.1.7 Stuck Jobs

Manually Deleting Torque amd PBS jobs using MAUI

Tracing Jobs
To trace a job with MAUI commands including the nodes the jobs are residing, you can use the commands

# showq -r

Alternatively, you can use the MAUI commands to trace the job activity

# trace job_id

Deleting Jobs
To delete a job with MAUI commands, you can use the commands,

# canceljob job_id

Alternatively, you can also use PBS commands to delete a job

# qdel job_id

PBS mom control
If not able to delete a stale job which has no process, you can use the momctl command

# momctl

If you are unable to delete the stale job with has no process, you can use momctl to do diagnostic. Basically The momctl command allows remote shutdown, reconfiguration, diagnostics, and querying of the pbs_mom daemon. For more information on momctl, do look at momctl by http://www.clusterresources.com/:

Example 1: Diagnosis of pbs_mom

# momctl -h node1 -d 1

Example 2: Cycle the pbs_mom on node 1

# momctl -h nod1 -C

Manually deleting the jobs
To manually delete the jobs, you should shutdown the pbs server

# service pbs_server stop

Remove the job spool files

# rm /var/spool/pbs/server_priv/jobs/111.host.SC 
# rm /var/spool/pbs/server_priv/jobs/111.host.JB

Restart the pbs_server

# service pbs_server restart

Further Information:

1. Deleting PBS/Maui Jobs

MAUI: Cannot send request to server ...... (server may not be running)

ERROR: cannot send request to server ................42559 (server may not be running)
ERROR: cannot request service (status)

A few assumptions for this blog entry. You are using Torque and MAUI only for the resource manager and scheduler respectively.

Several things to check.

1. Make sure your /etc/hostname reflect the correct entries for all the head and compute nodes
2. For the Torque configuration, do follow the configuration seen in the blog entry Setting up Torque Server on xCAT 2.x. Pay particular attendtion to Step 2 and 3
3. For the MAUI configuration, do follow the configuration seen in the blog entry MAUI Installation in xCAT 2.x
4. Check the logs for MAUI at /usr/local/maui/log

After following Pointer 1-4, you are still unable to solve, you can get some hints if you can go to

# cd /opt/maui/sbin/

and type

./maui

In my particular situation, I encounter server must be started on host 'xxx.xxx.xxx.xxx.xxx' (currently on 'xxx') which is my short-name. I resolve it quickly by going to /usr/local/maui/maui.cfg and edit the

SERVERHOST = xxxx

Overview of MAUI Scheduler Commands

Finally, I thought I summarise the documentation of MAUI Schduler Commands in one nice blog entry "Overview of MAUI Scheduler Commands". Hopefully, this will help summarise the commands used for system administration of the MAUI Scheduler

The concept of Processor Equivalent

Learned a new concept of Process Equivalent or PE. It is a measure of the actual impact of a set of requested resources by a job on the total resources system wide.

Taken from MAUI Admin Guide by Cluster Resources

PE =MAX (ProcsRequestedByJob / TotalConfiguredProcs,
MemoryRequestedByJob / TotalConfiguredMemory,
DiskRequestedByJob / TotalConfiguredDisk,
SwapRequestedByJob / TotalConfiguredSwap) * TotalConfiguredProcs

Assume a homogeneous 100 node system with 4 processors and 1 GB of memory per node. A job is submitted requesting 2 processors and 768 MB of memory. The PE for this job would be calculated as:

PE = MAX(2/(100*4), 768/(100*1024)) * (100*4) = 3.

This result makes sense since the job would be consuming 3/4 of the memory on a 4 processor node. The calculation works equally well on homogeneous or heterogeneous systems, uniprocessor or large way SMP systems.

MAUI Installation in xCAT 2.x

What is MAUI?
Maui Cluster Scheduler (a.k.a. Maui Scheduler) is our first generation cluster scheduler. Maui is an advanced policy engine used to improve the manageability and efficiency of machines ranging from clusters of a few processors to multi-teraflop supercomputers.

Taken and modified from https://xcat.wiki.sourceforge.net/Maui


Step 1: Download the tarball

• Go to http://clusterresources.com/, create an account and download here:
• http://www.clusterresources.com/product/maui/index.php?
• Untar in /tmp

Step 2: Setup Torque for xCAT

# cd /opt/torque
# ln -s x86_64/bin .
# ln -s x86_64/lib .
# ln -s x86_64/sbin .
# export PATH=$PATH:/opt/torque/x86_64/bin/ 

Step 3: Setup MAUI

# cd maui-3.2.6p21
# ./configure --prefix=/opt/maui --with-pbs=/opt/torque/
# make -j8
# make install
# cp /opt/xcat/share/xcat/netboot/add-on/torque/moab /etc/init.d/maui
(Edit /etc/init.d/maui so that all MOAB is MAUI and all moab becomes maui)
# service start maui
# chkconfig --level 345 maui on

Step 4: Configure MAUI

# touch /etc/profile.d/maui.sh
# vim maui (Type: export PATH=$PATH:/opt/maui/bin)
# source /etc/profile.d/maui.sh
# vim /usr/local/maui/maui.cfg
Change: RMCFG[] TYPE=PBS@...@ to: RMCFG[] TYPE=PBS
# service maui restart

Now run:

# showq

You should see all of the processors. Next try running a job to make sure that maui picks it up.