Inappropriate ioctl for device MSG=cannot create job file for Torque

I encountered this error on the cluster.

qsub: submit error (PBS_Server System error: 
Inappropriate ioctl for device MSG=cannot create job file 
/var/spool/torque/server_priv/jobs/497741.headnode-h00.cluster.com 
(28 - No space left on device)) 

I did a df -h and notice that there is still space. But when I did a df -i, I've noticed all the spaces iUSE% is almost 100% used up. To bring up the

# find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

References:

1. Find where inodes are being used

Error qmgr obj= svr=default: Bad ACL entry in host list MSG=First bad host

I encountered this error when following Torque Administratrion Guide. To mitigate the error, remember to put in the PATH

export PATH=$PATH:/opt/torque/x86_64.bin:/opt/torque/x86_64/sbin

Remember to source the file :)

If after the above insert, it does not work, just do it manually

pbs_server -t create

Manually check that the pbs_server database is created

[root@headenode torque-4.2.10]# ps -afe|grep "pbs_server -t create"
root     26644     1  0 16:06 ?        00:00:01 pbs_server -t create
root     30318  2682  0 16:21 pts/0    00:00:00 grep pbs_server -t create

Unable to Submit via Torque Submission Node - Socket_Connect Error for Torque 4.2.7

I am using Torque Server version 4.2.7. I was trying to configure a Submission Node. Here are a sample of my qmgr -c 'p s" output. Firewall has allows the necessary traffic in outr

# qmgr -c "p s"
.......... 
set server acl_hosts = submission_node.cluster.spms.ntu.edu.sg
set server acl_hosts += head_node.cluster.spms.ntu.edu.sg
set server submit_hosts = submission_node.cluster.spms.ntu.edu.sg
set server submit_hosts += head_node.cluster.spms.ntu.edu.sg
set server allow_node_submit = True 
.......

After we ssh into the submission_node, and as I simulate as a user, I got this errors. Yes, the submission_node has been configured as a conventional client.

socket_connect error (VERIFY THAT trqauthd IS RUNNING)
Error in connection to trqauthd (15137)-[could not connect to unix socket /tmp/trqauthd-unix: 111]
socket_connect error (VERIFY THAT trqauthd IS RUNNING)
Error in connection to trqauthd (15137)-[could not connect to unix socket /tmp/trqauthd-unix: 111]
socket_connect error (VERIFY THAT trqauthd IS RUNNING)
Error in connection to trqauthd (15137)-[could not connect to unix socket /tmp/trqauthd-unix: 111]
Unable to communicate with head_node(10.10.10.20)
Communication failure. qsub: cannot connect to server head_node (errno=15137) could not connect to trqauthd

Taking a look at the Torque 4.2.7 documentation, the documentation mentioned that you have to make sure the submission node have trqauthd script at /etc/init.d if you are  using RH / CentOS. You can easily scp the /etc/init.d/trqauthd to the submision node

From the head_node

# scp -v /etc/init.d/trqauthd root@submssion_node:/etc/init.d/

Create a /etc/hosts_equiv file

# touch /etc/hosts_equiv

Put the Submission_Node file name at the /etc/hosts.equiv of the head_node

submission_node 

At the Submission_Node, start the  trqauthd service

# service trqauthd start

Now trying submitting as a normal user

Forcibly kill or purge the Job in the Torque Scheduler

When there is a job stuck and cannot be remove by a normal qdel, you can use the command qdel -p jobid. Do note that this command should be used when there is no other way to kill off the job in the usual fashion especially if the compute node is unresponsive.

# qdel -p jobID

References:

1. [torqueusers] qdel will not delete

Checking for Torque Server Version Number

To check Torque Version Number, do issue the command

# qstat --version
Version: 4.2.7
Commit: xxxxxxxxxxxxxxxxxxxxxx

mca_oob_tcp_msg_recv: readv failed: Connection reset by peer

I was encountering the error from one of the users.

[compute-node1:00864] [[44805,0],0]-[[44805,1],7] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
[compute-node1:00864] [[44805,0],0]-[[44805,1],0] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
[compute-node1:00864] [[44805,0],0]-[[44805,1],1] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
[compute-node1:00864] [[44805,0],0]-[[44805,1],2] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
[compute-node1:00864] [[44805,0],0]-[[44805,1],5] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 869 on
node compute-node1 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).

There are several things to check

1. Disabled selinux
   

   vim /etc/selinux/config
   .....
   .....  
   SELINUX=disabled
   .....
   .....

   
   
2. Diagnose your IB Network
   For more information, see Diagnostic Tools to diagnose Infiniband Fabric Information


3. Check that your memory ulimit configuration is correct for /etc/security/limits.conf. See blog entry A relook at libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations
   
   
4. If your scheduler is torque, you have to configure the pbs_mom, see Default ulimit setting in torque overide ulimit setting

checkjob showq job is deferred. 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN')

I'm using Torque 4.2.6 and MAUI 3.3.1

When I submitted a job, the job landed with a Deferred command

job is deferred.  Reason:  RMFailure  (cannot start job - RM failure, rc: 15043, 
msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN')

You can do a tail -f /var/log/messages or /var/spool/torque/server_logs

LOG_ERROR::No route to host (113) in send_job_work, send_job failed to host 
comp-node-1, c0a832a7 port 15002

This gave a hint. I checked my iptables and I realised that the iptables was on and I shut accordingly and the issue was cleared.

For more information:

1. Troubleshooting deferred jobs, episode 80

An observation - MAUI MAXPROC initally not working on Torque 2.5

I reinstalled Torque 2.5 and MAUI. After going through the setup as found in Installing Torque 2.5 on CentOS 6. But instead of using the xcat scripts, I used the scripts found in contrib folder to do the installation

I also installed MAUi based on MAUI Installation on Torque and xCAT

After setting up the configuration, and testing with sample qsub

$ echo "sleep 30" |qsub -l nodes=2:ppn8 

. The MAXPROC limits in MAUI does not seems to work.

The solution  I found later on is that I made a mistake of starting service pbs_shed start before service pbs_server start which cause the MAXPROC in MAUI to stop functioning. When I started service pbs_server only, the MAXPROC in Maui works. strange....

Bad UID for job execution MSG=ruserok failed validating user1 from ServerNode while configuring Submission Node in Torque

Supposed you have the error when you are testing your submission node for torque. Firstly do take a look at the blog on Configuring Torque Submission Node

I'm using Torque 2.5.3 and MAUI 3.3.1

$ echo "sleep 30" | qsub -I
qsub: Bad UID for job execution MSG=ruserok failed validating ..... from server server1

The Error could be coming from 2 possible misconfigurations

Step 1: Do check that your Torque Server Configuration has the followings. You can check by doing

# qmgr -c "p s".
.....
.....
set server submit_hosts = submission_node
set server allow_node_submit = True 
.....
.....

If not you can add it in very quickly

# qmgr -c 'set server submit_hosts = submission_node'

# qmgr -c 'set server allow_node_submit = True'

Step 2: Start the PBS services in sequence.
From my observation, we do not need to start the pbs_sched just the pbs_server only

# service pbs_server start

Step 3: Finally make sure in the Primary Submission host, you configure to allow traffic from /etc/hosts.equiv

# vim /etc/hosts.equiv

# dns_name_of_secondary_hosts

Resolution for ERROR: torque.setup: line 45: pbs_server: command not found

# ./torque.setup root
initializing TORQUE (admin: root@headnode.com)
./torque.setup: line 45: pbs_server: command not found
ERROR: pbs_server failed to start, check syslog and server logs for more information

Resolution is very simple for this. Th reason for the error as indicated above is that the torque.setup was not able to find the path for pbs_server. Just do the following

# touch /etc/profile.d/torque.sh
vim /etc/profile.d/torque.sh
export PATH=$PATH:/opt/torque/x86_64/bin:/opt/torque/x86_64/sbin

# source /etc/profile.d/torque.sh 

Try ./torque.setup root again. It should work

Resolution for ERROR: cannot set TORQUE admins

Run the ./torque.setup root 

# ./torque.setup root
initializing TORQUE (admin: root@headnode.com)

You have selected to start pbs_server in create mode.
If the server database exists it will be overwritten.
do you wish to continue y/(n)?y
root     22809     1  0 11:57 ?        00:00:00 pbs_server -t create
Max open servers: 9
qmgr obj= svr=default: Bad ACL entry in host list MSG=First bad host: headnode.com
ERROR: cannot set TORQUE admins

Step 1: Configure the trqauthd daemon to start automatically at system boot for the PBS Server.
At $TORQUE_HOME

# cd $TORQUE_HOME
# cp contrib/init.d/trqauthd /etc/init.d/
# chkconfig --add trqauthd
# echo /usr/local/lib > /etc/ld.so.conf.d/torque.conf
# ldconfig
# service trqauthd start

Try the ./torque.setup root again. If there  is error like the above.

Step 2: Look at your host file. Make sure hostname of the server is the same as the first name with the corresponding IP Addresses

For example at /etc/hosts

# 192.168.1.1 headnode.com headnode
..... 
.....

At /etc/sysconfig/network

HOSTNAME=headnode.com
.....
.....

Step 3: Try the ./torque.setup root again.It should be successful

 ./torque.setup root
initializing TORQUE (admin: root@headnode.com)

You have selected to start pbs_server in create mode.
If the server database exists it will be overwritten.
do you wish to continue y/(n)?y
root     22843     1  0 12:00 ?        00:00:00 pbs_server -t create
Max open servers: 9
Max open servers: 9

Uninstalling Torque

Let's say you made a wrong installation and you wish to do a fresh install of torque, you can to the installation directory and issue the command

# make uninstall

Later you can proceed to remove the directory

# rm -Rv --force /var/spool/torque
# rm -Rv --force /opt/torque
# rm --force /etc/init.d/ trqauthd
# rm --force /etc/init.d/pbs_server
# rm --force /etc/init.d/pbs_sched
# rm --force /etc/init.d/pbs_mom 

PBS_MOM Error Mismatching protocols. Expected protocol 4 but read reply for 0

I encountered this error on my compute nodes using Torque 4.2.5.

pbs_mom.29384;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0

 This error was quite misleading. I was looking at my protocol which was IB and Ethernet.

 When I did a pbsnodes -l, all the compute nodes were down.

# pbsnodes -l
node-c00 down
node-c01 down
.....
..... 

After some troubleshooting, I realised that the error is due to use of inconsistent use of short hostname and long hostname. On my /etc/hosts, I used the long hostname for the compute node first (which Torque Server pick up.

192.168.1.2     node-c00.cluster.com    node-c00
......
...... 

But on each of the client nodes ie /etc/sysconfig/network, I used the short hostname. This create some confusion for the torque server

HOSTNAME=node-c00

To correct the matter, just rename the HOSTNAME to the long name

HOSTNAME=node-c00.cluster.com

Do a restart of the pbs_mom on the client node and you should get your nodes alive

# service pbs_mom restart

Deleting PBS and MAUI Jobs which cannot be purged

If the Compute Node pbs_mom is lost and cannot be recovered (due to hardware or network failure) and to purge a running job from the qstat output or show 1. Shutdown the pbs_server daemon on the PBS Server

# service pbs_server stop

2. Remove Job Spool Files that holds the hanged JobID (For example 4444)

# rm /var/spool/torque/server_priv/jobs/4444.headnode.SC
# rm /var/spool/torque/server_priv/jobs/4444.headnode.JB

3. Start the pbs_Server Daemon

# service pbs_server start

4. Restart the MAUI Daemon

# service maui restart

References:

1. Deleting PBS/Maui Jobs

RM Failure, rc: 15041, msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN'

I'm using Torque 4.2.5 and MAUI 3.3.1

When I was submitting a job, I always seem to get my job deferred and stuck. I used the MAUI "checkjob jobid" function and the detailed information will come up something like

RM Failure, rc: 15041, msg: 'Execution server rejected request 
MSG=cannot send job to mom, state=PRERUN'

The solution is quite simple. The compute nodes that the scheduler threw to have its iptables on and thus the job was blocked. To solve the issue, just turn off the iptables and it works.

PBS Error: Unable to change the status of compute nodes from down state when using Torque 4.2.5

 I did my installation according to Torque Administration 4.0.2

PBS_Server;LOG_ERROR::get_node_from_str, Node node-c00 is reporting on 
node node-c00.cluster.spms.ntu.edu.sg, 
which pbs_server doesn't know about

The solution is simple. Use the full hostname used by the client nodes and update the $TORQUE_HOME/server_priv/nodes

Restart the pbs_server, pbs_sched, trqauthd services

# service pbs_server restart
# service pbs_sched restart
# service trqauthd restart

TORQUE Security Advisory - 6 September 2013

Vulnerability: A non-privileged user who can run jobs or login to a node running pbs_server or pbs_mom can submit an arbitrary job to the cluster; that job can run as root. The user can submit a command directly to a pbs_mom daemon to queue and run a job. A malicious user could use this vulnerability to remotely execute code as root on the cluster.

Versions Affected: All versions of TORQUE

Mitigating Factors:
- The user must be logged in on a node that is already legitimately able to contact pbs_mom daemons or submit jobs.
- If a user submits a job via this defect and pbs_server is running, pbs_server will kill the job unless job syncing is disabled. It may take up to 45 seconds for pbs_server to kill the job.
- There are no known instances of this vulnerability being exploited.

Remedy: All TORQUE users should patch their systems using the following instructions:

For 2.5.x versions of TORQUE:
----------------------------
1. Download the patch file:
     $ wget http://www.adaptivecomputing.com/torquepatch/fix_mom_priv_2.5.patch
2. Run the patch command in the root directory of the TORQUE source tree:
     $ patch -p1 < fix_mom_priv_2.5.patch
3. Recompile TORQUE:
     $ make
4. Install TORQUE:
     $ sudo make install
5. Restart pbs_mom (pbs_server is not affected)


For 4.x versions of TORQUE:
----------------------------
1. Download the patch file:
     $ wget http://www.adaptivecomputing.com/torquepatch/fix_mom_priv.patch
2. Run the patch command in the root directory of the TORQUE source tree:
     $ patch -p1 < fix_mom_priv.patch
3. Recompile TORQUE:
     $ make
4. Install TORQUE:
     $ sudo make install
5. Restart pbs_mom (pbs_server is not affected)

What the Patch Does: The patch checks that the connection to the pbs_mom daemon is coming from a privileged port. This follows the security model that only privileged users should be able to submit arbitrary jobs.

Attribution: This vulnerability was discovered by John Fitzpatrick of MWR InfoSecurity. Matt Ezell of Oak Ridge National Laboratory assisted in creating the patch. We thank these individuals for helping to improve TORQUE.

Error on Torque .... received SISTER_EOF attempting to communicate with sister MOM's

Taken from https://nesccdocs.rdhpcs.noaa.gov/wiki/index.php/Frequently_Asked_Questions

When you experience the error below. According to the FAQ,

My job crashed with PBS: job killed: node 1 ($NODENAME) requested job terminate
, 'EOF' (code 1099) - received SISTER_EOF attempting to communicate with 
sister MOM's

This could be caused by a failed node.

But recently I'm seeing this error from a code run and the issue as that the code took took too much memory and the result was a node hang. The above result ensues.

Checking status of nodes for Torque

pbsnodes is a very useful command to check status of nodes. One of my favourite usage is check list the status of the nodes

# pbsnodes -l free
node-c00        free
node-c01        free

Other option includes "active", "all", "busy", "down", "free", "offline", "unknown", and "up"

# pbsnodes -a
 node-c00
     state = free
     np = 8
     properties = pulasan
     ntype = cluster
     jobs = 0/31516.starfruit-h00
     status = rectime=1371836955,varattr=,jobs=31516.node-c00,state=free,
netload=1810185981631,gres=,loadave=1.00,ncpus=8,physmem=16436612kb,
availmem=27307052kb,totmem=28626300kb,idletime=5540229,nusers=1,nsessions=1,
sessions=18708,uname=Linux node-c00.linuxtoolkit.blogspot.com 2.6.18-164.el5 
#1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64,opsys=linux

All attributes of a node or all nodes are listed. This is the default if no flag is given

Clear OFFLINE from listed nodes

# pbsnodes -c node-c00

Add the OFFLINE state

# pbsnodes -o node-c00

Using pam_pbssimpleauth.so to authorise login for users for Torque

For a cluster shared by many users, it is important to prevent errant users from directly ssh into the compute nodes, thus bypassing the scheduler. To implement the pam module, compile the Torque Server based on Installing Torque 2.5 on CentOS 6
Step 1: You should be able to find the pam_pbssimpleauth.so packages at

$TORQUE_HOME/tpackages/pam/lib64/security/pam_pbssimpleauth.a
$TORQUE_HOME/tpackages/pam/lib64/security/pam_pbssimpleauth.la
$TORQUE_HOME/tpackages/pam/lib64/security/pam_pbssimpleauth.so

Step 2: Copy the  pam_pbssimpleauth.so to the compute nodes. Step 2b: DO not put the pam_pbssimpleauth.so in on the Head Node

# scp $TORQUE_HOME/tpackages/pam/lib64/security/pam_pbssimpleauth.so node1:/lib64/security/

Step 3: Verify that the access.so is also present in the /lib64/security/ directory

# ls /lib64/security/access.so

Step 4: Add the access.so and pam_pbssimpleauth.so in the PAM configuration files

# vim /etc/pam.d/sshd

auth       required     pam_sepermit.so
auth       include      password-auth
account    required     pam_nologin.so

account    required     pam_pbssimpleauth.so
account    required     pam_access.so

account    include      password-auth
password   include      password-auth
.....
.....

When a user ssh’s to a node, this module will check the .JB files in $PBS_SERVER_HOME/mom_priv/jobs/ for a matching uid and that the job is running.
You can try the configuration