Sometimes as administrator, you may need to some immediate repair work and need to prevent users from logging in immediately, you can easily do that by doing the following:
# touch /etc/nologin
If the file exists, only the root can gain accessnow
It has been a long time since I used torque. I though I pen down some common commands I will used
pbsnodes -l (List node names and their state. If no state is specified, only nodes in the DOWN, OFFLINE, or UNKNOWN states are listed. Specifying a state string acts as an output filter. Valid state strings are "active", "all", "busy", "down", "free", "offline", "unknown", and "up". )
pbsnodes -o (Add the OFFLINE state. This is different from being marked DOWN. OFFLINE prevents new jobs from running on the specified nodes. This gives the administrator a tool to hold a node out of service without changing anything else. The OFFLINE state will never be set or cleared automatically by pbs_server; it is purely for the manager or operator.)
pbsnodes -c (Clear OFFLINE from listed nodes)
pbsnodes -a (All attributes of a node or all nodes are listed. This is the default if no flag is given)
We are using OpenFarics for our HPC. According to the FAQ, "With OpenFabrics (and therefore the openib BTL component), you need to set the available locked memory to a large number (or better yet, unlimited) -- the defaults with most Linux installations are usually too low for most HPC applications that utilize OpenFabrics"
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.