Diagnostic Tools to diagnose Infiniband Device

There are a few Diagnostic Tools to diagnose Infiniband Devices.

1. ibv_devinfo (Query RDMA devices)
2. ibstat (Query basic status of InfiniBand device(s))
3. ibstatus (Query basic status of InfiniBand device(s))

ibv_devinfo (Query RDMA devices) 
Print  information about RDMA devices available for use from userspace.

# ibv_devinfo

hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.10.2322
        node_guid:                      0002:c903:0045:1280
        sys_image_guid:                 0002:c903:0045:1283
        vendor_id:                      0x02c9
        vendor_part_id:                 4099
        hw_ver:                         0x0
        board_id:                       IBM0FD0140019
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               1
                        port_lmc:               0x00
                        link_layer:             IB

                port:   2
                        state:                  PORT_DOWN (1)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             IB

---

ibstat (Query basic status of InfiniBand device(s))

ibstat is a binary which displays basic information obtained  from  the local  IB  driver.  Output  includes LID, SMLID, port state, link width active, and port physical state.

It is similar to the ibstatus  utility  but  implemented  as  a  binary rather  than a script. It has options to list CAs and/or ports and displays more information than ibstatus.

# ibstat

CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.10.2322
        Hardware version: 0
        Node GUID: 0x0002c90300451280
        System image GUID: 0x0002c90300451283
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251486a
                Port GUID: 0x0002c90300451281
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x0002c90300451282
                Link layer: InfiniBand

---

ibstatus - (Query basic status of InfiniBand device(s))

ibstatus is a script which displays basic information obtained from the local IB driver. Output includes LID, SMLID,  port  state,  link  width active, and port physical state.

# ibstatus

Infiniband device 'mlx4_0' port 1 status:
        default gid:     fe80:0000:0000:0000:0002:c903:0045:1281
        base lid:        0x1
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      InfiniBand

Infiniband device 'mlx4_0' port 2 status:
        default gid:     fe80:0000:0000:0000:0002:c903:0045:1282
        base lid:        0x0
        sm lid:          0x0
        state:           1: DOWN
        phys state:      2: Polling
        rate:            40 Gb/sec (4X QDR)
        link_layer:      InfiniBand

Specifying Maximum and Minimum Walltime in Torque

Specifying Maximum and Minimum Walltime in the queue or globally within the Torque configuration is important or you have users who uses without restraint.

At Torque Server, you can set maximum walltime 

# qmgr -c "set server resources_max.walltime = 720:00:00"

At Queue, you can set maximum walltime,

# qmgr -c "set queue dqueue resources_max.walltime = 720:00:00

For minimum walltime, it seems that you can set at queue level at least for Torque 2.5.x

# qmgr -c "set queue dqueue resources_max.walltime = 00:00:10"

Great YouTube Video on how to find the functions that cause a crash in a Linux application

A tutorial showing how to find the functions that cause a crash in a Linux application if all you know is the address of the crash using the common ldd and nm tools by Gilad Ben-Yossef. See
Gilad Ben-Yossef on using ldd and nm (You Tube)

Modifying the walltime of a running job in Torque

The attribute qalter in Torque is a useful tool in helping to modify the attributes of the job or jobs specified by job_identifier on the command line.

One of the more commonly use of qalter is to modify the walltime of a running job in Torque. For example

# qalter jobid -l walltime=5:00:00

For more information on qalter, see qalter from Adaptive Computing

Quick method for estimating walltime

For Torque / OpenPBS or any other scheduler, walltime is a important parameter to allow the scheduler to determine how long the jobs will take. You can do a quick rough estimate by using the command time

# time -p mpirun -np 16 --host node1,node2 hello_world_mpi

real 4.31
user 0.04
sys 0.01

Use the value of 4:31 as the estimate walltime. Since this is an estimate, you may want to place a higher value in the walltime

$ qsub -l walltime=5:00 -l nodes=1:ppn=8 openmpi.sh -v file=hello_world