Encountering bash: ELF: command not found

I encounter one of the user having this issue "ELF: command not found" There is a good writeup and explanation on this phenomenon "bash: ELF: command not found" by Solution Blog

Diagnosing Mellanox Fabrics and RDMA Aware Networks Programming User Manual

If you wish to diagnose the Mellanox/Voltaire Fabric, do the following

1. Reset all the fabric PM counters

# ibdiagnet -pc

2. If any of the provided PM is greater then its provided value than print it

# ibdiagnet -P all=1

You will find logs of the findings from ibdiagnet such as

# ls -l /var/tmp/ibdiagnet2/
ibdiagnet2.aguid
ibdiagnet2.db_csv
ibdiagnet2.debug
ibdiagnet2.log
ibdiagnet2.lst
ibdiagnet2.nodes_info
ibdiagnet2.pkey
ibdiagnet2.pm
ibdiagnet2.sm

3. To check out the errors in your Fabric Environment, check the generated ibdiagnet2.log file for more information.

For references on

1. RDMA Aware Networks Programming User Manual (pdf)
2. Mellanox OFED for Linux User Manual (pdf)

Commercial Download for Intel Compilers

If you are downloading the Commercial version of the Intel Compilers, do log-on to the site below and access  https://registrationcenter.intel.com/RegCenter/MyProducts.aspx

Deleting PBS and MAUI Jobs which cannot be purged

If the Compute Node pbs_mom is lost and cannot be recovered (due to hardware or network failure) and to purge a running job from the qstat output or show 1. Shutdown the pbs_server daemon on the PBS Server

# service pbs_server stop

2. Remove Job Spool Files that holds the hanged JobID (For example 4444)

# rm /var/spool/torque/server_priv/jobs/4444.headnode.SC
# rm /var/spool/torque/server_priv/jobs/4444.headnode.JB

3. Start the pbs_Server Daemon

# service pbs_server start

4. Restart the MAUI Daemon

# service maui restart

References:

1. Deleting PBS/Maui Jobs

Compiling Chelseio IWARP Drivers (2.8.0.0) on Chelsio T4 Cards on CentOS 5

The below is a subset of the Chelsio 2.8.0.0 ReadMe

The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms. Following is the list of Drivers/Software and supported Linux distributions. Here is a subset of the README.

The OS I used was CentOS 5.8

|########################|#####################################################|
|   Linux Distribution   |                Driver/Software                      |
|########################|#####################################################|
|RHEL5.8,2.6.18-308.el5  |NIC/TOE,vNIC,iWARP,WD-UDP*,WD-TOE*,iSCSI Target*,    |
|                        |Bonding,IPv6,Bypass*,Sniffer & Tracer                |
|                        |UM(Agent,Client),UDP-SO,Filtering,TM                 |
|------------------------|-----------------------------------------------------|
|RHEL5.9,2.6.18-348.el5  |NIC/TOE*,vNIC*,iWARP*,WD-UDP*,WD-TOE*,iSCSI Target*, |
|                        |Bonding*,IPv6*,Bypass*,Sniffer & Tracer*,UDP-SO*,    |
|                        |Filtering*,TM*                                       |
|------------------------|-----------------------------------------------------|
|RHEL6.3,                |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE*,iSCSI Target*,     |
|2.6.32-279.el6          |iSCSI Initiator*,FCoE Initiator*,                    |
|                        |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO,        |
|                        |UM(Agent,Client,WebGUI),Filtering,TM                 |
|------------------------|-----------------------------------------------------|
|RHEL6.4,                |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target,       |
|2.6.32-358.el6          |iSCSI Initiator,FCoE Initiator,Bonding,IPv6,Bypass,  |
|                        |Sniffer & Tracer,UDP-SO,UM(Agent,Client,WebGUI),     |
|                        |Filtering,TM,uBoot(DUD)                              |
|------------------------|-----------------------------------------------------|

Strangely, I was not able to compile with 3.5.1. It seems that the compat-rdma on 3.5.1 is having issues with CentOS 5.8. See Failed to build compat-rdma RPM when compiling OFED 3.5.1 on CentOS 5.8

I tried with OFED 1.5.4.1, but errors occurred as well. But compiling OFED 1.5.3.2 works well and Chelsio T420-BCH was able to compile nicely with OFED 1.5.3.2. To download OFED 1.5.3.2, do visit the OFED Downloads Site

---

Part 1

To compile from source

 i.  Download the tarball ChelsioUwire-x.x.x.x.tar.gz

ii. Untar the tarball
  
[root@host]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz
 
iii. Change your current working directory to Chelsio Unified Wire package
     directory. Build the source:

[root@host]# make
 
iv. Install the drivers, tools and libraries:
  
[root@host]# make install
 
v. The default configuration tuning option is Unified Wire.
   The configuration tuning can be selected using the following commands:

[root@host]# make CONF=(T5/T4 Configuration)
[root@host]# make CONF=(T5/T4 Configuration install)

(where T5/T4 Configuration is
UNIFIED_WIRE, HIGH_CAPACITY_TOE, HIGH_CAPACITY_RDMA, LOW_LATENCY, UDP_OFFLOAD, T5_WIRE_DIRECT_LATENCY)

---

Part 2  - Installing Individual Drivers

i. To build and install iWARP driver against outbox OFED:
[root@host]# make iwarp
[root@host]# make iwarp_install

---

Part 3a - Loading IWARP Drivers

Manually  Load  Drivers
To load the iWARP driver we need to load the NIC driver & core RDMA drivers first:
[root@host]# modprobe cxgb4
[root@host]# modprobe iw_cxgb4
[root@host]# modprobe rdma_ucm

Part 3b - Automatic IWARP Drivers
To load the Chelsio iWARP drivers automatically, add this additional lines to /etc/modprobe.conf

options iw_cxgb4 peer2peer=1
install cxgb4 /sbin/modprobe -i cxgb4; /sbin/modprobe -f iw_cxgb4; /sbin/modprobe rdma_ucm
alias eth1 cxgb4 # assuming eth1 is used by the Chelsio interface

---

Finally Reboot the system to load the new modules

References:

1. Chelsio 2.8.0.0 ReadMe