How To Upgrade HCA Firmware Using mlxup

How To Upgrade HCA Firmware Using mlxup from Mellanox Technologies on Vimeo.

How To Tune Your Linux Server Using mlnx_tune Tool

This is a wonderful video from Mellanox

  

How To Tune Your Linux Server Using mlnx_tune Tool from Mellanox Technologies on Vimeo.

Mellanox OFED Version 2.1-1.0.6 for Linux Driver is now available

Product Updates

Mellanox OFED Version 2.1-1.0.6 for Linux Driver is now available

• IB Core – Added allocation success verification process to ib_alloc_device
• DAPL – DAPL is recompiled with no FCA support
• Openibd – Added the ability to bring up child interfaces even if the parent’s ifcfg file is not configured
• Ilibmlx4 – Unmapped the hca_clock_page parameter from mlx4_uninit_context.
• Scsi_transport_srp – scsi_transport_srp can't be cleared up when port reconnecting fails
• Mlnxofedinstall – Added support for the “—umad-dev-na” and “—without-” parameters

Relevant Information:

• Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)

Mellanox Ethernet EN Driver version 1.9.10-0 for VMware(R) vSphere5.1Ux/5.5Ux is now available.

• Single/Dual port
• MSI-X / Int-X
• NetQueue support
• Multiple Tx/Rx rings
• HW Tx/Rx checksum offload
• Large Send Offload ( TCP Segmentation Offload)
• VLAN Tx/Rx acceleration (HW VLAN stripping/Insertion)
• Ethtool support
• NAPI support
• WOL ( only on supported HW)
• NC-SI
• Auto moderation
• Net dump (Only for ESXu5.5Ux)
• RSS Queues
• Fixed Pass-Through
• 10Gb/s
• 40Gb/s (Only for ESXi5.5Ux)

Relevant Pages

• MLX4_EN Driver for VMware ESX/ESXi 5.x Product Page
• VMware: Firmware - Driver Compatibility Matrix

Mellanox MLNX_OFED 2.1 is released

Product Updates

1. Mellanox OFED Version 2.1-1.0.0 for Linux Driver is now available.

The release has the following new features:

• Signature Verbs (T10-PI) (beta level) RoCE Time Stamping
• PeerDirect™
• Inline-Receive
• Ethernet performance counters
• Memory window
• VMA 6.5.9 bundled with MLNX_OFED
• DCT support (at beta level)
• eIPoIB multicast support
• Connect-IB – Added the ability to resize CAs
• GA support for ConnectX-3 Pro
• IPoIB – Reusing DMA mapped SKB buffers
• IPoIB – Performance improvements when IOMMU is enabled
• mlnx_en – reporting auto-negotiation support
• mlnx_en – Trans00mit Packet Steering (XPS) support
• mlnx_en – reporting 56GbE link speed support
• mlnx_en – Receive Flow Steering (RFS) support in UDP
• mlnx_en – Low latency Socket (LLS) support
• mlnx_en – check for dma_mapping errors
• eIPoIB – non-virtualized environment support

Related Links

1. MLNX_OFED 2.1-1.0.0 Product Page
2. MLNX_OFED 2.1-1.0.0 Driver Compatibility Matrix
3. MLNX_OFED 2.1-1.0.0 Release Notes
4. MLNX_OFED 2.1-1.0.0 User Manual

For more information,

1. See Mellanox Product Updates 21 Jan 2014

Mellanox Online Academy

The Mellanox Online Academy platform includes various courses, videos, best practices, and other free content to support learners of all levels and backgrounds in educating themselves on Mellanox products and related technologies (including general technologies). The flexibility and variety of the content enables students to acquire the knowledge to improve their personal and organizational performance.

Key Features:

• Courses oriented toward both general and Mellanox-based technologies
• Courses built and delivered by leading industry and product experts and certified instructors
• Flexible learning options (content, time)
• Flexible payment options (credit cards, PayPal, standard PO and purchase via phone)
• Varied subscription plans, duration and additional services
• Access to Mellanox's expert to support the learning process

Error during installing Mellanox MLNX_OFED 1.5.3 for CentOS 6.3.

I downloaded the Mellanox MLNX_OFED Drivers for RH 6.3 and CentOS 6.3

[root@node-c01 iso]# ./mlnxofedinstall
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be r                                                                                                                       ved.
Do you want to continue?[y/N]:y


rpm -e --allmatches --nodeps kernel-mft kernel-mft knem kernel-mft

Starting MLNX_OFED_LINUX-1.5.3-4.0.42 installation ...

Installing kernel-mft RPM
Preparing...                ##################################################
kernel-mft                  ##################################################
Installing knem RPM
Preparing...                ##################################################
knem                        ##################################################
libibumad was not created

This look like the libibumad packages are missing so I did a

# yum install  libibumad* 

But when I launched the ./mlnxofedinstall, the error appear again. Somehow the scripts remove the libibumad packages that I installed and the error occured again.

To resolve the issue, I used MLNX_OFED 2.0.3 and everything was ok.

Mellanox OFED 2.0-3.0.0 Linux Driver is now available.

Mellanox OFED 2.0-3.0.0 Linux Driver is now available.
MLNX_OFED 2.0-3.0.0 main features:

1. SR-IOV for InfiniBand Ethernet         
2. RoCE Over SR-IOV
3. Port base QoS for both Ethernet and InfiniBand
4. eIpoIB
5. Connect-IB™
6. Low Latency Socket
7. 56GbE
8. XRC
9. Dynamically Connected Transport ( DCT)  - alpha level
10. Flow Steering

MLNX_OFED 2.0-3.0.0 Download and Manual Page
MLNX_OFED 2.0-3.0.0 Release notes

 Differences between MLNX_OFED 1.5.3 and MLNX_OFED 2.0:

1. MLNX_OFED 2.0 is based on Kernel 3.7 which is a huge leap forward from the 2.6.30 Kernel that the 1.5.x is based on.
2. MLNX_OFED 2.0 supports the operating systems that are based on newer Kernels such as RHEL 6.x and SLES 11 SPx. MLNX_OFED 1.5.x will continue to support OS’s based on older kernels such as RHEL 5.x and SLES 10 SPx.
3. MLNX_OFED 2.0 includes new functionality, 1.5.x does not.

Full Support Matrix for Both MLNX_OFED Branches 

For more information, do read

1. Online version of Product and Support Updates Notification

Registering sufficent memory for OpenIB when using Mellanox HCA

If you encountered errors like "error registering openib memory" similar to what is written below. You may want to take a look at the OpenMPI FAQ - I'm getting errors about "error registering openib memory"; what do I do? .

WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              node02
  Registerable memory:     32768 MiB
  Total memory:            65476 MiB

Your MPI job will continue, but may be behave poorly and/or hang.

The explanation solution can be found at How to increase MTT Size in Mellanox HCA.

In summary, the error occurred when applications which consumed a large amount of memory, application might fail when not enough memory can be registered with RDMA. There is a need to increase MTT size. But increasing MTT size hasve the downside of increasing the number of "cache misses" and increases latency.

For a more details writeup. See  Registering sufficent memory for OpenIB when using Mellanox HCA (linuxcluster.wordpress.com)

Adding and Removing the 2nd Mellanox Ethernet Port as an uplink to an Existing Vswitch using the CLI

At VSphere 5.1 Client. I was able to see the Dual-Port Network Adapter (vmnic22.p1, vmnic22.p2) after I install the Vmware Installing Mellanox ConnectX® EN 10GbE Drivers for VMware® ESX 5.x Server.

But somehow I am not able to use the 2nd port of the Mellanox ConnectX 10G on the VSphere Client > Configuration > Networking. It will not be visible. However at the VSphere Client > Configuration > Networking > Add Networking, I not able to see the 2nd Port being available.

I found the document from Mellanox (MellanoxMLX4_ENDriverforVMwareESXi-5.xREADME) which is useful to resolve the issue.

At Page 10, Adding the Device as an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permission

Step 2: Add an uplink to a vswitch, run:

# esxcli network vswitch standard uplink add –u <uplink_name> -v <vswitch_name>

* Uplink_name refer to the name used by ESX for the network Adapter. For example, vmnic22.p2 is the uplink name

Step 3: Check that uplink was added successfully. Run:

# esxcli network vswitch standard list -v <vswitch_name>

Removing the Device an an uplink to an Existing Vswitch using the CLI

Step 1: Log into the ESXi server with root permissions Step 2: Remove an uplink from a vswitch, run:

# esxcli network vswitch standard uplink remove -u <uplink_name> -v <vswitch_name>