Switching between Ethernet and Infiniband using Virtual Protocol Interconnect (VPI)

This short writeup is a summary of the article Switching between Ethernet and Infiniband using Virtual Protocol Interconnect (VPI). Of course you will need to use the QSA Adapter (QSFP+ to SFP+ adapter) which is the world's first solution for the QSFP to SFP+ conversion challenge for 40GB/Infiniband to 10G/1G. For more information, see Quad to Serial Small Form Factor Pluggable (QSA) Adapter to allow for the hardware


For the full article, see Switching between Ethernet and Infiniband using Virtual Protocol Interconnect (VPI)

Overview
mlx4 is the low level driver implementation for the ConnectX adapters designed by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter, as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports Infiniband and Ethernet NIC configurations. To accommodate the supported configurations, the driver is split into three modules:

1. mlx4_core
   Handles low-level functions like device initialization and firmware commands processing. Also controls resource allocation so that the InfiniBand and Ethernet functions can share the device without interfering with each other.
2. mlx4_ib
   Handles InfiniBand-specific functions and plugs into the InfiniBand midlayer
3. mlx4_en
   A new 10G driver named mlx4_en was added to drivers/net/mlx4. It handles Ethernet specific functions and plugs into the netdev mid-layer.

Using Virtual Protocol Interconnect (VPI) to switch between Ethernet and Infiniband
Loading Drivers

1. The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and Infiniband drivers. It supplies the user with the ability to run Infiniband and Ethernet protocols on the same HCA.
2. Check the MLX4 Driver is loaded, ensure that the

   # vim /etc/infiniband/openib.conf

   # Load MLX4_EN module
   MLX4_EN_LOAD=yes

3. If the MLX4_EN_LOAD=no, the Ethernet Driver can be loaded by running

   # /sbin/modprobe mlx4_en

Port Management / Driver Switching

1. Show Port Configuration

   # /sbin/connectx_port_config -s

   --------------------------------
   Port configuration for PCI device: 0000:16:00.0 is:
   eth
   eth
   --------------------------------

2. Looking at saved configuration

   # vim /etc/infiniband/connectx.conf

3. Switching between Ethernet and Infiniband

   # /sbin/connectx_port_config

4. Configuration supported by VPI

   - The following configurations are supported by VPI:
    Port1 = eth   Port2 = eth
    Port1 = ib    Port2 = ib
    Port1 = auto  Port2 = auto
    Port1 = ib    Port2 = eth
    Port1 = ib    Port2 = auto
    Port1 = auto  Port2 = eth
   
     Note: the following options are not supported:
    Port1 = eth   Port2 = ib
    Port1 = eth   Port2 = auto
    Port1 = auto  Port2 = ib

For more information, see

1. ConnectX -3 VPI Single and Dual QSFP+ Port Adapter Card User Manual (pdf)
2. Open Fabrics Enterprise Distribution (OFED) ConnectX driver (mlx4) in OFED 1.4 Release Notes