Shipments of Disk Drives Declines in 5% in 2013

I refer to this article Shipments of Disk Drives Decline 5% in 2013 from StorageNewsletter.com . According to the article,

A near-doubling of the market for SSDs in 2013 was unable to help the computer-related storage industry last year, which slid 5% in shipments because of continuing contractions in the hard-disk drive and optical disk drive segments, according to a new report from IHS Technology.

(Source: IHS Technology, January 2014)'

HDD down but not out
....
In the HDD segment, the rise of smartphones and tablets has dented the once-powerful appeal of computers, impacting HDD volume. The losses are especially apparent in the so-called client PC market-the consumer side of the PC business.

.....

Settimg up a new network device in CentOS

If you added a new network device, but upon reboot, CentOS did not represent the card in the /etc/sysconfig/network-scripts/ifcfg-eth*, you may have to manually make aware the device to the OS.

Step 1: Make sure that the OS has detected the new card. From the example below, it detected my 2nd card. If you are unable to detect, you may want to reinstall or check the driver.

# lspci |grep -i ethernet
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

Step 2: Launch the system-config-network and fill up the necessary information.

# system-config-network

You will see that the system-config-network create an entry at /etc/sysconfig/network-scripts/ifcfg-eth1.

Restart the network

# service network restart

NetworkManager causing Device eth0 does not seem to be present, delaying initialization on CentOS

"Device eth0 does not seem to be present, delaying initialization on CentOS" error can be due to

1. Cannot get device settings No such device. 
2. "Device eth0 does not seem to be present" on cloned CentOS VM

Step 1. But it could be also due to the NetworkManager overridding your manual settings in the network configuration. You may want to disabled NetworkManager

# chkconfig --levels 2345 NetworkManager off

Step 2: At the /etc/sysconfig/network-scripts/ifcfg-eth0, do remember to put this line in

.....
..... 
NM_CONTROLLED="no"
.....
.....

Step 3: Reboot the machine and do a ifconfig

# ifconfig

You should be able to see the eth0

Disabling the virbr0 NAT Interface

Taken from Linux KVM: Disable virbr0 NAT Interface

 If you accidentally install the KVM components and you do not wish to use the interface, you will notice when you do ifconfig, you will see the "virbro"

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          inet addr:192.168.1.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fee5:70b0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6885 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:17192337 (16.3 MiB)  TX bytes::1107966 (1.0 MiB)

If you use the follow command

# virsh net-list

Name                 State      Autostart
-----------------------------------------
default              active     yes

# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart

Device eth0 does not seem to be present, delaying initialization on CentOS

When I use the command

# ifup eth3

I received the error

"device eth3 does not seem to be present, delaying initialization"

First thing first, Do check whether the network card is there, do use the command

# lspci |grep Mellanox

If the card is detected, do use this command to show the link. This will give you the clues on

# ip -o link

.....
.....
2: eth3:  mtu 1500 qdisc pfifo_fast qlen 1000\ 
link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
3: eth4:  mtu 1500 qdisc pfifo_fast qlen 1000\ 
link/ether 00:00:00:00:00:02 brd ff:ff:ff:ff:ff:ff
.....
.....

References:

1. device eth1 does not seem to be present, delaying initialization from AR

Quad to Serial Small Form Factor Pluggable (QSA) Adapter

Quad to Serial Small Form Factor Pluggable (QSA) Adapter designed by Mellanox Technologies is the world’s first solution for the QSFP to SFP+ conversion challenge.

The QSA enables smooth, cost-effective, connections between Virtual Protocol Interconnect® (VPI) or 40 Gigabit Ethernet adapters using contemporary QSFP ports and 1 or 10 Gigabit Ethernet networks using existing SFP or SFP+ based cabling. Similarly Ethernet switches with 40Gb/s QSFP ports can connect to servers with 10Gb/s Ethernet NIC ports using QSA.

For more information, see Quad to Serial Small Form Factor Pluggable (QSA) Adapter from Mellanox

Importance of configuring ifcfg-ethX properly to resolve DNS

I have an interesting problem today. I have configured my compute nodes to access the public network via the head node as the gateway. A good tutorial can be found from
Using iptables to allow compute nodes to access public network from Linux Cluster Blog.


Occasionally, 1 or 2 nodes will not be able to resolve even though /etc/resolv.conf is configured correctly. It seems that only after putting in the localised DNS at /etc/sysconfig/network-script/ifcfg-ethX at the compute node, it was able to resolve.

DEVICE=eth0
BOOTPROTO=static
HWADDR=E4:1F:13:CC:51:54
ONBOOT=yes
HOTPLUG=no
IPADDR=192.168.5.17
NETMASK=255.255.255.0
GATEWAY=192.168.5.1
DNS1=155.1.1.2
DNS2=155.1.1.3
PEERDNS=yes

I suspect it could be due to the multiple NIC with multiple network segment issues and localised DNS resolution at network level.

Take a look at  Redhat Document 8.2 Interface Configuration Files

Using netstat to diagnose network

Netstat is one good "swiss army knife" to look deeper into the workings linux networking.

I thought I just quickly put some netstat commands which I often used and find it helpgul in resolving networking issue.

1. Checking of networking interfaces

netstat -i

Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 95453656      0      0      0   177764      0      0      0 BMRU
lo        16436   0       70      0      0      0       70      0      0      0 LRU

 

2. Show Kernel Routing Table Information

# netstat -r 

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  1.1.57.28     *               255.255.255.128 U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default             1.1.1.125   0.0.0.0         UG        0 0          0 eth0

3. Show active listening ports ( -l) with the specific process (-p) associated with listening port

# netstat -ap

....
unix  2      [ ACC ]     STREAM     LISTENING     21138  2474/gnome-session  @/tmp/.ICE-unix/2474
unix  2      [ ACC ]     STREAM     LISTENING     23166  2674/pulseaudio     /tmp/.esd-0/socket
....

3a. To show a specific process ie ssh

# netstat -ap |grep ssh

tcp        0      0 *:ssh                       *:*                         LISTEN      1771/sshd
tcp        0     52 1.1.57.28:ssh            172.21.4.129:50591          ESTABLISHED 7837/sshd
tcp        0      0 *:ssh                       *:*                         LISTEN      1771/sshd
unix       2      [ ACC ]     STREAM     LISTENING     21646  2464/gnome-keyring- /tmp/keyring-i1zxcd/socket.ssh
unix       2      [ ]         DGRAM                    8766783 7837/sshd

4. View operational statistics for network protocol

# netstat -s

Ip:
    12311840 total packets received
    1801583 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    10510256 incoming packets delivered
    174002 requests sent out
Icmp:
    300 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 18
        echo requests: 282
    555 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 125
        echo request: 148
        echo replies: 282
...

A good resource can be found at

1.  UNIX / Linux: 10 Netstat Command Examples

Listing of Hardware Address of Network Cards

To quickly list Hardware Address of all your Network Cards in a nice listings, you can just use a simple commands

# ifconfig -a |grep HWaddr

eth0      Link encap:Ethernet  HWaddr 00:00:00:00:00:1a
eth1      Link encap:Ethernet  HWaddr 00:00:00:00:00:1b
eth2      Link encap:Ethernet  HWaddr 00:00:00:00:00:1c

Using iptables to allow compute nodes to access public network

Have written a tutorial on Using iptables to allow compute nodes to access public network on LinuxCluster. Do read it.

Using Ethtool to manage Ethernet Card

Ethtool is really a swiss army knife for diagnosing and managing and manipulating Network Card. I used it rather often. Here are some which I use more often.

Usage 1: ethtool eth0

# ethtool eth0

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes

Interesting information include "Link Detected Status", Duplex, Speed, Supported Ports etc. I particular find the Link Detection Usage one of my most used tool. :)


Usage 2: Checking of Ethernet Drivers Version

# ethtool -i eth0

driver: bnx2
version: 2.0.23b
firmware-version: bc 1.9.6
bus-info: 0000:03:00.0

Usage 3: Identify the Physical Network Card (Blink LED Port of NIC Card)

# ethtool -p eth0

Usage 4: Display Auto-negotiation, RX and TX of Network Card

# ethtool -a eth0

Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

Usage 5: Show offl-load property of Network Card.

# ethtool -k eth0

Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

For more information, do look at the excellent article on

1. Ethtool man page
2. 9 Linux ethtool Examples to Manipulate Ethernet Card (NIC Card)

High Performance Data Transfers on TCP/IP

This writeup is a summary of the excellent article from Pittburgh Supercomputing Centre “Enabling High Performance Data Transfers“

For more information, see  High Performance Data Transfers on TCP/IP

Tuning TCP for Linux 2.4 and Linux 2.6

I have read an interesting article on TCP Tuning for the Linux 2.4 and 2.6 from (Advanced Networking, Pitsburgh Supercomputing Centre). From the article

"NB: Recent versions of Linux (version 2.6.17 and later) have full autotuning with 4 MB maximum buffer sizes. Except in some rare cases, manual tuning is unlikely to substantially improve the performance of these kernels over most network paths, and is not generally recommended
 
All system parameters can be read or set by accessing special files in the /proc file system. E.g.:
 

cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf

If the parameter tcp_moderate_rcvbuf is present and has value 1 then autotuning is in effect. With autotuning, the receiver buffer size (and TCP window size) is dynamically updated (autotuned) for each connection. (Sender side autotuning has been present and unconditionally enabled for many years now)."

Encountering eth0 NIC SerDES Link is Down

I was noticing this error on my HS22 Blade log files occasionally and on one occasion the NFS which was relying on the ethernet connection got disconnected and hang when the load is exceedingly high. The problem is that it is very hard to reproduce the problem as it is quite random

My Server is using the Broadcom chipset bnx2 and my version of my CentOS is 5.4 or kernel  version is 2.6.18-164.el5

After a bit of searching, this particular Red Hat Bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=520888) reflects the problem and workaround very well. I encourage you to take a closer look. If you are not planning to upgrade your RHEL or CentOS to 5.6 ( http://rhn.redhat.com/errata/RHSA-2011-0017.html ) and above yet, you may want to consider the workaround as mentioned in the bugzilla



 From Comments 14

"Configuring IRQ SMP affinity has no effect on some devices that use message signalled interrupts (MSI) with no MSI per-vector masking capability. Examples of such devices include Broadcom NetXtreme Ethernet devices that use the bnx2 driver. 

If you need to configure IRQ affinity for such a device, disable MSI by creating a file in /etc/modprobe.d/ containing the following line: 

options bnx2 disable_msi=1 

Alternatively, you can disable MSI completely using the kernel boot parameter pci=nomsi. (BZ#432451)

" http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Technical_Notes/Known_Issues-kernel.html

To Check whether you are still having issues, you can use the command

# dmesg |grep bnx2

I guess the best way is to update your broadcom drivers. For latest update on this "NIC SerDES Link is Down", see my writeup on Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down

Myricom DBL 2.0 Achieves Lowest UDP and TCP Latency for High Frequency Trading

For the full article, see Myricom DBL 2.0 Achieves Lowest UDP and TCP Latency for High Frequency Trading

Myricom DBL 2.0 software has benchmarked application-to-application UDP latency of under 3.5 microseconds and transparent sockets TCP latency of 4.0 microseconds. For HFT applications, DBL enables unmatched networking performance for UDP multicast and TCP order execution, all over industry-standard 10-Gigabit Ethernet
 .......
DBL reduces latency by microseconds for existing applications running on standard TCP/UDP Ethernet networks. With the DBL solution, end-users can achieve extreme performance without rewriting their applications or resorting to specialty networks such as Infiniband. DBL provides transparent acceleration in both Linux and Windows environments.
......
In addition to extremely low UDP and TCP communication latency, DBL 2.0 delivers repeatable low latency, rather than unpredictable and variable latency performance found with competing solutions. Repeatable low latency performance is critical, as packet delay or loss in mission-critical trading and order environments can be devastating to the traders' bottom line.

Strange /etc/sysconfig/network-scripts/ifcfg-eth*.bak problems on CentOS

I think there is a bug with Kudzu that causes sometimes to change network setting from /etc/sysconfig/network-scripts/ifcfg-eth* to /etc/sysconfig/network-scripts/ifcfg-eth*.bak causing loss of network connectivity after there is a reboot. The original setting was replaced with one with DHCP. In other words, the original ifcfg-eth0 becomes ifcfg-eth0.bak and a new one ifcfg-eth0 is a config scripts with DHCP settings. 

It seems that the cause of this issue seems to be coming from the Kudzu Daemon. Once I stop the service, my network configuration remain as it is during bootup.

# chkconfig --levels 2345 kudzu off

# service kudzu stop

For more information, do look at  CentOS Discussion Thread http://www.centos.org/modules/newbb/viewtopic.php?topic_id=8376

Building HPC Clusters with 10Gigabit Ethernet (White Paper)

This document presents data comparing the performance of two well-known HPC applications and the impact of performance and bench-marked performance between 1GE, 10GE and InfiniBand. This paper is from ARISTA.

1. Building HPC Clusters with 10Gigabit Ethernet (White Paper)

Configuring an Ethernet Bridge

According to the Bridge Web site:

Ethernet bridging is a way to connect networks together to form a larger network. The standard for bridging is ANSI/IEEE 802.1d. A bridge is a way to connect two separate network segments together in a protocol-independent way. Packets are forwarded based on Ethernet address, rather than IP address (like a router). Since forwarding is done at Layer 2, all protocols can go transparently through a bridge.

For a more simplier explaination,

A Bridge is a device that links 1 or more network segment that uses the same network technologies. One simple analogies is the typical hub where you can plug in as many boxes you wish and they become part of a single hub device. It can also be used to modify or sniff the traffic

To bridge 2 physical network interfaces, you will need root access

# brctl addbr br0
# brctl addif br0 eth0
# brctl addif br0 eth1
# echo 1 > /proc/sys/net/ipv4/ip_forward
# ifconfig eth0 up
# ifconfig eth1 up
# ifconfig br0 up

For more information on Ethernet Bridge, do read up the following article which this blog entry relies totally on

1. Kernel Korner - Linux as an Ethernet Bridge (Linux Journal)
2. Linux Networking Bridge by Tom Salmon

Getting DUP! error when pinging a IP Address which uses trunked port

I have aggregated 2 ports for the external LAN and incoming LAN for my Big-IP F5 Load Balancer. Interestingly when I was pinging the IP which uses the aggregated ports of the switch, I got a

..............
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=12.4.8 ms (DUP!)
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.216 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=12.4.8 ms (DUP!)
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.196 ms
.............

After a while, the DUP! disappeared. As usual I google around and notice the article which explain the phenenomen. In a nutshell, the (DUP!) is caused by

......it is a side effect of how many switches update their MAC forwarding tables. Initially, the switch does not associate the MAC address in the packet with a particular switch port, and so it may send the
traffic to all ports until its MAC forwarding table is updated.


Since the interfaces attached to the bond may occupy multiple ports on a single switch, when the switch (temporarily) floods the traffic to all ports, the bond device receives multiple copies of the same packet (one per slave device)..........

For more information, see Duplicate ping error with network bonding driver in Linux

Managing F5 LTM TCP Profile Idle Timeout Settings

For those running Remote Desktop Protocol and facing idle timeout problems, may want to take a look at the Article "Investigating the LTM TCP Profile: Max Syn Retransmissions & Idle Timeout". The default 300 seconds is good enough for most application, but for applications with long-lived connections like remote desktop protocol, the settings should be tweaked to a larger figures or configure to "indefinite"