Tools to speed up kernel crash hang analysis with the kernel log

This is a summaries article taken from RHEL6: Speeding up kernel crash / hang analysis with the kernel log. When there is a kernel crash or hang, there is often a very large file is produced containing a memory dump of the entire system called a vmcore. Analysis of the kernel crash or hang often requires this large file be uploaded to Red Hat for analysis (if you have subscription)  

 For RHEL 6.4 and above Starting with RHEL 6.4, Starting with Red Hat Enterprise Linux 6.4 and kexec-tools-2.0.0-258.el6, the kdump process will dump the kernel log to a file called vmcore-dmesg.txt before creating the vmcore file.

# ls /var/crash/127.0.0.1-2012-11-21-09\:49\:25/
vmcore  vmcore-dmesg.txt
# cp /var/crash/127.0.0.1-2012-11-21-09\:49\:25/vmcore-dmesg.txt /tmp/00123456-vmcore-dmesg.txt

For RHEL 6.0 to RHEL 6.3, 
Do take a look at Speeding up kernel crash hang analysis with the kernel log

Clearing memory cache

First thing you need is kernel 2.6.16
The command to clear the memory cache.

# sync
# echo 3 > /proc/sys/vm/drop_caches

sync -> refers to the tells the kernel that you want the data written to the disk
echo 3 > /proc/sys/vm/drop_caches -> "To free pagecache, dentries and inodes:"

From the article Invalidating the Linux buffer cache ,

To free pagecache:    
# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:
# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:
# echo 3 > /proc/sys/vm/drop_caches

References:

1. Invalidating the Linux buffer cache 
2. HOWTO: Clear filesystem memory cache
3. drop_caches

Tunable TCP/IP kernel options

Linux has placed each of the tunable kernel variable into the /proc virtual filesystem. The networking variables are in /proc/sys/net/ipv4. Here are some of the trimmed list.

# cd /proc/sys/net/ipv4

# ls -F

......... 

tcp_abc                           tcp_keepalive_time    tcp_sack
tcp_abort_on_overflow             tcp_low_latency       tcp_slow_start_after_idle
tcp_adv_win_scale                 tcp_max_orphans       tcp_stdurg
tcp_allowed_congestion_control    tcp_max_ssthresh      tcp_synack_retries
tcp_app_win                       tcp_max_syn_backlog   tcp_syncookies
tcp_available_congestion_control  tcp_max_tw_buckets    tcp_syn_retries
tcp_base_mss                      tcp_mem               tcp_thin_dupack
tcp_congestion_control            tcp_moderate_rcvbuf   tcp_thin_linear_timeouts
tcp_dma_copybreak                 tcp_mtu_probing       tcp_timestamps
tcp_dsack                         tcp_no_metrics_save   tcp_tso_win_divisor
tcp_ecn                           tcp_orphan_retries    tcp_tw_recycle
tcp_fack                          tcp_reordering        tcp_tw_reuse
tcp_fin_timeout                   tcp_retrans_collapse  tcp_window_scaling
tcp_frto                          tcp_retries1          tcp_wmem
tcp_frto_response                 tcp_retries2          tcp_workaround_signed_windows
tcp_keepalive_intvl               tcp_rfc1337
tcp_keepalive_probes              tcp_rmem

.....................

To make changes on the fly, you can just simply echo the value and pipe it to the options. For example,

# echo 1 > /proc/sys/net/ipv4/ip_forward

Inserting leap second causing kernel to hang for CentOS 5.7

I have encountered an interesting problem today. I encountered an error on one of my server

kernel: Clock: inserting leap second 23:59:60 UTC

It results in the hanging of one of my node. There is a report on this Bug 479765 -Leap second message can hang the kernel. Apparently, the CPU ran into a

According to the report, the widely circulated fix as root fixes the symptoms

# date -s "`date -u`"

BUG soft lockup - CPU#3 stuck for 10s! on CentOS 5

I noticed that the under intense CPU load on CentOS 5.4, I've got this bug BUG soft lockup - CPU#3 stuck for 10s! on CentOS 5.

This particular bug have been mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=484590

There are mentioned solution such as upgrading the kernel and changing the softlockup_thresh and softlockup_panic parameters at /proc/sys/kernel/softlockup_thresh and /proc/sys/kernel/softlockup_panic

I increase the /proc/sys/kernel/softlockup_thresh from 10 to 60. The original figure may be too aggressive.

# echo 30 > /proc/sys/kernel/softlockup_thresh

Check that the  /proc/sys/kernel/softlockup_panic = 0
# less  /proc/sys/kernel/softlockup_panic

Add this line to /etc/sysctl.conf (takes effect on next reboot):

kernel.softlockup_thresh=30

For more information, see

1. Why do I see "cpu soft lockup" messages in Red Hat Enterprise Linux on a Unisys E7600 or NEC 5800 Express with 96 cores?

Unable to find kernel source for CentOS 5

Sometimes when installing an application that requrie the kernel source of the CentOS 5, somehow even with yum install of the kernel-devel, the application is not able to find, one reason among the whole host of possbilities is that the pathing of the source is incorrect

In my case, at CentOS 5.4
If you go to /lib/modules/{kernel-version}. In my case 2.6.18-164.el5, you will see that there is a the build has a red background to it, apparently the path is wrong. You have to correct the soft-link error

#cd /lib/modules
#ln -s /usr/src/kernels/2.6.18-164.9.1.el5-x86_64

You will notice that the /lib/modules/source is linked nicely to "build"

Looking for solution.... CPU soft lockup detected for CentOS 5 and IBM BladeCentre

One of my IBM BladeCentre node hangs and the log messages generated
"BUG: soft lockup - CPU#1 stuck for 10s! [rpciod/1:3646]"

I have been looking around, but found a reference but not a solution though. I have seen other forum that 
RHEL 5.X CPU soft lockup detected in PAGE_LOCK_ANON_VMA - IBM BladeCenter and System x 

Symptom
While running the Red Hat Enterprise Linux 5.x (RHEL5) family of products, the kernel may report the following error:

BUG: soft lockup - CPU#XX stuck for 10s! Where XX can be the number of any processor in the system.

The associated stack backtrace points to page_lock_anon_vma as the code running at the time of the soft lockup detection.

Affected configurations
The system is configured with at least one of the following:

• Red Hat Enterprise Linux 5, any update

This tip is not system specific.
This tip is not option specific.
Note: This does not imply that the network operating system will work under all combinations of hardware and software. Please see the compatibility page for more information: http://www.ibm.com/servers/eserver/serverproven/compat/us/

Solution
This behavior will be corrected in subsequent families of Red Hat products. For more information, contact Red Hat at the following URL:

http://www.redhat.com/about/contact/

Memory Management - Preventing the kernel from dishing out more memory than required

I think for us who have been running computational jobs have seen the memory got eaten up by some buggy or stray applications. Hopefully the kernel kills it. But somehow you must have seen that the kernel may not have kill the culprit and the server go to a linbo.

Let's say if we wish to ensure that the kernel only gives out memory to processes equal to the physical memory, then we have to do the following at /etc/sysctl.conf or /etc/sysctl.d/myapp.conf

My assumption is that you have 10GB of swap and 20GB of memory and you wish the kernel to stop handling processes at  18GB RAM, then the calculation should be (swap size +  0.4 * RAM size)

So at /etc/sysctlf.conf, the configuration will be

vm.overcommit_memory = 2
vm.overcommit_ratio = 40

Note: The ratio is (40/100). For explanation of vm.overcommit_memory =2. Do look at Tweaking Linux Kernel Overcommit Behaviour for memory

Once the memory hits 18GB, the so-called OOM killer of the Linux kernel will kick in.

Another calculation example is that your RAM size and  SWAP size are the same and you wish exactly the physical memory to be used only. then

vm.overcommit_memory = 2
vm.overcommit_ratio = 0

For more information, do read

1. When Linux runs out of memory 
2. vm.overcommit_memory = 2, vm.overcommit_ratio = 0

Tweaking Linux Kernel Overcommit Behaviour for memory

You can change the behaviour of the Linux Kernel in regardings to its Overcommit Behaviour for memory. You can change the behaviour at /etc/sysctl.conf/ or /etc/sysctl.d/myconfig.conf

You will write something like

vm.overcommit_memory = (a integer from 0 to 2)

where
0 - means that the kernel will use predefined heuristics when deciding whether to allow such an overcommit. This is the default.
1 -always overcommits (Very dangerous)
2 - Prevents overcommits from exceeding a certain value. Within this mode, the total commit can not exceed the swap space(s) size + overcommit_ratio percent * RAM size. By default, the overcommit ratio is 50.

For example, do look at Memory Management - Preventing the kernel from dishing out more memory than required

Boot Time Parameters for Linux Kernel on CentOS 5

In CentOS, the default boot lader is GRUB (Grand Unified Boot loader). GRUB allows users to choose which operating system you would like to boost. In addition, GRUB can be used to inform or pass parameters to the kernel

How to modify GRUB

Step 1:
When GRUB start, press any keys and it will show you a boot menu.

Step 2:
In the menu, you can select the kernel to boot and press "e"

Step 3: You will see something like this on the screen

grub append > ro root=LABEL=/ rhgb quiet

Step 4a: Put in the necessary parameters. For example, if I wish to boot to "single"

grub append > ro root=LABEL=/ single

Step 4b: If I wish to boot into console with network support, use the parameter: init=/sbin/init 3

grub append > ro root=LABEL=/ init=/sbin/init 3

Step 4c: If I wish to specify the memory, use the parameters: mem=MEMORY_SIZE

grub append > ro root=LABEL=/ mem=1024M

Step 4d: If you wish to reboot the kernel after the kernel panic, you can set the parameter panic=SECOND

grub append > ro root=LABEL=/ panic=30

For more parameters, do read the excellent article on

1. 10 boot time parameters you should know about the Linux kernel
2. Booting to Single Mode for Centos LINUX