Finding the user who used excessive memory and crash the Server

Using linux tools, we can quickly deduce the users who have crashed the system. For me, one of the harder to trace are researchers who run large memory arrays. And if the array is not calculated correctly, it will consume excessive memory than what the server can provide including physical and swap, the system will crash.

One of the fastest is to check who is the users who is last online and having crash and check with /var/log/messages. I just issue the command.

# last

reboot   system boot  2.6.18-238.9.1.e Mon Feb 25 08:42         (1+01:12)
user1    pts/30       :3.0             Sun Feb 24 15:19 - crash  (17:22)
user1    pts/36       :3.0             Sun Feb 24 14:26 - crash  (18:15)

If you can also take a look at the log file, you can see that the crash appeared after the last user log in.

Feb 24 14:25:19 Head-Node gconfd (user1-29338): Resolved address "xml:readwrite:/home/user1/.gconf" to a writable configuration source at position 0
Feb 24 16:15:11 Head-Node mmfs: Error=MMFS_PHOENIX, ID=0xAB429E38, Tag=2739215:   Reason code 668 Failure Reason Lost membership in cluster nsd1-nas. Unmounting file systems.
Feb 24 16:15:11 Head-Node mmfs: Error=MMFS_PHOENIX, ID=0xAB429E38, Tag=2739215:
Feb 24 16:18:08 Head-Node kernel: TCP: time wait bucket table overflow
Feb 24 16:18:08 Head-Node kernel: mmfsd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
Feb 24 16:18:08 Head-Node kernel:
Feb 24 16:18:08 Head-Node kernel: Call Trace:
Feb 24 16:18:08 Head-Node kernel:  [] out_of_memory+0x8e/0x2f3
Feb 24 16:18:08 Head-Node kernel:  [] __wake_up+0x38/0x4f
Feb 24 16:18:08 Head-Node kernel:  [] autoremove_wake_function+0x0/0x2e
Feb 24 16:18:08 Head-Node kernel:  [] __alloc_pages+0x27f/0x308
Feb 24 16:18:08 Head-Node kernel:  [] read_swap_cache_async+0x45/0xd8
Feb 24 16:18:08 Head-Node kernel:  [] swapin_readahead+0x60/0xd3
Feb 24 16:18:08 Head-Node kernel:  [] __handle_mm_fault+0xb62/0x1039
Feb 24 16:18:15 Head-Node kernel:  [] thread_return+0x62/0xfe
Feb 24 16:18:28 Head-Node kernel:  [] do_page_fault+0x4cb/0x874
Feb 24 16:18:53 Head-Node kernel:  [] hrtimer_cancel+0xc/0x16
Feb 24 16:19:16 Head-Node kernel:  [] do_nanosleep+0x47/0x70
Feb 24 16:19:29 Head-Node kernel:  [] hrtimer_nanosleep+0x58/0x118
Feb 24 16:19:33 Head-Node kernel:  [] error_exit+0x0/0x84
 

To manage memory more effectively, do look at

1. Tweaking the Linux Kernel to manage memory and swap usage  

SingCERT Alerts - SPAM Emails from Yahoo Accounts

The Singapore Computer Emergency Response Team (SingCERT) has received reports of spam emails from Yahoo! accounts containing links to websites selling “work from home” schemes and packages.

The spam emails may have the following characteristics:

• Empty subject line
• Email content contains just a website link

For more information, see Yahoo plugs hole that allowed hijacking of email accounts

Running OpenMPI in oversubscribe nodes

Taken from OpenMPI FAQ  21. Can I oversubscribe nodes (run more processes than processors)?
Open MPI basically runs its message passing progression engine in two modes: aggressive and degraded.

• Degraded: When Open MPI thinks that it is in an oversubscribed mode (i.e., more processes are running than there are processors available), MPI processes will automatically run in degraded mode and frequently yield the processor to its peers, thereby allowing all processes to make progress.
• Aggressive: When Open MPI thinks that it is in an exactly- or under-subscribed mode (i.e., the number of running processes is equal to or less than the number of available processors), MPI processes will automatically run in aggressive mode, meaning that they will never voluntarily give up the processor to other processes. With some network transports, this means that Open MPI will spin in tight loops attempting to make message passing progress, effectively causing other processes to not get any CPU cycles (and therefore never make any progress).

Example of Degraded Modes (Running 4 Slots on 1 Physical  cores). MPI knows that there is only 1 slot and 4 MPI process are running on the single slot.

$ cat my-hostfile
localhost slots=1
$ mpirun -np 4 --hostfile my-hostfile a.out

Example of Aggressive Modes (Running 4 slots on 4 or more Physical Cores). MPI knows that there is at least 4 slots for the 4 MPI process.

$ cat my-hostfile
localhost slots=4
$ mpirun -np 4 --hostfile my-hostfile a.out

Error when yum install ganglia on CentOS 5.8

 I was doing a yum install of Ganglia on CentOS 5.8,

# yum install php rrdtool ganglia ganglia-gmetad ganglia-gmond ganglia-web httpd

I got the error as below. 

ganglia-web-3.1.7-3.el5.rf.x86_64 from rpmforge has depsolving problems
  --> Missing Dependency: php-gd is needed by package ganglia-web-3.1.7-3.el5.rf                                                              .x86_64 (rpmforge)
rrdtool-1.4.7-1.el5.rf.x86_64 from rpmforge has depsolving problems
  --> Missing Dependency: libdbi.so.0()(64bit) is needed by package rrdtool-1.4.                                                              7-1.el5.rf.x86_64 (rpmforge)
ganglia-web-3.1.7-3.el5.rf.x86_64 from rpmforge has depsolving problems
  --> Missing Dependency: php is needed by package ganglia-web-3.1.7-3.el5.rf.x8                                                              6_64 (rpmforge)
Error: Missing Dependency: php is needed by package ganglia-web-3.1.7-3.el5.rf.x                                                              86_64 (rpmforge)
Error: Missing Dependency: libdbi.so.0()(64bit) is needed by package rrdtool-1.4                                                              .7-1.el5.rf.x86_64 (rpmforge)
Error: Missing Dependency: php-gd is needed by package ganglia-web-3.1.7-3.el5.r                                                              f.x86_64 (rpmforge)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest

To avoid the error during the installation, see Installing and Configuring Ganglia on CentOS 5.8
for more information.

Step 1, install the libdbi.so.0(64bits) package

1. Make sure you have the RPM Repositories installed. For more information, see Useful Repositories for CentOS 5
2. Make sure installed the libdbi-0.8.1-2.1.x86_64.rpm for CentOS 5.9 be installed on the CentOS 5.8. Apparently, there is no conflict or dependency issues. See RPM Resource libdbi.so.0()(64bit)

   # wget 
   ftp://rpmfind.net/linux/centos/5.9/os/x86_64/CentOS/libdbi-0.8.1-2.1.x86_64.rpm

   
   

   # rpm -ivh libdbi-0.8.1-2.1.x86_64.rpm

3. Install PHP 5.4. See Installing PHP 5.4 on CentOS 5
4. Finally do the 

# yum install php rrdtool ganglia ganglia-gmetad ganglia-gmond ganglia-web httpd

5. For the rest of the steps do look at Installing and Configuring Ganglia on CentOS 5.8

Displaying Repository List with yum

If you wish to display a list of configured repositories which are enabled, you can use the command

# yum repolist

repo id        repo name                                                  status
epel           Extra Packages for Enterprise Linux 5 - x86_64              7,242
ius            IUS Community Packages for Enterprise Linux 5 - x86_64        229
rpmforge       Red Hat Enterprise 5 - RPMforge.net - dag                  11,158
xcat-2-core    xCAT 2 Core packages                                           14
xcat-dep       xCAT 2 depedencies                                             38
repolist: 18,681

# yum -v repolist

Repo-id      : epel
Repo-name    : Extra Packages for Enterprise Linux 5 - x86_64
Repo-revision: 1360778202
Repo-tags    : binary-x86_64
Repo-updated : Thu Feb 14 01:59:23 2013
Repo-pkgs    : 7,242
Repo-size    : 5.4 G
Repo-mirrors : http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=x86_64
Repo-expire  : 3,600 second(s) (last: Sat Feb 16 01:03:31 2013)

Repo-id      : ius
Repo-name    : IUS Community Packages for Enterprise Linux 5 - x86_64
Repo-revision: 1360910159
Repo-updated : Fri Feb 15 14:36:21 2013
Repo-pkgs    : 229
Repo-size    : 441 M
Repo-mirrors : http://dmirr.iuscommunity.org/mirrorlist/?repo=ius-el5&arch=x86_64
Repo-expire  : 3,600 second(s) (last: Sat Feb 16 01:03:35 2013)

Repo-id      : rpmforge
Repo-name    : Red Hat Enterprise 5 - RPMforge.net - dag
Repo-updated : Fri Dec 21 10:44:36 2012
Repo-pkgs    : 11,158
Repo-size    : 5.8 G
Repo-baseurl : http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/
Repo-mirrors : http://apt.sw.be/redhat/el5/en/mirrors-rpmforge
Repo-expire  : 3,600 second(s) (last: Sat Feb 16 01:03:38 2013)

Repo-id      : xcat-2-core
Repo-name    : xCAT 2 Core packages
Repo-updated : Wed Nov 28 11:01:51 2012
Repo-pkgs    : 14
Repo-size    : 3.2 M
Repo-baseurl : https://sourceforge.net/projects/xcat/files/yum/2.7/xcat-core/
Repo-expire  : 3,600 second(s) (last: Sat Feb 16 01:03:44 2013)

Repo-id      : xcat-dep
Repo-name    : xCAT 2 depedencies
Repo-updated : Wed Feb  6 05:37:39 2013
Repo-pkgs    : 38
Repo-size    : 82 M
Repo-baseurl : https://sourceforge.net/projects/xcat/files/yum/xcat-dep/rh5/x86_64/
Repo-expire  : 3,600 second(s) (last: Sat Feb 16 01:03:51 2013)

repolist: 18,681