Relooking at usage of /home directory in Linux

File System in /home can be clustered very quickly nowadays where the amount of data are massive. Residing all the users in one /home directory will cause the administrator to quickly reach the limits of the File System especially if the File System is not a Parallel File System......

For more information, do take a look at Relooking at usage of /home directory in Linux on Linux Cluster

Quick check for directory size using du

The utilities du has a few parameter which could affect timing of checking directory size. But the difference is not that large 


Using du -csh on the current directory 

time du -csh .
544G

real    0m20.485s
user    0m0.308s
sys     0m2.146s

Using du -cmshis rather fast

$ time du -cmsh
544G

real    0m17.306s
user    0m0.266s
sys     0m1.951s

Using du -sh is the longest

$ time du -sh .
544G    .

real    0m21.790s
user    0m0.312s
sys     0m2.289s

Using passwd command to lock and unlock Linux Account

To lock a linux account, you can use the command

# passwd -l username
Locking password for user username.
passwd: Success

You will notice on the /etc/shadow file, that there will be a ! before the encrypted password string
user1:!...........................

To unlock the linux account, you can use the command

# passwd -u username
Unlocking password for user username.
passwd: Success.

Using Ctrl-R to search for terminal history

Press Ctrl and R together and the linux terminal will give  (reverse-i-search)`': Type a search word inside and the command will try to find the most recent command match

For example, If I input 'ssh' (without the quotation)

(reverse-i-search)`':`ssh': ssh node-c00

For more information

1. Ctrl+R to search and other terminal history tricks

Memory Consumption in Linux

This web article from http://emilics.com talks about memory consumption by a process in Linux. As Linux has a sophisticated  memory management system, measuring may not be that simple.

Inside the article, it mention VSZ (Virtual Memory Size) and Demand Paging, RSS (Resident Set Size) and Shared Libraries, PSS (Proportional Set Size) and more.

Do read the article from Memory Consumption in Linux

Clearing memory cache

First thing you need is kernel 2.6.16
The command to clear the memory cache.

# sync
# echo 3 > /proc/sys/vm/drop_caches

sync -> refers to the tells the kernel that you want the data written to the disk
echo 3 > /proc/sys/vm/drop_caches -> "To free pagecache, dentries and inodes:"

From the article Invalidating the Linux buffer cache ,

To free pagecache:    
# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:
# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:
# echo 3 > /proc/sys/vm/drop_caches

References:

1. Invalidating the Linux buffer cache 
2. HOWTO: Clear filesystem memory cache
3. drop_caches

Backup for /etc/passwd , /etc/group , /etc/shadow

/etc/passwd /etc/group /etc/shadow are very essential file systems for Linux. You should not be surprised that linux does do a backup of the /etc/passwd /etc/group /etc/shadow. They are represented by

/etc/passwd-
/etc/group-
/etc/shadow-

So in case your any of your /etc/passwd /etc/group /etc/shadow are corrupted, just do a copy and replace

Great YouTube Video on how to find the functions that cause a crash in a Linux application

A tutorial showing how to find the functions that cause a crash in a Linux application if all you know is the address of the crash using the common ldd and nm tools by Gilad Ben-Yossef. See
Gilad Ben-Yossef on using ldd and nm (You Tube)

OFED Performance Micro-Benchmark Latency Test

Open Fabrics Enterprise Distribution (OFED) has provided simple performance micro-benchmark has provided a collection of tests written over uverbs. The notes taken from OFED Performance Tests README

1. The benchmark uses the CPU cycle counter to get time stamps without a context switch.
2. The benchmark measures round-trip time but reports half of that as one-way latency. This means that it may not be sufficiently accurate for asymmetrical configurations.
3. Min/Median/Max results are reported.
   The Median (vs average) is less sensitive to extreme scores.
   Typically, the Max value is the first value measured Some CPU architectures
4. Larger samples only help marginally. The default (1000) is very satisfactory.   Note that an array of cycles_t (typically an unsigned long) is allocated once to collect samples and again to store the difference between them.   Really big sample sizes (e.g., 1 million) might expose other problems with the program.

On the Server Side

# ib_write_lat -a

On the Client Side

# ib_write_lat -a Server_IP_address

For more information, do take a look at OFED Performance Micro-Benchmark Latency Test

Software Updates for critical vulnerability in Java 7 is available

Oracle has released an emergency software update today to fix a vulnerability that is exploited by a zero-day Trojan horse called Mal/JavaJar-B, which was already identified as attacking Windows, Linux and Unix systems and being distributed in exploit kits "Blackhole" and "NuclearPack," making it far more convenient to attackers. More information of Update Release  of the Software Update.

More Information on Java 7 Vulnerability:

1. Oracle Java 7 Security Manager Bypass Vulnerability

Killing a SSH User Shell Session

As Administrator, when you do a notice that you will some of your users have idle and active ssh session. The idle ssh session could be be due to the hanged ssh session. So the question is how to remove the individual session without killing the active and genuine session.

First thing first, do

# w 

You may get some information like this

USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT 

.....
user1   pts/31   :24.0            08Oct12 22days  0.05s  0.05s -bash
user2   pts/24   :30              02Jan13  2days  0.66s  0.66s -bash
user3   pts/55   :17              12Nov12 59days  0.01s  0.01s -bash
.....

To get process id (PID) of the idle session, do the command

# ps -aux |grep 'pts/31'

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root     27552  0.0  0.0  61172   776 pts/1    S+   00:41   0:00 grep pts/31
546      30050  0.0  0.0  64188  1516 pts/31   Ss+   2012   0:00 -bash

Kill the Process

# kill -9 30050 

The idle ssh session has been removed. You can verify with the command "w"

Prepending number of lines to standard output using nl

nl copies files to standard output, with lines added. It is very flexible as it can prepend numbers to non blank lines and even justify left or right.

Usage 1: To add lines to standard output

$ nl -ba /etc/hosts.allow 

     1  #
     2  # hosts.allow   This file describes the names of the hosts which are
     3  #               allowed to use the local INET services, as decided
     4  #               by the '/usr/sbin/tcpd' server.
     5  #
     6

where -ba => prepend numbers to all lines

Usage 2: To add lines to non-blank text

$ nl -bt /etc/hosts.allow
     
     1  #
     2  # hosts.allow   This file describes the names of the hosts which are
     3  #               allowed to use the local INET services, as decided
     4  #               by the '/usr/sbin/tcpd' server.
     5  #

where -bt => prepend lines to non-blank lines

Usage 3: Format the numbering to Left Justify

$ nl - bt -nln /etc/hosts.allow

1       #
2       # hosts.allow   This file describes the names of the hosts which are
3       #               allowed to use the local INET services, as decided
4       #               by the '/usr/sbin/tcpd' server.
5       #

where -bt => prepend lines to non-blank lines
-nln => format the number to be left-justify

Usage 4: Format the numbering to Right Justify

 $ nl -bt -nrn /etc/hosts.allow

     1  #
     2  # hosts.allow   This file describes the names of the hosts which are
     3  #               allowed to use the local INET services, as decided
     4  #               by the '/usr/sbin/tcpd' server.
     5  #

Debugging gmond issue quickly

If you need to debug gmond issues quickly, use the command

# /usr/sbin/gmond --debug=9

loaded module: core_metrics
loaded module: cpu_module
loaded module: disk_module
loaded module: load_module
loaded module: mem_module
loaded module: net_module
loaded module: proc_module
loaded module: sys_module
loaded module: multicpu_module
udp_recv_channel mcast_join=NULL mcast_if=NULL port=8649 bind=NULL
tcp_accept_channel bind=NULL port=8649
Unable to create tcp_accept_channel. Exiting.

For more information and example see: 

1. Ganglia Node unable to update Gmetad Node 
2. Gmond dead but subsys locked for ganglia monitoring daemon

Default values used by useradd command

When the users issues a useradd command, the useradd commands reads the /etc/default/useradd and the /etc/login.defs and determine the default value for useradd. To display the value for /etc/defaults/useradd, see Displaying defaults for useradd

Do read also Modifying default template for user settings in Linux which will add in the files settings for the users.

To read the /etc/login.defs,

# vim /etc/login.defs

# Password aging controls:
#
#       PASS_MAX_DAYS   Maximum number of days a password may be used.
#       PASS_MIN_DAYS   Minimum number of days allowed between password changes.
#       PASS_MIN_LEN    Minimum acceptable password length.
#       PASS_WARN_AGE   Number of days warning given before a password expires.
#
PASS_MAX_DAYS   99999
PASS_MIN_DAYS   0
PASS_MIN_LEN    5
PASS_WARN_AGE   7

#
# Min/max values for automatic uid selection in useradd
#
UID_MIN                   500
UID_MAX                 60000

#
# Min/max values for automatic gid selection in groupadd
#
GID_MIN                   500
GID_MAX                 60000

#
# If defined, this command is run when removing a user.
# It should remove any at/cron/print jobs etc. owned by
# the user to be removed (passed as the first argument).
#
#USERDEL_CMD    /usr/sbin/userdel_local

#
# If useradd should create home directories for users by default
# On RH systems, we do. This option is overridden with the -m flag on
# useradd command line.
#
CREATE_HOME     yes

# The permission mask is initialized to this value. If not specified,
# the permission mask will be initialized to 022.
UMASK           077

# This enables userdel to remove user groups if no members exist.
#
USERGROUPS_ENAB yes

# Use MD5 or DES to encrypt password? Red Hat use MD5 by default.
MD5_CRYPT_ENAB yes

ENCRYPT_METHOD MD5

Quick Listing of users who have current login session

If you wish to have quicklisting of users who are logging on to your servers, you can use the command "users". If a user is running multiple session, they will appear multiple time.

# users
root root user1 user2 user3 user3

There are more comprehensive tools like who and finger. Will write in future blog entries.

Using Chage to manage password expiration and aging

As administrators, tools like chage to help manage the /etc/shadow information. /etc/shadow contains information such as

myuseid:$xxxxxxxxxeeerrrrr:15607:0:900:10:0::

Column	Description
1	UserID
2	Encrypted Password
3	Set the number of days since January 1st, 1970 when the password was last changed.
4	Minimum number of days between password changes to MIN_DAYS
5	Maximum number of days during which a password is valid
6	Set the number of days of warning before a password change is required
7	Set the number of days of inactivity after a password has expired before the account is locked
8	Set the date or number of days since January 1, 1970 on which the userid account will no longer be accessible. The date may also be expressed in the format YYYY-MM-DD

Best if you can use the command chage. For example, you can use the command

1. Listing of  password details

# chage --list  username

Last password change                                    : Feb 03, 2012
Password expires                                        : Jul 22, 2013
Password inactive                                       : Jul 22, 2013
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 60
Number of days of warning before password expires       : 10

2. Disable password aging for an user account

# chage -m 0 -M 99999 -I -1 -E -1 username

-m 0 (Min number of  days between password change to 0)
-M 99999 (Max Number of days between password change to 99999)
-I -1 (Set "Password Inactive" to never)
-E -1 (Set "Account expires" to never)

Last password change                                    : Feb 03, 2012
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 10

For more complete information, see

1.  6.6. Linux Password & Shadow File Formats
2. 7 Examples to Manage Linux Password Expiration and Aging Using chage

Udev Rules Documention

Taken from the documentation Writing udev rules

Udev is targetted to provide a userspace for a dynamic /dev directory with persistent device naming. If  you need to lock down your device naming, do take a look at this good writeup

Unable to open /dev/sdb with fdisk

Fdisk is a menu driven program for creation and manipulation of partition tables. The device is usually something like /dev/sda, /dev/sdb. A device name refers to the entire disks. /dev/sd? is the partition of the device. For example, /dev/sda1 refers to the first partition of the first device.

If you issued a command and you receive a corresponding message "unable to open /dev/sdb"

# fdisk /dev/sdb

Unable to open /dev/sdb

Linux is unable to locate or find the partition. One method to verify that it is so, do a listing of the
devices fdisk can see. In this example below, the partition has been created already.

# fdisk -l

Disk /dev/sdb: 2997.4 GB, 2997426536960 bytes
255 heads, 63 sectors/track, 364416 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      267349  2147480811   83  Linux

WARNING: The size of this disk is 3.0 TB (2997400633344 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Once you have verified the presence of the device, do a fdisk /dev/sdb again

Another look at Changing hostname for CentOS

This is an extension of the article Changing the hostname on CentOS. You can replace Step 2 and 3 found in the article with a one-line hostname command

# hostname www.hostserver.com

To test the hostname

# hostname

www.hostserver.com

Using MAUI checkjob command

The command checkjob is very useful if you are using MAUI or MOAB to identify jobs issues

1. One of my favourite commands to check why a Job is IDLE is

$ checkjob -v 6235

The above commands display reasons why idle job is blocked ignoring node state and current node utilization constraints

node-c00              accepted : 8 tasks supported
node-c02              accepted : 8 tasks supported
node-c03              accepted : 8 tasks supported
node-c04              accepted : 8 tasks supported
node-c05              accepted : 8 tasks supported
node-c06              accepted : 8 tasks supported
node-c07              rejected : CPU
node-c08              rejected : CPU
node-c09              rejected : CPU

Some of the reasons why it is rejected can be due to. The list is not exhausted.

Features - queue or features does not meet requirement 
CPU - CPU does not match requirements
State - Used by other jobs

For more information, see Adaptive Computing checkjob. Do note if you are using MAUI some of the checkjob features are available in the commercial scheduler.