ntpdate no server suitable for synchronization found

I was configuring one of head node of our cluster server to synchronise from our time server in our organisation, and in turn the head node becomes a NTP Server for the compute nodes that on a private IP addresses. As I was configuring the compute nodes and encounter "ntpdate[4933]: no server suitable for synchronization found"

I configured the ntp servers and ntp clients according to Setting up NTP Server for Local Network. And yet the error remain unresolved. Finally, I came across a solution

1. Open /etc/ntp/step-tickers and replace with the name of your local NTP Server
2. Open /etc/ntp/ntpservers and replace with the name of your local NTP Server
3. Restart your ntp services. You should eradicate the problem

Configuring NFS Server for Performance

Finally, a more detailed writeup on NFS configuration which I was hoping to blog. Finally it is done.
Do read Configuring NFS Server for Performance

Configure TCP for faster connections and transfers

On a default Linux Box, the TCP settings may not be optimise for “bigger” available network bandwidth connections and transfer available for 100MB+. Currently, most TCP settings are optimise for 10MB settings. For more information, on how to configure, read my Linux Cluster Blog Configure TCP for faster connections and transfers

NFS Client Recommended Configuration

NFS is one of the most commonly used file networked file system. Quite often we build it ourselves using standard out-of-the-box hardware we find. This NFS Client Recommended Configuration (my blog too) shows in slightly more details how to configure the mounting of the NFS clients

Compiling with MPI Parallel Library

This is a good site for MPICH Parallel Library Compilation from Duke University

booh - photo album generator

Booh is a free software static Web-Album generator for Linux. It takes series of photos and videos, and automatically build static web pages to browse them.

Booh has a number of unqiue features compared to the other dynamic web-album genertor. The fulling listings is at the Booh website. Here a subset of the list

1. automatic rotation of portrait photos thanks to information read in .jpg file (EXIF)
2. immediate display of images (preloading in browser)
3. sub-albums support
4. multi-processor support to speed up thumbnails generation
5. Full video support, including thumbnailing (with seek time specification possibility) and transcoding
6. ..... and much more. See Booh Website
   

Using yum to download the rpm package that have already been installed

Firstly, download the yum-utils method

yum -y install yum-utils

To download a package that have been installed, please use yumdownloader instead

yumdownloader ganglia-gmond

Error: Missing Dependency: librrd.so.2()(64bit) is needed by package ganglia-gmetad (epel)

This is the continuation of Blog Entry Error: Missing Dependency: rrdtool = 1.2.27-3.el5 is needed by package rrdtool-perl

1. When I'm installing Ganglia on my CentOS, I've did the following:
   

   # yum install rrdtool ganglia ganglia-gmetad ganglia-gmond ganglia-web httpd php

   However, instead I got the followings "Error: Missing Dependency: rrdtool = 1.2.27-3.el5 is needed by package rrdtool-perl"
   
2. To resolve it, similar to the earlier Blog Entry, I install rrdtool first. But when I use the command
   

   # yum install ganglia ganglia-gmetad ganglia-gmond ganglia-web httpd php

   There is still error "Error: Missing Dependency: librrd.so.2()(64bit) is needed by package ganglia-gmetad (epel)...."
3. To resolve the issue, you have to manually get rid of the latest rrdtool (x86-64) installed by EPEL and install the rrdtool-1.2.27-3.el5.x86_64
   

   # yum remove rrdtool-1.3.8-2.el5.rf.x86_64

   
   

   # yum install rrdtool-1.2.27-3.el5.x86_64

4. Install the Ganglia and its related tools
   

   # yum install ganglia ganglia-gmetad ganglia-gmond ganglia-web httpd php

Using CentOS GUI Package Manager

If you wish to X forward the CentOS GUI Package Manager to your Windows Terminal ot Linux Box, you may wish to do the following first....

Step 1: Establish the Remote Connection

1. Using Putty, Xming to forward X11 (For Windows Terminal)
2. Speeding and Compressing X forwarding Traffic (For Linux Box)

Step 2: Install pirut

1. Using yum effectively (Part 2)

Implementation of HPC as a Service - the Shanghai Supercomputer Center

The Shanghai Supercomputer Center (SSC) was founded in December 2000. It is the first high performance computing platform open to the whole public in China and currently is the country’s leading supercomputer centre.......

Read more on HPC as a Service from Scientific Computing World

Installing Cairo-Dock for Fedora 11

Cairo-Dock is an animated application launch bar for the desktop, comparable to the dock in Mac OS X. Cairo-Dock. You can use Cairo-Dock with OpenGL and is compatible with Compiz-Fusion, Beryl, Compiz and Xcompmgr,

Currently Cairo-Dock packages are available on rpmfusion-free repositories.. To install cairo-dock, simply
# yum install cairo-dock\*


For further information

1. Cairo-Dock website (non-English)
2. Cairo-Dock Wiki (English)
3. Screenshots of Cairo-Dock

Sun HPC ClusterTools 8.2.1

Sun HPC ClusterTools 8.2.1 software is an integrated toolkit that allows developers to create and tune Message-passing Interface (MPI) applications that run on high performance clusters and SMPs. Sun HPC ClusterTools 8.2.1 is based on OpenMPI v1.3.4

OpenMPI 1.4 is ready for download

OpenMPI v1.4 is ready for download. According to the OpenMPI v1.4 README file, the different between v1.4 and the previous version 1.3.4 is the fix for Libtool 2.2.6b's issues.

Showing uptime

uptime is a good old favourite where system admin uses it to show how long the system has been up and running.

$ uptime
01:17:33 up 1 day, 18:16,  4 users,  load average: 0.00, 0.02, 0.00

2 Storage Trends from SuperComputing 2009

This is an interesting articles about the rise of smaller storage vendors who is making wave with their storage innovation. An interesting read Two Storage Trends From SuperComputing 2009

Fixing your system with FirstAidKit

Since Fedora 10, FirstAidKit package has been included in the Fedora Distribution. Basically, Firstaidkit is an automated recovery tool that brings together common recovery processes and applies them to a system. It is built on top of a pluggable architecture.

1. To install FirstAidKit
   # yum install firstaidkit
2. To run the automated process of the recovery tool, run
   # firstaidkit -a
3. To view a list of available plug-ins
   # firstaidkit --list

For more information on FirstAidKit, see First Aid Kit by Fedora

Tips on useradd and usermod commands

Today it is back to the basis again.

useradd and usermod are very often used utilities. They are several useful paramenters which we can use for the creation and maintenance of user accounts.

• -c (Comments and Descriptions of the account. Using quotation is necessary if you are using multiple words
• -d (home directory of the account. If -m is used, copy the home directory as well)
• -e (Assign a expiration date for the account in MM/DD/YYYY format)
• -f (Set the number of days after a password expires until the account is permanently disabled. The default option is -l which disable this option)
• -g (primary group that the account belongs to)
• -G (list of groups that user belongs to)
• -l (Change login name of the account to the name supplied after -l option)
• -u (Change the userid for the account) 
  

Forcing NIC to operate at Full Duplex using Ethtool

This is a follow up blog entry from Using ethtool to check and change Ethernet Card Settings

To force the NIC to use full duplex, 100Mbps, and autonegotiate off, you can
put this in /etc/sysconfig/network-scripts/ifcfg-eth0
--------------------------------------------------------------------------
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
------------------------------------------------------------------------- 

This is taken from ethtool from softpanorama.

Using ethtool to check and change Ethernet Card Settings

ethtool is used for querying settings of an ethernet device and changing them. ethtool is the replacement tool for mii-tool

# ethtool eth0

 As seen in the above diagram, ethtool information is very informative and helpful for diagnostic

Other useful  command includes

# ethtool -i eth0 (provide information regarding eth0 including drivers)

# ethtool -S eth0 (provide information regarding the speed and other information)

Taken from ethtool from softpanorama. You can use ethtool command to set specific parameter on the NIC

# ethtool -s eth0 speed 100 duplex full autoneg off

Mounting CD or DVD with the command line

Back to the Basics.....

To mount a CD-ROM or DVD on LINUX, you can use the following command,

# mount -t iso9660 -o ro /dev/dvd /media/dvd

(assuming /media/dvd is the mount point)

Compiz and Emerald to make Linux Desktop cool

Emerald is a custom window decorator that is shipped with Compiz. It is one of the most and easily configurable window manager.

Emerald is usually available as an additional package and has to install seperately.
# aptitude install emerald

Once installed you can launch the Emerald Theme Manager and choose your preferred theme or customise your look and replace. Once configured
$ emerald --replace

To import interesting theme, you can go to http://www.compiz-themes.org/ to download the latest theme. To load the theme, simply use the Emerald Theme Manager to import in the .emerald file

Awesome.

Speeding and Compressing X forwarding Traffic

To run the an X application over SSH connection, you can use the

$ ssh -X user@computername.com

Do note that for the remote Server shich you are connecting to must have X forwarding enabled. To configure, go to /etc/ssh/ssh_config/

X11Forwarding yes

If the SSH is setup with trusted X11 Forwarding ie in the /etc/ssh/ssh_config file,

ForwardX11Trusted yes

You can compress and speed up the X forwarded connection

$ ssh -Y -C user@computername.com

• -Y to enable trusted X11 forwarding. Trusted X11 forwardings are not subjected to the X11 SECURITY extension controls. So it will boost speed.
• -C to compress the data

Articles on Parallel::ForkManager on Perl

You may want to take a look at

1. Use Parallel Processing For Faster Perl Scripts and
2. Parallel::ForkManager - A simple parallel processing fork manager

Installing Subversion on Fedora 11

I was installing Subversion on Fedora 11 using the "Subversion from CentOS" taken from CentOS Wiki and the instruction set work without a hitch. Just a few important commands to take note

Importing commands to take note:

1. Initial repository layout for MyTestProject
   # svn import /tmp/mytestproject to file://var/www/svn/repos/mytestproject -m
   
   
2. Checking Out
   $ svn co http://yoursvnserver/repos/mytestproject
   
   
3. Edit and Commit
   $ svn co http://yoursvnserver/repos/mytestproject
   $ svn commit -m "Added a line to testconf1.cfg."
   
   
4. Adding and Commit
   $ svn co http://yoursvnserver/repos/mytestproject
   $ svn add configurations/AnotherTestConf2.cfg
   $ svn commit -m "Added something else"
   
   
5. Deleting
   $ svn co http://yoursvnserver/repos/mytestproject
   $ svn delete configurations/AnotherTestConf2.cfg
   $ svn commit -m "Delete something"
   
   
6. Reverting to Previous Version
   $ svn log http://yoursvnserver/repos/mytestproject
   (You will see a complete list of revision numbers along with the comments)
   $ svn co -r (some number) http://yoursvnserver/repos/mytestproject
   
   
7. To add back files to the current repository
   $ svn copy http://yoursvnserver/repos/mytestproject/oldversionfile@NNN ./oldversionfile (where NNN is the revision number)
   

For more informatio

1. Version Control with Subversion (Free Online Documents and Ebook)
   

Using read-edid to extract useful information on PnP Monitor on Ubuntu or Linux Mint

Read-edid utility is a useful tool to help you  extract useful information on PnP Monitor.

According to the read-edid website, if you can't find the data to set your monitor up under XFree86, then read-edid is for you; it should auto detect all necessary info and should even provide a few modes with better refresh rates/resolutions than the standard VESA ones. However, as of now it won't do everything for you; you'll have to edit XF86Config yourself.

To install read-edid utility in Ubuntu or Linux Mint, just do a
# aptitude install read-edid OR
# apt-get install read-edid


The  data obtained by get-edid is in a binary format, so the parse-edid command is available to interpret  it  and  generate  a  human-readable block  of  text  information  that  can  also be included in an XFree86 XF86Config file. It is customary  to  invoke  get-edid  and  parse-edid  together  in  a pipeline.
# get-edid | parse-edid


More information:

1. Get-edid Website

Webkit Open Source Project

According to the WebKit Open Source Project website, WebKit is an open source web browser engine. WebKit is also the name of the Mac OS X system framework version of the engine that's used by Safari, Dashboard, Mail, and many other OS X applications.
 
The lightweight browser midori is another example

# aptitude install midori

Diagnosing Network Information Services (NIS) Issues guide from IBM

Although the article Network Information Services (NIS and NIS+) Guide is meant more for the IBM AIX System, however, you can get very good troubleshooting tips for the Linux NIS

1. Troubleshooting NIS-Related Problems
2. Troubleshooting NIS+ Namespace Administration Problems
3. Troubleshooting NIS+ Namespace Database Problems 

Using aptitude vs apt-get

Taken the Debian Wiki on Aptitude

Aptitude is an Ncurses based FrontEnd to Apt, the debian package manager. Since it is text based, it is run from a terminal or a CLI (command line interface). Aptitude has a number of useful features, including:

1. a mutt-like syntax for matching packages in a flexible manner
2. mark packages as "automatically installed" or "manually installed" so that packages can be auto-removed when no longer required
3. colorful preview of actions about to be taken
4. dselect-like persistence of user actions
5. the ability to retrieve and display the Debian changelog of most packagesAptCLI-like (= apt-get + apt-cache) command line mode ("aptitude install foo")
6. Score-based and (usually) smarter dependency resolver than apt-get 

For Linux Mint 6, it already come with Aptitude pre-installed. Some useful commands are

# aptitude (To run from the terminal)

After running it, use:

• F10 to access the menu and use aptitude. This is the main key.
• ? for help
• The 'up', 'down', 'left', 'right' keys to navigate.
• The 'Enter' key to select
• The '+' or '-' key to install/update or remove a package
• The 'g' key to preview/confirm actions
• 'q' to quit 

 You can also commands simliar to apt-get like

# aptitude install mplayer

# aptitude update mplayer

# aptitude upgrade mplayer

# aptitude remove mplayer
# aptitude search mplayer
# aptitude --help (To get the rest of the options)

What is the advantages of using Aptitude versus apt-get? There are a lot similarities and overlap. From what I have gather from the forum and discussion, the advantages of Aptitude over Apt-get are as followed:

1. Better Dependency Handling
2. Better at removing obsolete packages from your system
3. A different log presentation of the what packages have been affected during the installation / removal of package. On the other hands, apt-get tend to log what was present the information on the console during installation.
   

What's my verdict? I pretty neutral to both. But may stick to apt-get due to familiarity.... :)

Using Gummi LateX editor for Linux

Gummi is a LaTeX editor for the Linux platform, written in Python and GTK. It was designed with simplicity in mind, but hopes to attract both novice and more advanced users.

It has received good reviews and is growing into an excellent tool for Latex users.

If you are using Fedora 11. Good news indeed. you can download the rpm to install. However, there is a lot of dependency from GTK and Python, you may want to use "yum localinstall" to resolve dependences

After downloading the rpm,

# yum localinstall gummi-0.4.2-1.noarch.rpm

Dstat a versatile resource statistics tool

Taken from DAG

Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).

Some of the features includes:

• Combines vmstat, iostat, ifstat, netstat information and more
• Shows stats in exactly the same timeframe
• Enable/order counters as they make most sense during analysis/troubleshooting
• Modular design
• Written in python so easily extendable for the task at hand
• Easy to extend, add your own counters (please contribute those)
• Includes about 10 external plugins to show how easy it is to add counters
• Can summarize grouped block/network devices and give total numbers
• Can show interrupts per device
• Very accurate timeframes, no timeshifts when system is stressed
• Shows exact units and limits conversion mistakes
• Indicate different units with different colors
• Show intermediate results when delay > 1
• Allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs

Make sure you have rpmforge repository installed.

# yum install dstat

Some of the useful commands are as followed:

# dstat --help 

# dstat --full 

# dstat --vmstat 

# dstat --mem 

# dstat --load

Turn off checking for signature during yum localinstall

yum localinstall is a wonderful utility where I can download and install an rpm package and let yum resolve all the dependencies issues for me. Occasionally, during installation, if I don't want to do a signature check. I can use this commands with a --nogpgcheck flag

# yum localinstall gummi-0.4.2-1.noarch.rpm --nogpgcheck

And it work wonderfully

Linux File System throughput performance

The articles from Linux Magazine

1. I Feel the Need for Speed: Linux File System Throughput Performance, Part 1 and
2. IOzone Performance Exploration, Part 2: The Rest of the Crowd (Almost)

are interesting writeup on throughput performance among the commonly and less-well known Linux file system. the benchmarking tool used is IOzone. IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. IOzone has been ported to many machines and runs under many operating systems.

Do take a look

Using glxinfo to diagnose 3D acceleration issues

glxinfo is a command-line tool that can help you diagnose problems with your 3D acceleration setup.

How do we check that our 3D Acclerlation are working, you can use the command.

# glxinfo |grep render

direct rendering: Yes
OpenGL renderer string: ATI MOBILITY RADEON 9600/9700 Series

The OpenGL rendering tells you what driver was used in the rendering. It distinuishes between software and hardware rendering. The article from dri.freedesktop.org titled glxinfo provides a good articles on how glxinfo  information canbe interpreted to understand on hardware / indirect / software rendering which will help to understand 3D Acceleration Issues. For 3D Acceleration performance hardware is the best. The slowest is software obviously.  If you are using Ubuntu, you can go to the Ubuntu X/Troubleshooting to take a look

Limiting Latencies in the Cloud

This is a very good article on HPCwire that summarise on Limiting the Latencies in the Cloud by Dr. Casimer DeCusatis and Todd Bundy. A must-read on the convergence of the numerous protocol used in storage, HPC and Enterprise Computing

Changing Default Editor for visudo

Instead of editing /etc/sudoers directly, it is better to edit /etc/sudoers by using visudo.

The beauty of visudo is  that visudo locks the sudoers file against multiple simultaneous edits, provides basic sanity checks, and checks for parse errors. Look at man page for more information.

However, visudo uses nano as a default editor. How do I change the editor as I'm very used to vi or vim to do my editing. Here is a simple way of doing.

Put the following inside your .bashrc file

# vim .bashrc

-------------------------------------------
export EDITOR=vim
sudo visudo
--------------------------------------------

# source .bashrc

You should be able to use vim as the default editor for visudo

Ksplice - Install updates without Rebooting

Ksplice Uptrack and discover how you can:

1. Keep your systems up to date and secure, without rebooting.
2. Save time by updating in seconds, while your systems are running.
3. Avoid disruptive downtime.
4. Prevent costly security incidents by patching in a timely manner.

It is free for Ubuntu 9.04 and 9.10. For more technical details on how ksplice works, you can take a look at
ksplice technical paper

Cannot open/read repomd.xml file for repository: epel

If you encountering this error with yum, you may want to read the Blog Entry Yum error: Cannot find a valid baseurl for repo: with CentOS

Yum error: Cannot find a valid baseurl for repo: with CentOS

If you encounter the error " Error: Cannot find a valid baseurl for repo: ". There are a few possible steps which might fix as least for myself

Step 1: You may want to uncomment the baseurl for the repo
# cd /etc/yum.repos.d
# vim CentOS-Base.repo
-----------------------------
baseurl=..................
-----------------------------
(and all other repo you are using)
# yum clean all
# yum check-update

Step 2: If you are still getting error "Cannot open/read repomd.xml file for repository: epel", it could be due to the resolving of DNS or you have set the proxy wrongly at /etc/yum.conf
# wget the http://download.fedoraproject.org/pub/epel/4/i386/repodata/repomd.xml (or other url in the repo)
(If you encountered error information like "Name or Service not known". It is likely a DNS resolution issues)

# vim /etc/resolv.conf
(and resolve the error)

Step 3: If you are still encountering error, you may want to check /etc/yum.conf and comment out proxy
# vim /etc/yum.conf
--------------------
# proxy=xxxxxxx
--------------------
Finally do a unset to remove a http proxy  
# unset http_proxy
# echo $http_proxy (To check)

Step 4: If you are still encountering error, you may finally want to check /etc/profile.d/. Maybe there is a proxy script there.

Jobs submitted placed on the queue and not able to run on HPC

Sometimes, when out users submit job via MOAB using Torque as a resource manager, their jobs is just not able to run even though there are resources to accomdoate the run. Of course there are many ways to troubleshoot. Here is how I troubleshoot 1 of them.

Step 1: Use the command checkjob on the process ID of the job that is stuck.
# checkjob 100001
--------------------------------------
Partition List:      xxxxxxxx
Flags:                 RESTARTABLE
Attr:                   checkpoint
StartPriority:       94
rejected for CPU              - (null)
rejected for State              - (null)
NOTE: job req cannot run in partition xxxxx (available procs do not meet requirements : 0 of 32 procs found)
---------------------------------------
Here's the key hint under NOTE. There is no node suitable for this user. But why? Launch another command


# qstat -f 100001
----------------------------------
euser = xxxxx
egroup = xxxxx
queue_rank = xxxxx
queue_type = xxxxx
etime = xxxxxxxx
submit_args = -1 nodes2:ppn=16 ./run.sh
-----------------------------------
Note the key issue is that users submitted a job requesting 2 nodes with 16 cores each. This does not exist in our cluster configuration. Hence it is not able to run....

Using MOAB and Torque Commands together to analyse the problem is useful indeed.

Mozilla Firefox losing its Title Bar

I was fiddling with my keyboard when I accidentally hide my firefox title bar. Quite a frustration as I cannot minimise my browser easily.

To restore the Firefox, I just need to type F11 and I got back the Titlebar....What a relief...

Article on Red Hat Cloud Developments

A good article of Red Hat Cloud Developments. You can download some of the conference presentation slides from the Redhat Summit there as well.

Using find to locate files in Linux (Part 1)

Find is a useful command line to locate and search files easily. Some usage includes

Note 1: To find a file (For example *.mp3) and print it to your console

$ find ~ -name *.mp3 -print

Note 2: Finding files that have been modified

$ find $HOME -mtime 0

(-mtime refers to modification since the last 24 hr. The time when the file was modified is divided by 24hr with the remainder removed. )

Note 3: Finding files that have been modified with file name in leading directory removed

$ find ~ -type f -mtime 0

Note 4: To serve for files and make the match case-insensitve

$ find ~ -type f -mtime 0 -iname '*.mp3'

Note 5: Finding permission not owned by you
$ find ~ ! -user ${USER}

Removing old package for Linux Mint

To remove the orphaned dependencies using apt-get

# apt-get autoremove

To clean old versions of packages using apt-get

# apt-get autoclean

To remove all packages files from the acahe using apt-get
# apt-get clean

Installing Java Runtime Environment (JRE) from the command line on Linux Mint

If you are planning to install JRE and the web browser plugin, you will need to do the following:

# apt-get install sun-java6-jre sun-java6-plugin sun-java6-fonts

Prevent non-users from login into the system

Sometimes as administrator, you may need to some immediate repair work and need to prevent users from logging in immediately, you can easily do that by doing the following:

# touch /etc/nologin

If the file exists, only the root can gain accessnow

pbs node manipulation Revision

It has been a long time since I used torque. I though I pen down some common commands I will used

• pbsnodes -l (List node names and their state. If no state is specified, only nodes in the DOWN, OFFLINE, or UNKNOWN states are listed. Specifying a state string acts as an output filter. Valid state strings are "active", "all", "busy", "down", "free", "offline", "unknown", and "up". )
  
• pbsnodes -o (Add the OFFLINE state. This is different from being marked DOWN. OFFLINE prevents new jobs from running on the specified nodes. This gives the administrator a tool to hold a node out of service without changing anything else. The OFFLINE state will never be set or cleared automatically by pbs_server; it is purely for the manager or operator.)
  
• pbsnodes -c (Clear OFFLINE from listed nodes)
  
• pbsnodes -a (All attributes of a node or all nodes are listed. This is the default if no flag is given)
  

libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations.

We are using OpenFarics for our HPC. According to the FAQ, "With OpenFabrics (and therefore the openib BTL component), you need to set the available locked memory to a large number (or better yet, unlimited) -- the defaults with most Linux installations are usually too low for most HPC applications that utilize OpenFabrics"

libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.

For the solution, look here at http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Or you can also look at my blog entry Encountering Segmentation Fault, Bus Error or No output

Looking at Xorg High CPU Usage Issue for netbook

If you are suffering from excessive slowness from your netbook or your old and slow notebook, you may want to tweak the CPU frequency governor to performance mode instead of the general mode. How do you do that? If you are using Linux Mint, you have to install a utilities called cpufrequtils

Step 1: Install CPU Frequency Utilities
# apt-get install cpufrequtils

Step 2: Query the CPU frequency governors
# cpfreq-info

Step 3: Force the CPU to use performance mode
# cpufreq-set -g performance

If this does not resolve the issues, you really have to look at other possibilities that is impacting your Xorg. Sometimes it could be your video drivers not the CPU governor issues....

Encountering Segmentation Fault, Bus Error or No output

When you encounter some of the issues like Segmentation Fault, Bus Error or No output during your serial or parallel run, it is could be due that the user limits are insufficient

Firstly do a ulimit -a and check the actual limits. Here is a sample output
# ulimit -a
===================================
core file size                 (blocks, -c) 0


data seg size                 (kbytes, -d) unlimited
scheduling priority         (-e) 0
file size                         (blocks, -f) unlimited
pending signals              (-i) 24574
max locked memory      (kbytes, -l) 64
max memory size          (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size                       (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority           (-r) 0
stack size                     (kbytes, -s) 10240
cpu time                       (seconds, -t) unlimited
max user processes      (-u) 1024
virtual memory             (kbytes, -v) unlimited
file locks                      (-x) unlimited
====================================
To solve the issues, you have to look at important parameters like data size, stack size, max memory size, and virtual memory. The parameters should be either unlimited or as big as your total RAM. 


# vim /etc/security/limits.conf
==================
* soft memlock unlimited
* hard memlock unlimited
* soft stack 4194303
* hard stack unlimited
==================

Log-off and Login again ad type # ulimit -a You should see that parameter has been fixed

Conversely, You can also use the configuration file to restrict memory and processes. See blog entry Using /etc/security/limits.conf to restrict processes and memory

Reference:

• Turbomole FAQ

Using Top and renice

When you are running a server that have more than 1 user competing for the limited number of resources, how do you prioitise the run manually?

I use top to check who is hogging the resources and I use the command

# renice +5 -u peter
# renice + 5 -p 8765

It helps alot. But I prefer the htop to influence process directly. See Using htop to influence processes directly

Using /etc/security/limits.conf to restrict processes and memory

The configuration file at /etc/security/limits.conf help in settings limits to processes or logins oe more. This helps in preventing a rogue applications in consuming too much processes that might cause the servers to crash.

* soft nproc 400

* hard nproc 500

(If you hit 400 processes, the system will warn you. If you hit 500, the system will block you) 

Alternatively if you do not wish to set a limit to the number of processes that can be run on your servers due to the particular application, you can also

* soft nproc unlimited

* hard nproc unlimited

You do not need to reboot to effect the settings, just relogin and the users will be subjected to the /etc/security/limits.conf configuration file

User friendly command line shell - Fish

fish is a user friendly command line shell for UNIX-like operating systems such as linux.

Some of the cool features of fish. It uses:

1. Uses fresh sprinkling of colour to indicate files, directory, prompt
2. Tab Completion for your commands. After typing the first few letters of the command, you can tab and suggested commands are made available
3. You can pick out command history by typing a few letter and clicking the "up" key instead of cycling through the whole history
4. A html document is made available when yo type fish --help.

To install fish at Fedora

# yum install fish

# fish

Similarly for Linux Mint

# apt-get install fish

# fish

Learning MOAB - Using showq

showq displays information about queued jobs including active, eligible, blocked, and/or recently completed jobs. Do note that the -c, -i, and -r flags can be used  Administrator only

Note 1: Display blocked jobs only
> showq -b

Note 2: Display details about recently jobs (by MOAB Admin)
# showq -c

Note 3: Display local and full resource manager job ids. If specified with the -i option, will display job reservation time. (If using MOAB
# show -vi

Note 4: Display jobs associated with specified constraint. Valid constraint include user, group, account, class and qos
> showq -w user=john

Note 5: Display extended details about active (running jobs)
# show -r
------------------------------------------------------
S - Job State. Either "R" for Running or "S" for Starting.
PAR - Partition in which job is running.
EFFIC - CPU efficiency of job.
XFACTOR - Current expansion factor of job, where XFactor = (QueueTime + WallClockLimit) / WallClockLimit
Q - Quality Of Service specified for job.
USERNAME - User owning job.
GROUP - Primary group of job owner.
MHOST - Master Host running primary task of job.
PROC  - Number of processors being used by the job.
--------------------------------------------------------

Learning MOAB. What is workload?

When you are doing High Performance Computing or Cloud, the concept of workload is very important. Basically there are 4 kinds of workloads (Taken and summarise from MOAB Admin Guide 5.2)

1. Batch Workload - With a Batch Job, the job is submitted to a job queue, and is run somewhere on the cluster as resources becomes available
2. Interactive Workload - Requestors are interested in immediate response and are generally waiting for interactive request to be executed before going on to other activities. To manage interactive jobs, the focus is usually on setting aside resources to guarantee immediate execution or at least a minimal wait time for interactive jobs
3. Calendar Workload - Calendar workload must be executed at a particular time and possibly in a regular periodic manner. There must be guarantee of resource availability at the needed time to allow calendar jobs to run as required.
4.  Service Workload - Service workload processes externally generated transaction requests while a scheduler or resource allocation mechanism provides the distributed service with needed resources to meet target backlog or response targets to the service. Examples are web farms etc. 

Installing VirtualBox on Fedora 11

Step 1: You can either install by downloading and installing the rpm directly of use the Fedora repo. There are repo for RHEL and OpenSuSE as well

Step 1a: If you wish to download and install the rpm directly, type the following
# wget
http://download.virtualbox.org/virtualbox/3.0.8/VirtualBox-3.0.8_53138_fedora11-1.i586.rpm
# yum localinstall VirtualBox-3.0.8_53138_fedora11-1.i586.rpm

Step 1b: A better method in my opinion will be by using the repository method
Create a virtualbox.repo file at /etc/yum.repos.d/
# touch /etc/yum.repos.d/virtualbox.repo
# vim virtualbox.repo
-------------------------------
[virtualbox]
name=VirtualBox
baseurl=http://download.virtualbox.org/virtualbox/rpm/fedora/$releasever
enabled=1
gpgcheck=0
---------------------------------
# yum check-update


Step 2: Run setup file for Virtualbox
# /etc/init.d/vboxdrv setup


Step 3: Add yourself as a user in the vboxusers group at /etc/group 
# usermod -G vboxusers myserid


Step 4: USB Fix for VirtualBox
# vim /etc/group
----------------------------------------------
# create a new group and add your name to it
usb:x:502:myserid
----------------------------------------------

# vim /etc/fstab
----------------------------------------------
none /sys/bus/usb/drivers usbfs devgid=502,devmode=664 0 0
----------------------------------------------

Step 5: Run VirtualBox
# VirtualBox


Reference:

• USB Fix for VirtualBox on Fedora 8 Host

Changing Compilers from Intel to PGI for OpenMPI

Extending from Building OpenMPI with Intel Compiler (Ver 2). Assuming you are using Intel Compiler as the default compiler.

Step 1: If you wish to change to another compiler like PGI (in this example) from the Intel Compiler,
You can issue the commands
# export CC=pgcc
# export CXX=pgCC
# export F77=pgf77
# export FC=pgf90

To check that the openmpi is using the PGI compiler
# /usr/local/openmpi/bin/mpicc --showme
# /usr/local/openmpi/bin/mpiCC --showme
# /usr/local/openmpi/bin/mpif77 --showme
# /usr/local/openmpi/bin/mpif90 --showme


Step 2: If you wish to "return" to the default
# unset CC
# unset CXX
# unset F77
# unset FC


To check that the openmpi is using the PGI compiler
# /usr/local/openmpi/bin/mpicc --showme
# /usr/local/openmpi/bin/mpiCC --showme
# /usr/local/openmpi/bin/mpif77 --showme
# /usr/local/openmpi/bin/mpif90 --showme

eyeos - The Open Source Cloud Web's Desktop

According to the eysos website,

eyeos is an open source web desktop following the cloud computing concept. It is mainly written in PHP, XML, and Javascript. It acts as a platform for web applications written using the eyeos Toolkit. It includes a Desktop environment with 67 applications and system utilities. It is accessible by portable devices via its mobile front end. eyeos lets you upload your files and work with them no matter where you are. It contains applications like Word Processor, Address Book, PDF reader, and many more developed by the community

Intel Data Center Manager Software

Intel® Data Center Manager (Intel® DCM) SDK provides power and thermal monitoring and management for servers, racks and groups of servers in data centers. Management Console Vendors (ISVs) and System Integrators (SIs) can integrate Intel® DCM into their console or command-line applications and provide high value power management features to IT organizations


Intel® DCM features include:

• Built-in policy based intelligent heuristics engine that can maintain group power capping while dynamically adapting to changing server loads and minimizing performance impact of workloads
• Uses Intel® Intelligent Power Node Manager for node power and thermal management
• Designed as an SDK to integrate into existing management software products by means of a web service application programming interface (WSDL API)

Video on Overview of Intel Data Center Manager

Free ebook - High Performance Computing for Dummies

This special edition eBook "High Performance Computing for Dummies" from Sun and AMD shares details on real-world uses of HPC, explains the different types of HPC, guides you on how to choose between different suppliers, and provides benchmarks and guidelines you can use to get your system up and running.


Happy Reading!

Integrated Development Environment for C, C++, Python and GTK

Anjuta is a versatile Integrated Development Environment (IDE) for C and C++ on GNU/Linux.

Anjuta also works well with Glade fro building Gnome Apps GTK. Do note that there is also support for Java and Python.

The application wizards is a wonderful feature as it guide you along to setting up all the files for the project.

One limiting feature is the lack of quality documentation.

To install for Debian distro derivatives
# apt-get install anjuta


To install for Red-Hat Distro Derivatives
# yum install anjuta

My Thoughts on what is Cloud Computing? (Part 1)

Cloud Computing is gaining a lot of publicity. I have attended a few myself. But up to now even if you gather the experts on this, even the definition is abstract. Maybe one of the best way is to ask a question and try to answer it

So what is Cloud Computing?

• Cloud Computing can be referred to as IT as a service. In the same way, a utility company delivers electricity for the end users for their specific and unique consumption, the challenge is whether Cloud Computing can deliver cheaper, faster, superior user experiences.

What are the essential components of Cloud?

• Service Catalogue - From my limited understanding, Service catalogue is the standardised IT offering you are offering to the customers. Maybe it is just some virtualised infrastructure for development, maybe it is a development environment like Hadoop, or even maybe it is even a Windows environment inside for an application requirement....
• Automation - By automation, what I meant was the ability to auto-provisioned by users for the standardised environment they require. In other words, there is some sort like a self-service portal where the users can select their choices from. Another essential atribute of automation comes in the form of workload analysis. The ability of the Cloud Computing infrastructure to meet the demand based on the workload analysis
• Virtualisation - Virtualisation is very essential as many image format of the computing environment or the service catalogue can be kept efficiently. It is also a mean where the image can be deployed very quickly and the environment consistently kept for the particular user. Bare-metal provisioning is not impossible for the Cloud Environment, it is just less efficient and more time consuming as the nodes have to be reformatted with the OS and the image. Of course you can dual-boot or triple-boot. If your number of images are small or have very standardised user base, it is still manageable.

What are the catalysts pushing industry to Cloud Computing?

• I think the current IT infrastructure limitation will gets more and more glaring as the explosion of services and data are harder to manage in a distributed environment. On average, about 85% of a compute resource remain idle. This is definitely not the best use of resources.
• Another is the amount of budget spent on maintaining instead of adding new capabilities is another "push" factor. With a consolidated environment, cost will definitely be more managed.
• I guess the last catalyst will be is the rising perception of the users. IT services is going to be just a utility. IT is so pervasive now like electricity. If you check in a hotel and there is no Internet, it is perceived as very out-of-touch. Rising expectation will definitely put demands on IT resources and yet the need to be flexible to provision with peak and trough of computing demands. This is the impetus given to the Cloud

Installing Presto Plugin for yum to boost speed

You can install the Presto Plugin for yum to download the changes from previous download which will reduce download time

# yum install yum-presto

Making yum use the fastest mirror

To make sure yum plugin download packages from the fastest mirror, you can install the following plugins

# yum install yum-fastestmirror

Restarting X server with keyboard short-cut on Fedora 11

If you need to restart X Server quickly and do not wish to use the commandline, you can use CRTL-ALT-BACKSPACE.

If you use the keyboard short-cut and it does not restart the X11, you may want to check the settings on

1. the System > Preferences > Keyboard. 
2. Select the Layout Tab and click the "Layout Options...." Button. 
3. Finally, click the item "Key sequence to kill the X Server" and check the item
   

 

And that's it.....

Enable Automatic Login for GNOME on Fedora 11

For debian derivative linux, you may want to view Enable Automatic Login for GNOME on Linux Mint,

For Fedora 11, there is a slightly different way to configure if you need automatic login. Fedora 11 uses the xml schematic to configure.

# vim /etc/gdm/gdm.schemas

 

You can tweaked the xml schematic from the screen shot above to enable automatic login

Enable Automatic Login for GNOME on Linux Mint

If you wish to speed up your boot-time and totally not concerned with security, you may want to enable automatic login.

For Linux Mint, my favourite Linux desktop, you can use the command line to enable automatic login.

# vim /etc/gdm/gdm.conf

-----------------------------------------

[daemon]

......

AutomaticLoginEnable=true
AutomaticLogin=(your userid to login)

-----------------------------------------

Alternatively, you can use the gdmsetup to configure. See pix below.

Installing chrome on Fedora 11

Installing Chrome is not too difficult on Fedora 11. Assuming you are using yum, you can create a new repository file called "chromium.repo" and input the following contents into the file

[chromium]
name=Chromium Test Packages
baseurl=http://spot.fedorapeople.org/chromium/F11/
enabled=1
gpgcheck=0

After saving the file, you should run the following
# yum update
# yum install chromium


Alternatively, you can install by downloadin the rpm from http://spot.fedorapeople.org/chromium/
and run a
# rpm -ivh (chromium package)

Rocky Mountain Supercomputing Centers (RMSC) - Supercomputer platform as a Service

Cloud Computing is evolving so quickly. Usually HPC and Enterprise Computing are usually very seperate with distinct characeristics and usually it does not converge so easily.

Cloud Computing which is the natural evolution of virtualisation and consolidation and usually comes from the Enterprise Computing paradigm is  finally closing the gap of this convergence.

One important development can be seen from Rocky Mountain Supercomputing Centers (RMSC) may be one of the first HPC Cloud being developed with the concept of supercomputing on the demand concept or Supercomputer platform as a service (SPaaS). See Rocky Mountain Supercomputing Centers (RMSC) offering.

Very thought provoking

Video of IBM Cloud Computing Community

IBM has setup a IBM Cloud Channel via Youtube. In the Cloud Channel, you have a collection of customers testimonials and how IBM is shaping business strategy via Cloud.

Good site to know more about IBM Cloudburst offering.

Using Preload to boot startup and general performance for Fedora

Preload is a daemon that analyses what you do and fetches the binaries you're mostly likely to use to boost starup and general performance. To make a very superficial comparison, it is similar to Windows Superfetch

Note 1: To install preload on Fedora 11
# yum install preload

Note 2: To tweak the preload configuration, you can tweak the
# /etc/preload.conf
(the default value should work well for all most users)

Note 3: To view the log file of the preload daemon
# less /var/log/preload.log

Note 4: If you wish to take a look on what Preload is caching for you, you go to
# /var/lib/preload/preload.state

Note 5: To ensure that the preload daemon is startup everytime
# chkconfig --level 35 preload on

OpenMPI, CentOS Linking Error - libmpi.so.0 :cannot open shared object file: No such file or directory

I was installing OpenMPI on CentOS 5.x. After installation, I  tried to  use pirun, I encountered this error
libmpi.so.0 :cannot open shared object file: No such file or directory

The reason for the error is that the the loader is not able to find the libmpi.so because /usr/local/lib is not in its PATH

To resolve the issues, you have to do the following
# vim /etc/ld.so.conf
(Add the /usr/local/lib into the file)
# ldconfig

Gnome evine - Document Viewer

Evine is a simple document viewer that support a number of format such as PDF, Postscript, DjVu, DVI, tiff. he goal of evince is to replace the multiple document viewers that exist on the GNOME Desktop with a single simple application.

Evine comes by default when you install GNOME for most Linux Distribution. If evine is not installed by default, just do a
# apt-get install evine (for debian variants) OR
# yum install evine (RH variants)

rsync with the soft-links

Simple entry for today.
I was trying to rsync 1 server to another server. To ensure that the symlinks are copy across as symlinks, do the following:

# rsync -rvl /opt/intel server2:/opt/intel
(Make sure the "-l" is included)

Installing MPI4py on CentOS

MPI for Python (mpi4py) provides bindings of the Message Passing Interface (MPI) standard for the Python programming language, allowing any Python program to exploit multiple processors.



Step 1: Install Python and dependencies
# yum install python python-devel python-lib

Step 2: Build OpenMPI with Intel Compiler
See Blog Entry Building OpenMPI with Intel Compiler (Ver 2)

Step 3: Get and install MPI4py
# wget http://mpi4py.googlecode.com/files/mpi4py-X.X.X.tar.gz

Step 4: Build the MPI4py

# cd /usr/local

# tar -zxf mpi4py-X.X.X.tar.gz

# cd mpi4py-X.X.X

Step 5: Setup of MPI
# python setup.py build
# python setup.py install

Step 6: Testing
Issuing at the command line:
$ mpiexec -n 5 python demo/helloworld.py
(will launch a five-process run of the Python interpreter and run the test scripts)

Important Notes
For more information on the installation of MPI4py, see the official MPI4py User Guide

Scientific Software for Python - Numpy, scipy, pyMPI

NumPy is the fundamental package needed for scientific computing with Python. It contains among other things:

SciPy  is open-source software for mathematics, science, and engineering. It is also the name of a very popular conference on scientific programming with Python. The SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation. The SciPy library is built to work with NumPy arrays, and provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization.

pyMPI is a project integrating the Message Passing Interface (MPI) into the Python interpreter.

Installing xrdp on Fedora 11

What is xrdp? According to the xrdp project website, xrdp is an open source remote desktop protocol(rdp) server. Xrdp uses Xvnc or X11rdp to manage the X session and will display X window desktop to the user via Remote Desktop Protocol.

To install manually, you may want to look at my blog entry Supporting Windows Terminal Clients from LINUX with xrdp

However if you are using Fedora 11, it is a breeze,

Step 1: Install xrdp

# yum install xrdp

 Step 2: Create soft link to startup xrdp automatically in case of reboot
# ln -s ../init.d/xrdp s99xrdp
# service xrdp restart


Step 3: Configure the firewall via System > Administration > Firewall
Allow port 3389 (TCP) in as shown below 

Step 4: Use Remote Desktop Protocol to connect to the Linux Box

Installing Nvidia Drivers on Linux Mint

Nvidia Drivers requires you to exist the X-Windows environment completely before you can install the drivers properly. Instead of booting into console mode, you can do the following:

Step 1: Down the Nvidia drivers from http://www.nvidia.com/Download/index.aspx?lang=en-us

Step 2: CTRL+ALT+F1 to take you into a non-X terminal

Step 3: Stop /etc/init.d/gdm (GNOME)
# /etc/init.d/gdm stop

Step 4: Install Nvidia Drivers
# ./NVIDIA-Linux-x86-185.18.36-pkg1.run
(Install Nvidia drivers that match your graphics card and follow the installation wizard)

Step 5: Start /etc/init.d/gdm
# /etc/init.d/gdm start

Speeding up boot time by Optimising Physical Memory and Swap

If you have a lot of physical memory, you may want to reduce your swappiness to increase performance. The performance is boosted as the kernel can cache data in memory for faster access and reduce the amount of data being swapped in and out of the swap space.

Alternatively, if you have very little memory (For example: 256MB or below), you may want to increase swappiness to boost performance.

To change the swappiness, You have to edit the /etc/sysctl.conf. /etc/sysctl.conf is a file tocontain sysctl which configure kernel parameters at runtime.

# vim /etc/sysctl.conf

At the bottom of the file /etc/sysctl.conf, add the following parameters
---------------------------------
vm.swappiness=5
---------------------------

Reboot and you will notice the difference. You may want to use conky to monitor the memory/swap usages

Speeding up boot time by running boot processes in parallel for Linux Mint

If you are using Linux Mint and have multi-core computer at home and wishes to speed up boot time by running processes in parallel, you can tweaked Grub to take advantage of parallelism.

Step 1: Edit /etc/init.d/rc and change CONCURRENCY from none to shell
# vim /etc/init.d/rc
---------------------------
CONCURRENCY=shell
---------------------------

You should notice a performance improvement in your boot-time.

The netbook distro - Moblin 2.0 beta

Moblin is an open source project focused on building a Linux-based platform optimized for the next generation of mobile devices including Netbooks, Mobile Internet Devices, and In-vehicle infotainment systems.

Moblin is Intel's initiative to develop a linux-based OS for mobile devices like netbook. According to the Moblin website, Moblin v2.0 boast the following feastures:

• New, visually rich user experience
• The myzone, acting as the 'home screen' panel
• Aggregation of your social networking content
• A web browser optimized for the Moblin 2.0 netbook user interface
• A 'Zoomable' media player
• A user interface for connection management

Currently, it is v2.0 beta and according to the website tested moblin on the following netbook

• Acer Aspire*One,
• Asus eeePC* 901, 1000H,
• Dell Mini 9, MSI Wind,
• Lenovo S10,
• Samsung NC10,
• HP Mini 1010 and 1120NR (wired networking only for now).

For more information, do go to Moblin Website

Remove the timeout during boot time

When you boot, there is a small count down before continuing with the boot process. This was originally introduced to ensure that older hardware load modules in time for the kernel to boot. Nowsaday, it is used for users to select their kernel modules, to dual boot etc.

However, if you have a modern machine and you wish to boot fast, you can edit
# /boot/grub/menu.lst

----------------------------------
timeout 0                                    
----------------------------------
(Change timeout from 5 to 0)

You just save 5 seconds from your boot time

Using dhcpcd to fix dhcp client settings

dhcpcd is an useful tool to help you fix dhcp client settings without the need to use the X windows network configuration tools.

To install dhcpcd, just install
# apt-get install dhcpcd

Some application of dhcpcd.
Note 1: Don't let the DHCP program modified your /etc/resolv.conf
# dhcpcd -R eth1

Note 2: Getting your preferred IP Address from the dhcp server. This is especially useful in a small network like home network
# dhcpcd -k (release the dhcp release)
# dhcpcd -s xxx.xxx.xxx.xxx eth1

Note 3: Preventing DHCP from installing default route
# dhcpcd -k (release the dhcp release)
# dhcpcd -G xxx.xxx.xxx.xxx eth1

For more usages, you can visit the online man page for dhcpcd

Installing Adobe Air on 32-bit Linux Mint

To install Adobe AIR on 32-bit Linux Mint, you can do the following:

Step 1: Use dpkg to enumerate adobe application
# dpkg -l | grep adobe


Step 2: Install Dependency for Adobe Flash
# apt-get install libnspr4-dev libnss3-dev libcurl3


Step 3: Download the Adobe Flash from http://www.adobe.com/
# dpkg -i install_flash_player_10_linux.deb


Step 4: Download the Adobe AIR from http://www.adobe.com/
# chmod 755 AdobeAIRInstaller.bin
# ./AdobeAIRInstaller.bin