A relook at libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations.

There was an prior blog entry written in Oct 2009.libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations.

I would like to add on to this entry. In the FAQ 17 from OpenMPI,  17. I'm still getting errors about "error registering openib memory"; what do I do?, the FAQ mentioned about the scheduler

 Make sure that the resource manager daemons are started with unlimited memlock limits (which may involve editing the resource manager daemon startup script, or some other system-wide location that allows the resource manager daemon to get an unlimited limit of locked memory). Otherwise, jobs that are started under that resource manager will get the default locked memory limits, which are far too small for Open MPI.

The files in limits.d (or the limits.conf file) does not usually apply to resource daemons! The limits.s files usually only applies to rsh or ssh-based logins. Hence, daemons usually inherit the system default of maximum 32k of locked memory (which then gets passed down to the MPI processes that they start). To increase this limit, you typically need to modify daemons' startup scripts to increase the limit before they drop root privliedges.

Some resource managers can limit the amount of locked memory that is made available to jobs. For example, SLURM has some fine-grained controls that allow locked memory for only SLURM jobs (i.e., the system's default is low memory lock limits, but SLURM jobs can get high memory lock limits). See these FAQ items on the SLURM web site for more details: propagating limits and using PAM.

Other related Issues

1. For Torque, you may want to tweak the /etc/init.d/pbs_mom See blog entry Default ulimit setting in torque overide ulimit setting

# service pbs_mom restart

2. See also Encountering Segmentation Fault, Bus Error or No output . In that blog, you have to edit /etc/security/limit.conf

* soft memlock unlimited
* hard memlock unlimited

3. If you still have memory issues and using Mellanox IB Cards, do take a look at Registering sufficent memory for OpenIB when using Mellanox HCA

Understanding the Parameter for ulimit in a glance on CentOS

If you need to look and understand the parameters of the ulimit. Just use the command

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 311296
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 311296
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Look at the output and you will see that the parameter are within the ()

For example, if you wish to tweak the max locked memory to be unlimited, you can use -l

# ulimit -l unlimited 

Or

For example if you wish to  tweak the core file size to be unlimited, you can use -c

# ulimit -c unlimited

Memory Consumption in Linux

This web article from http://emilics.com talks about memory consumption by a process in Linux. As Linux has a sophisticated  memory management system, measuring may not be that simple.

Inside the article, it mention VSZ (Virtual Memory Size) and Demand Paging, RSS (Resident Set Size) and Shared Libraries, PSS (Proportional Set Size) and more.

Do read the article from Memory Consumption in Linux

Clearing memory cache

First thing you need is kernel 2.6.16
The command to clear the memory cache.

# sync
# echo 3 > /proc/sys/vm/drop_caches

sync -> refers to the tells the kernel that you want the data written to the disk
echo 3 > /proc/sys/vm/drop_caches -> "To free pagecache, dentries and inodes:"

From the article Invalidating the Linux buffer cache ,

To free pagecache:    
# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:
# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:
# echo 3 > /proc/sys/vm/drop_caches

References:

1. Invalidating the Linux buffer cache 
2. HOWTO: Clear filesystem memory cache
3. drop_caches

Using free to see memory and cache usage

The utility free is a useful tool for Linux to help to analyse the memory usage

To get the output below, I issued a command

# free -mt

where
-m => Display amount in  megabytes
-t => Add a Total row at the bottom
-s N => Run continuously and update the display  every N seconds

             total       used       free     shared    buffers     cached
Mem:          7871       5995       1875          0        310       3729
-/+ buffers/cache:       1956       5914
Swap:        10047        283       9764
Total:       17919       6279      11640

How do we interpret the results. I would like to credit the article from Determining free memory on Linux for a simple but very good explanation on memory.Here is what I have gathered from the read-up.

Basically, Linux caches blocks from the disk in memory to make data reading as efficient as possible. The buffer cache will shrink to accommodate the increased memory needs.

The actual free memory is column free (1875) + Buffers (310) + Cached (3729). So it is more than we think

Tweaking the Linux Kernel to manage memory and swap usage

This writeup is assuming you are tweaking to the minimise swap and maximise physical memory. This tweaking should be considered especially for High Performance MPI applications where good low latency parallelism between nodes is very essential.

In addition, this writeup also help you to “kill” runaway memory applications. For more information, see Tweaking the Linux Kernel to manage memory and swap (Linux Cluster)

Green Memory and SSDs

Interesting Information on Green Memory and SSDs from Samsung.

From Samsung

"In a server, distribution of power is directly related with its memory density and as such the need to reduce memory power consumption is becoming increasingly critical. When memory density increases among different servers, its contribution to total system power increases in other words, more memory requires a server to allocate more power for memory usage....."

1. 30nm class Green DDR3
2. Green SSD

 

RAMSpeed cache and memory benchmarking tool

RAMspeed is a free open source command line utility to measure cache and memory performance of computer systems. For more information on the algorithm and other information, do read the RAMSpeed site.

RAMspeed is more accurate than many other benchmarking tools, more customisable, open source, compact, and gives you much more information to analyse. Some people may say that the lack of some graphical interface is a large drawback, but it may be considered as an advantage as well.


Now for the download information.....taken from RAMSpeed site.


RAMspeed (UNIX) v2.6.0 (August, 2009) — for uniprocessor machines running UNIX-like operating systems. The source code is available for download (76Kb).

RAMspeed/SMP (UNIX) v3.5.0 (August, 2009) — for multiprocessor machines running UNIX-like operating systems and supporting System V IPC extensions. The source code is available for download (78Kb).

RAMspeed (DOS) v2.5.0 (August, 2009) — for DOS as well as 32-bit Windows operating systems (95 to 2003; i386 only). Both the source code and a pre-compiled executable are available for download (109Kb).

RAMspeed (Win32) v1.1.1 (August, 2009) — for 32-bit as well as 64-bit Windows operating systems (95 to Vista; i386 or amd64). Both the source code and a pre-compiled executable are available for download (71Kb). 

Memory Management - Preventing the kernel from dishing out more memory than required

I think for us who have been running computational jobs have seen the memory got eaten up by some buggy or stray applications. Hopefully the kernel kills it. But somehow you must have seen that the kernel may not have kill the culprit and the server go to a linbo.

Let's say if we wish to ensure that the kernel only gives out memory to processes equal to the physical memory, then we have to do the following at /etc/sysctl.conf or /etc/sysctl.d/myapp.conf

My assumption is that you have 10GB of swap and 20GB of memory and you wish the kernel to stop handling processes at  18GB RAM, then the calculation should be (swap size +  0.4 * RAM size)

So at /etc/sysctlf.conf, the configuration will be

vm.overcommit_memory = 2
vm.overcommit_ratio = 40

Note: The ratio is (40/100). For explanation of vm.overcommit_memory =2. Do look at Tweaking Linux Kernel Overcommit Behaviour for memory

Once the memory hits 18GB, the so-called OOM killer of the Linux kernel will kick in.

Another calculation example is that your RAM size and  SWAP size are the same and you wish exactly the physical memory to be used only. then

vm.overcommit_memory = 2
vm.overcommit_ratio = 0

For more information, do read

1. When Linux runs out of memory 
2. vm.overcommit_memory = 2, vm.overcommit_ratio = 0

Tweaking Linux Kernel Overcommit Behaviour for memory

You can change the behaviour of the Linux Kernel in regardings to its Overcommit Behaviour for memory. You can change the behaviour at /etc/sysctl.conf/ or /etc/sysctl.d/myconfig.conf

You will write something like

vm.overcommit_memory = (a integer from 0 to 2)

where
0 - means that the kernel will use predefined heuristics when deciding whether to allow such an overcommit. This is the default.
1 -always overcommits (Very dangerous)
2 - Prevents overcommits from exceeding a certain value. Within this mode, the total commit can not exceed the swap space(s) size + overcommit_ratio percent * RAM size. By default, the overcommit ratio is 50.

For example, do look at Memory Management - Preventing the kernel from dishing out more memory than required

Preventing Swapping unless absolutely necessary

If you have lots of RAM, you may want to use RAM as I/O caches and buffers. The benefits of using RAM as I/O caches and buffers are definitely speed when compared to swapping data.

To review the value of swappiness can be seen by running the following commands

# cat /proc/sys/vm/swappiness

To modified by running the following command (0 will prevent swapping unless absolutely required)

# echo 0 > /proc/sys/vm/swappiness

To make the settings permanent, edit /etc/sysctl.conf

vm.swappiness=0

Speeding up boot time by Optimising Physical Memory and Swap

If you have a lot of physical memory, you may want to reduce your swappiness to increase performance. The performance is boosted as the kernel can cache data in memory for faster access and reduce the amount of data being swapped in and out of the swap space.

Alternatively, if you have very little memory (For example: 256MB or below), you may want to increase swappiness to boost performance.

To change the swappiness, You have to edit the /etc/sysctl.conf. /etc/sysctl.conf is a file tocontain sysctl which configure kernel parameters at runtime.

# vim /etc/sysctl.conf

At the bottom of the file /etc/sysctl.conf, add the following parameters
---------------------------------
vm.swappiness=5
---------------------------

Reboot and you will notice the difference. You may want to use conky to monitor the memory/swap usages