do_vfs_lock: VFS is out of sync with lock manager for CentOS 5

If you are reading at the "do_vfs_lock: VFS is out of sync with lock manager" messages at your screen or in your log file,

According to RedHat Site,

The message will be printed whenever there is locking contention (two or more processes trying to lock the same file) and the mount had nolock specified.

The RHEL-5 code prints  the message unconditionally, while on the upstream code it is a debugging message, so it won't be seen on normal operation  there.

Do take a look at your /etc/fstab and the mounting option. You should remove the "nolock" options

References:

1. Many "do_vfs_lock: VFS is out of sync with lock manager" messages on a "-o nolock" NFS mount in RHEL?

NFS mount errors with "clnt_create: RPC: Unknown host" for CentOS 6

When attempting to mount CentOS 6, my mount fails with

clnt_create: RPC: Unknown host

Diagnostic:
If we do a more thorough diagnostic, this is the issue

# showmount -e  
clnt_create: RPC: Unknown host  
# showmount -e localhost  
Export list for localhost:  
/export/my_data \* 

Resolution:

Taken from Redhat Site

Implement forward and reverse lookups (A records and CNAME records) in DNS and have the system point towards the DNS servers. Implement for both IPv4 and IPv6. If unable to resolve DNS issues, change the /etc/hosts file from this:

Change from 

::1          localhost localhost.localdomain localhost6 localhost6.localdomain6

To

::1          machine_hostname localhost localhost.localdomain localhost6 localhost6.localdomain6

Restart the NFS service and check on the showmount -e localhost and showmount -e and attempt to mount the share.

# service nfs restart  
# showmount -e localhost  
# showmount -e 

NFS Share getting a (1) appended when adding NFS storage at VCentre

I seems to get be getting this (1) appended to the NFS Share when I enter the NFS storage and volume and Datastore Name at the VCentre "Add Storage" dialogue box. I was using VSphere 5.1 and VCentre 5.1

There are 2 scenarios

Scenario 1 - Case Sensitive for the ServerName
Do look at NFS share getting a (1) appended when adding to a new host in existing datacenter . The solution is due to different caps for the storage name


Scenario 2- Missing "/" for the Storage Folder

For example ( I have this in my existing host):

Server: 192.168.1.1
Folder: vol/vol1
DataStore Name: MyDataStore 

Do note that the volume has already mapped in other hosts. I realized that the correction error was very simple. I missed the "/" in front on the vol/vol1

Server: 192.168.1.1
Folder: /vol/vol1
DataStore Name: MyDataStore 

Installing NFS4 on CentOS 5 and 6

Taken from Installing NFS4 on CentOS 5 and 6 (my alternative Linux Cluster Blog). This tutorial is a guide on how to install NFSv4 on CentOS 5 and 6.

Step1: Installing the packages

# yum install nfs-utils nfs4-acl-tools portmap

Some facts about the tools above as given from yum info.
nfs-utils -  The nfs-utils package provides a daemon for the kernel NFS server and related tools, which provides a much higher level of performance than the traditional Linux NFS server used by most users.
This package also contains the showmount program.  Showmount queries the mount daemon on a remote host for information about the NFS (Network File System) server on the remote host. For example, showmount can display the clients which are mounted on that host. This package also contains the mount.nfs and umount.nfs program.
nfs4-acl-tools – This package contains commandline and GUI ACL utilities for the Linux NFSv4 client.
portmap - The portmapper program is a security tool which prevents theft of NIS (YP), NFS and other sensitive information via the portmapper. A portmapper manages RPC connections, which are used by protocols like NFS and NIS.
The portmap package should be installed on any machine which acts as a server for protocols using RPC.


Step 2: Exports the File System from the NFS Server (Similar to NFSv3 except with the inclusion of fsid=0)

/home           192.168.1.0/24(rw,no_root_squash,sync,no_subtree_check,fsid=0)
/install        192.168.1.0/24(rw,no_root_squash,sync,no_subtree_check,fsid=1)

The fsid=0 and fsid=1 option provides a number to use in identifying the filesystem. This number must be different for all the filesystems in /etc/exports that use the fsid option. This option is only necessary for exporting filesystems that reside on a block device with a minor number above 255.one directory can be exported with each fsid option.

Exports the file system

# exportfs -av

Restart the NFS service

# service nfs start

If you are supporting NFSv3,  you have to start portmap as NFSv3 requires them. As such, NFSv4 does not need to interact with rpcbind[1], rpc.lockd, and rpc.statd daemons. For more information see Fedora Chapter 9.  Network File System (NFS) – How it works for a more in-depth understanding.

# service portmap restart

Step 2: Client Mapping

# mount -t nfs4 192.168.1.1:/ /home

NFS4 Information from the University of Michigan

NFSv4 information can be found from the University of Michigan Centre for Information Technology Integration Project NFS Version 4 Open Source Refrence Implementation

I like the rfc3530 definition of NFS v4 definition as written in the site

The Network File System (NFS) version 4 is a distributed filesystem protocol which owes heritage to NFS protocol version 2, RFC 1094, and version 3, RFC 1813. Unlike earlier versions, the NFS version 4 protocol supports traditional file access while integrating support for file locking and the mount protocol. In addition, support for strong security (and its negotiation), compound operations, client caching, and internationalization have been added. Of course, attention has been applied to making NFS version 4 operate well in an Internet environment. 

 Interesting  and relevant information

1. NFSv4 wiki
   (Includes information on 4.1, pNFS prototype)
2. Connectathon Test Suite
3. General troubleshooting recommendations
4. Performance and Stress tests for NFS

NFS4 Client unable to mount Server NFS4 file

When I was mounting NFSv4 on a CentOS 5 client with a CentOS 6 Server. I receive the error......

# mount -t 192.168.1.1:/tmp /home
mount.nfs4: 192.168.1.1:/tmp failed, reason given by server: 
No such file or directory.

My NSFv4 Server exports

/tmp   192.168.1.0/255.255.255.0(rw,no_root_squash,sync,no_subtree_check,fsid=0)

On the NFSv4 Server, I recall I export file system and restart the nfs server

# exportfs -av
# service nfs start

 I was having the issue, because I fail to understand the characteristics of the NFSv4. In NFSv4, it uses the virtual file system to present the server’s export and associated root filehandles to the client. The keep idea is to look what it mean fsid=0 on the NFS Server. For more information, do look at
A brief look at the difference between NFSv3 and NFSv4

The solution is to

# mount -t nfs4 192.168.1.1:/ /home

and the solution is there.

A brief look at the difference between NFSv3 and NFSv4

There are a few interesting differences between NFSv3 and NFSv4. Comparison of  NFSv3 and NFSv4 is quite hard to obtain and the information is referenced from NFS Version 4 Open Source Project.
From a File System perspective, there are
Export Management

1. In NFSv3, client must rely on auxiliary protocol, the mount protocol to request a list of server’s exports and obtain root filehandle of a given export. It is fed into the NFS protocol proper once the root filehandle is obtained.
2. In NFSv4 uses the virtual file system to present the server’s export and associated root filehandles to the client.
3. NFSv4 defines a special operation to retrieve the Root filehandle and the NFS Server presents the appearance to the client that each export is just a directory in the pseudofs
4. NFSv4 Pseudo File System is supposed to provide maximum flexibility. Exports Pathname on servers can be changed transparently to clients.

State

1. NFSv3 is stateless. In other words if the server reboots, the clients can pick up where it left off. No state has been lost.
2. NFSv3 is typically used with NLM, an auxiliary protocol for file locking. NLM is stateful that the server LOCKD keeps track of locks.
3. In NFSv4, locking operations are part of the protocol
4. NFSv4 servers keep track of open files and delegations

Blocking Locks

1. NFSv3 rely on NLM. Basically, Client process is put to “sleep”. When a callback is received from the server, client process is granted the lock.
2. For NFSv4, the client to put to sleep, but will poll the server periodically for the lock.
3. The benefits of the mechanism is that there is one-way reachability from client to server. But it may be less efficient.

Tuning rsize and wsize on NFS for a 10GbE network

Taken from Myricom Site "Do you have recommendations for tuning NFS on a 10GbE network"

1. Use Recent Linux Kernel 2.6.19 or later. CentOS 6 will be a good candidate to implement.
2. On /etc/fstab, you can set rsize=1048576,wsize=1048576
3. You can use the above buffers on NFSv3
4. Do note that for Linux Kernel 2.18 and below, the rsize and wsize is 32KB.

Network File System (NFS) in High Performance Network

This article "High Performance (NFS) in High Performance" by Carnegic Mellon is very interesting article about NFS Performance. Do take a look. Here is a summary of their fundings

1. For point-to-point throughput, IP over InfiniBand (Connected Mode) is comparable to a native InfiniBand.
2. When a disk is a bottleneck, NFS can benefit from neither IPoIB nor RMDA
3. When a disk is not a bottleneck, NFS benefits significantly from both IPoIB and RDMA. RDMA is better than IPoIB by ~20%
4. As the number of concurrent read operations increases, aggregate throughputs achieved for both IPoIB and RDMA significantly improve with no disadvantage for IPoIB

High NFS Load causing echo 0 > /proc/sys/kernel/hung_task_timeout_secs

Do note that simultaneous numerous write by the NFS Clients on the NFS Server will cause tremendous performance penalty and system lock-out as describe below. You will notice if you use "top" utilities, the load can be extremely high as numerous system locks are queued. 

One of my researcher was running a intense load on the NFS Server that cause an  eventual  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs". Before that, I saw on the log file "rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket"

To solve the problem, you have to lighten the load of the NFS or improve the setting. You may want to take a look at the Configuring NFS Server for Performance. A longer term solution will be to move to parallel file system.

Sometimes, it could be caused by other factors like drivers. You may want to take a look at Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down

Clock skew detected. Your build may be incomplete

Sometimes when you compiling a program in your /home directoryand when you run a make you receive this error

make: :  Clock skew detected.  Your build may be incomplete

This occurs when the Server you are logging on and the NFS Server has discrepancy in their time. The solution is to synchronize their time, best with a time server. For more information, see Configuring NTP Server and Client on CentOS 5.x on how to set up a NTP Server and NTP Client

Articles on Parallel NFS

A good series of articles for reading on pNFS which hold promises for those who like the user-friendly NFS protocols and need the Intensive I/O requirements bveyond what the traditional NFS can offer

1. Scale your file system with Parallel NFS (IBM DeveloperWorks)
2. pNFS.com (Information on Parallel NFS)
3. NFSv4 delivers seamless network access (IBM DeveloperWorks)

NFS share on Linux client not immediately visible to other NFS clients

If you are using NFS as the shared file system, you may encounter this issue where NFS share on Linux client not immediately visible to other NFS clients. This is due to caching parameters which you must take note of on the NFS Client side. These are

1. acregmin=n. The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 3 seconds.
2. acregmax=n. The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 60.
3. acdirmin=n. The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
4. acdirmax=n. The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
5. actimeo=n. When you wish to sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.

For more information, see

1. Configuring NFS Client for Performance
2. Why are changes made on an NFS share on my Red Hat Enterprise Linux 5 client not immediately visible to other NFS clients?

Testing for Saturated Network for NFS

How do we test for saturated network for NFS. Is there a systematic way of doing it? You can read my Linux Cluster blog entry Testing for Saturated Network for NFS

Network design consideration for NFS

Network Design are an important consideration for NFS. Often we miss the infrastructure design consideration of NFS. Read further at Network design consideration for NFS

NFS and "Permission Denied" Mount Problem at the Client

I've encountered a strange issue with the NFS mount today.

1. On the LINUX Client
# mount /nfs-server
mount: xxx.xxx.xxx.xxx:/nfs-server failed,
reason given by server: Permission denied

We have checked the permission on the nfs-server:/etc/exports and everything is correct. We did a:

# showmount -e nfs-server
Everything was correct too and seem correct. If we do a "rpcinfo -p", it also show correct information
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 615 status
100024 1 tcp 618 status
100021 1 udp 32771 nlockmgr
100021 3 udp 32771 nlockmgr
100021 4 udp 32771 nlockmgr
100021 1 tcp 37569 nlockmgr
100021 3 tcp 37569 nlockmgr
100021 4 tcp 37569 nlockmgr

On the Server side,
We check the /etc/exports and everything is correct. Next we restarted the NFS and the Portmap Services. But the Clients still show the "Permission Denied" Error

We noticed that the /proc/mounts/ did not have the NFS Information. To resolve this issues, we have to manually add NFS information inside the /proc/mounts
nfs-server:/home /home nfs rw,vers=3,rsize=8192,wsize=8192,hard,intr,proto=tcp,timeo=14,retrans=2,sec=sys,addr=n00 0 0

We did the NFS Mount and everything was working!


Related Articles:

• Linux NFS and "Permission denied" mount problem

NFS Performance Tuning

According to an article by NFS Performance Tuning by Ben Martin, adding a async parameter in the NFS mount in fstab. Tested on Fedora 9

fileserver:/home /home nfs async,rsize=8192,wsize=8192,intr 0 0

According to the article, using async will cause the speed of the NFS mount to be close to actual local disk speed. Furthermore, you can increase the buffer size of rsize and wsize from the default 4096 to 8192

intr - allow signal to interrupt file operation especialy if it is hard-mounted
hard - by default