NFS

NFS - "Network File System" - developed by Sun MicroSystem is the most widely used HPC to share filesystem. It is very easy to set up and performs reasonably well for small to medium clusters as primary storage. You can even use it for larger clusters too.

along with SMB, NFS serves essentially the same purpose as SMB to access files systems over a network as if they were local.

One of the most common questions about NFS configuration is how to tune it for performance and management and what options are typically used. So it is important to know them NFS export and mount options especially when you are facing a performance issue or a functional issue with the NFS mount over network.

NFS version

version	Description
Version 2	RFC 1094, March 1989 only allowed access to the first 2 GB of a file. The client made the decision about whether the user could access the file or not. This version supported UDP only, and it limited write operations to 8 kbytes. Ugh. It was a nice start, though.
Version 3	RFC 1813, June 1995 supported much larger files. The client could ask the server to make the access decision — although it didn't have to and so we still had to fully trust all client hosts. NFSv3 over UDP could use up to 56 kbyte transactions. There was now support for NFS over TCP, although client implementations varied quite a bit as to details with associated compatibility issues
Version 4	RFC 3010, December 2000; revised in RFC 3530, April 2003 and again in RFC 7530, March 2015 added statefulness, so an NFSv4 client could directly inform the server about its locking, reading, writing the file, and so on. All the NFS application protocols — mount, stat, ACL checking, and the NFS file service itself, run over a single network protocol always running on a predictable TCP port 2049. There is no longer any need for an RPC port mapper to start first, or for separate mount and file locking daemons to run independently. Access control is now done by user name, not possibly inconsistent UID and GID as before, and security could be further enhanced with the use of Kerberos.
Version 4.1	RFC 5661, January 2010; revised in RFC 8881, August 2020 added Parallel NFS (or pNFS), which can stripe data across multiple NFS servers. You can do block-level access and use NFS v4.1 much like Fibre Channel and iSCSI, and object access is meant to be analogous to AWS S3. NFS v4.1 added some performance enhancements Perhaps more importantly to many current users, NFS v4.1 on Linux better inter-operates with non-Linux NFS servers and clients. Specifically, mounting NFS from Windows servers using Kerberos V5.
Version 4.2	RFC 7862

Basic command

Commands	Description	Command on
# exportfs -r	Re-export your shares	Server
# exportfs -a	Export your shares	Server
# exportfs -v	Verify the NFS Share permissions	Server
$nfsstat -m	Verify Current NFS Mount Options	Client

NFS export on Server

NFS exports options are the permissions that is applied on NFS Server when we create a NFS Share under /etc/exports

Here are most common(important) options that administrator must understand, full list of options are available on the man pages

Export Options	NFS Server	Default
secure/insecure	NFSv4 only uses port 2049 while to check the list of ports used by NFSv3 use in server port With secure the port number from which the client requests a mount must be lower than 1024. To allow client any available free port use insecure in the NFS share	secure
rw/ro	ro means read-only access to the NFS Share rw means read write access to the NFS Share	rw
root_squash/no_root_squash	squash literally means to squash (destroy) the power of the remote root user. root_squash prevents remote root users from having superuser (root) privileges on remote NFS-mounted volumes. no_root_squash allows root user on the NFS client host to access the NFS-mounted directory with the same rights and privileges that the superuser would normally have	root_squash
all_quash/no_all_squash	all_squash will map all User IDs (UIDs) and group IDs (GIDs) to the anonymous user. all_squash is useful for NFS-exported public FTP directories, news spool directories	no_all_squash
sync/aysnc	sync reply to requests are done only after the changes have been committed to stable storage async allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage Using aysnc option usually improves performance, but at the cost that an unclean server restart (i.e. a crash) can cause data to be lost or corrupted	sync

Check exports list and options

#with following set
$ cat /etc/exports 
/nas * (rw,sync,no_subtree_check)
/nas 192.168.1.0/24 (rw,sync,no_subtree_check,no_root_squash,insecure)

$ rpcinfo -p | grep -i nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100003    3   udp   2049  nfs
    
# detailed export status with default export options
$ sudo exportfs -v
/nas   <world>(ro,wdelay,root_squash,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)

NFS exports examples

/etc/exports file

/srv/nfs4         192.168.33.0/24(rw,sync,no_subtree_check,crossmnt,fsid=0)
/srv/nfs4/backups 192.168.33.0/24(ro,sync,no_subtree_check) 192.168.33.3(rw,sync,no_subtree_check)
/srv/nfs4/www     192.168.33.20(rw,sync,no_subtree_check)

Ubuntu NFS service dependency

Service	nfs-utils.service	nfs-server.service	config file (22.04)	config file (< 22.04) /etc/default/nfs-*
nfs-blkmap	PartOf		nfs.conf
nfs-mountd		BindsTo	nfs.conf	nfs-kernel-server
nfsdcld
nfs-idmapd		BindsTo	nfs.conf, idmapd.conf	idmapd.conf
rpc-gssd	PartOf		nfs.conf
rpc-statd	PartOf		nfs.conf	nfs-common
rpc-svcgssd	PartOf	BindsTo	nfs.conf	nfs-kernel-server

NFS Client mount options

# mount -t nfs -o [options] remote:/nfs /mount

Mount Options	NFS Client	Default	Notes
nfsvers=n or vers=n	The version of the NFS protocol to use. By default, the local NFS client will attempt to mount the file system using NFS version 3. If the NFS server does not support version 3, the file system will be mounted using version 2. If you know that the NFS server does not support version 3, specify vers=2, and you will save time during the mount, because the client will not attempt to use version 3 before using version 2	3 and 4	NFS tips and tricks#Check NFS server supported version
rw (read/write) / ro (read-only)	Use rw for data that users need to modify. In order for you to mount a directory read/write, the NFS server must export it read/write. Use ro for data you do not want users to change. A directory that is automounted from several servers should be read-only, to keep versions identical on all servers.	rw.
suid / nosuid	Specify suid if you want to allow mounted programs that have setuid permission to run with the permissions of their owners, regardless of who starts them. If a program with setuid permission is owned by root, it will run with root permissions, regardless of who starts it. Specify nosuid to protect your system against setuid programs that may run as root and damage your system.	suid
hard / soft	Specify hard if users will be writing to the mounted directory or running programs located in it. When NFS tries to access a hard-mounted directory, it keeps trying until it succeeds or someone interrupts its attempts. If the server goes down, any processes using the mounted directory hang until the server comes back up and then continue processing without errors. Interruptible hard mounts may be interrupted with CTRL-C or kill (see the intr option, later). Specify soft if the server is unreliable and you want to prevent systems from hanging when the server is down. When NFS tries to access a soft-mounted directory, it gives up and returns an error message after trying retrans times. Any processes using the mounted directory will return errors if the server goes down.	hard
nconnect=<value>	NFS-over-TCP mount for one or more NFS shares from an individual NFS server, the traditional behavior is that all those mounts share one TCP connection if they are using the same NFS protocol version.  In cases of high NFS work load at the client, this connection sharing may result in lower performance or unnecessary bottlenecks.	16	In Linux kernel 5.3 (and higher), the nconnect option allows multiple TCP connections for a single NFS mount. example, mount -t nfs -o ro,nconnect=16 198.18.0.100:/datasets /mnt/datasets
intr / nointr	Specify intr if users are not likely to damage critical data by manually interrupting an NFS request. If a hard mount is interruptible, a user may press [CTRL]-C or issue the kill command to interrupt an NFS mount that is hanging indefinitely because a server is down. Specify nointr if users might damage critical data by manually interrupting an NFS request, and you would rather have the system hang while the server is down than risk losing data between the client and the server.	intr	In Linux kernel 2.6.25 (and higher), the intr and nointr mount options are deprecated. If you use the hard option on modern Linux kernels, you must use the kill -9 (SIGKILL) command to interrupt a stuck NFS mount.
fg (foreground) / bg (background)	Specify fg for directories that are necessary for the client machine to boot or operate correctly. If a foreground mount fails, it is retried again in the foreground until it succeeds or is interrupted. All automounted directories are mounted in the foreground; you cannot specify the bg option with automounted directories. Specify bg for mounting directories that are not necessary for the client to boot or operate correctly. Background mounts that fail are re-tried in the background, allowing the mount process to consider the mount complete and go on to the next one. If you have two machines configured to mount directories from each other, configure the mounts on one of the machines as background mounts. That way, if both systems try to boot at once, they will not become deadlocked, each waiting to mount directories from the other. The bg option cannot be used with automounted directories.	fg
devs / nodevs	Specify devs if you are mounting device files from a server whose device files will work correctly on the client. The devs option allows you to use NFS-mounted device files to read and write to devices from the NFS client. It is useful for maintaining a standard, centralized set of device files, if all your systems are configured similarly. Specify nodevs if device files mounted from a server will not work correctly for reading and writing to devices on the NFS client. The nodevs option generates an error if a process on the NFS client tries to read or write to an NFS-mounted device file.	devs
timeo=n	The timeo (timeout) option is the amount of time the NFS client waits on the NFS server before retransmitting a packet (no ACK received). The value for timeo is given in tenths of a second, so if timeo is 5, the NFS client will wait 0.5 seconds before retransmitting The timeout, in tenths of a second, for NFS requests (read and write requests to mounted directories). If an NFS request times out, this timeout value is doubled, and the request is retransmitted. After the NFS request has been retransmitted the number of times specified by the retrans option, a soft mount returns an error, and a hard mount retries the request. The maximum timeo value is 30 (3 seconds). Try doubling the timeo value if you see several servers not responding messages within a few minutes. This can happen because you are mounting directories across a gateway, because your server is slow, or because your network is busy with heavy traffic.	The default is 0.7 (0.07 seconds)
retrans=n	retrans, specifies the number of tries the NFS client will make to retransmit the packet. If the value is 5, the client resends the RPC packet five times, waiting timeo seconds between tries. If, after the last attempt, the NFS server does not respond, you get a message Server not responding. The number of times an NFS request (a read or write request to a mounted directory) is retransmitted after it times out. If the request does not succeed after n retransmissions, a soft mount returns an error, and a hard mount retries the request. Increase the retrans value for a directory that is soft-mounted from a server that has frequent, short periods of downtime. This gives the server sufficient time to recover, so the soft mount does not return an error	4
retry=n	The number of times the NFS client attempts to mount a directory after the first attempt fails. If you specify intr, you can interrupt the mount before n retries. However, if you specify nointr, you must wait until n retries have been made, until the mount succeeds, or until you reboot the system. If mounts are failing because your server is very busy, increasing the retry value may fix the problem	1
rsize=n	The number of bytes the NFS client requests from the NFS server in a single read request. If packets are being dropped between the client and the server, decrease rsize to 4096 or 2048. To find out whether packets are being dropped, issue the “nfsstat -rc” command. If the timeout and retrans values returned by this command are high, but the badxid number is close to zero, then packets are being dropped somewhere in the network.	8192	if nfsstat -cr reports a significant number (> 5% of the total) of both timeouts and badxids, you could increase the timeo parameter
wsize=n		8192
O (Overlay mount)	Allows the file system to be mounted over an existing mount point, making the underlying file system inaccessible. If you attempt to mount a file system over an existing mount point without the -O option, the mount will fail with the error device busy. Caution: Using the -O mount option can put your system in a confusing state. The -O option allows you to hide local data under an NFS mount point without receiving any warning. Local data hidden beneath an NFS mount point will not be backed up during regular system backups. On HP-UX, the -O option is valid only for NFS-mounted file systems. For this reason, if you specify the -O option, you must also specify the -F nfs option to the mount command or the nfs file system type in the /etc/fstab file.	No Default value
remount	If the file system is mounted read-only, this option remounts it read/write. This allows you to change the access permissions from read-only to read/write without forcing everyone to leave the mounted directory or killing all processes using it	No Default value
noac	If specified, this option prevents the NFS client from caching attributes for the mounted directory. Specify noac for a directory that will be used frequently by many NFS clients. The noac option ensures that the file and directory attributes on the server are up to date, because no changes are cached on the clients. However, if many NFS clients using the same NFS server all disable attribute caching, the server may become overloaded with attribute requests and updates. You can also use the actimeo option to set all the caching timeouts to a small number of seconds, like 1 or 3. If you specify noac, do not specify the other caching options.	No Default value
nocto	If specified, this option suppresses fresh attributes when opening a file. Specify nocto for a file or directory that never changes, to decrease the load on your network	No Default value
acdirmax=n	The maximum number of seconds a directory’s attributes are cached on the NFS client. When this timeout period expires, the client flushes its attribute cache, and if the attributes have changed, the client sends them to the NFS server. For a directory that rarely changes or that is owned and modified by only one user, like a user’s home directory, you can decrease the load on your network by setting acdirmax=120 or higher	60
acdirmin=n	The minimum number of seconds a directory’s attributes are cached on the NFS client. If the directory is modified before this timeout expires, the timeout period is extended by acdirmin seconds. For a directory that rarely changes or that is owned and modified by only one user, like a user’s home directory, you can decrease the load on your network by setting acdirmin=60 or higher	30
acregmax=n	The maximum number of seconds a file’s attributes are cached on the NFS client. When this timeout period expires, the client flushes its attribute cache, and if the attributes have changed, the client sends them to the NFS server. For a file that rarely changes or that is owned and modified by only one user, like a file in a user’s home directory, you can decrease the load on your network by setting acregmax=120 or higher	60
actimeo=n	Setting actimeo to n seconds is equivalent to setting acdirmax, acdirmin, acregmax, and acregmin to n seconds. Set actimeo=1 or actimeo=3 for a directory that is used and modified frequently by many NFS clients. This ensures that the file and directory attributes are kept reasonably up to date, even if they are changed frequently from various client locations. Set actimeo=120 or higher for a directory that rarely or never changes. If you set the actimeo value, do not set the acdirmax, acdirmin, acregmax, or acregmin values	No Default value
grpid	Forces a newly created file in the mounted file system to inherit the group ID of the parent directory. By default, a newly created file inherits the effective group ID of the calling process, unless the GID bit is set on the parent directory. If the GID bit is set, the new file inherits the group ID of the parent directory	No Default value
lock / nolock	Selects whether to use the NLM sideband protocol to lock files on the server. If neither option is specified (or if lock is specified), NLM locking is used for this mount point. When using the nolock option, applications can lock files, but such locks provide exclusion only against other applications running on the same client. Remote applications are not affected by these locks. NLM locking must be disabled with the nolock option when using NFS to mount /var because /var contains files used by the NLM implementation on Linux. Using the nolock option is also required when mounting exports on NFS servers that do not support the NLM protocol.
local_lock=mechanism	Specifies whether to use local locking for any or both of the flock and the POSIX locking mechanisms. mechanism can be one of all, flock, posix, or none. The Linux NFS client provides a way to make locks local. This means, the applications can lock files, but such locks provide exclusion only against other applications running on the same client. Remote applications are not affected by these locks. If all is specified, the client assumes that both flock and POSIX locks are local. If flock is specified, the client assumes that only flock locks are local and uses NLM sideband protocol to lock files when POSIX locks are used. If posix is specified, the client assumes that POSIX locks are local and uses NLM sideband protocol to lock files when flock locks are used. To support legacy flock behavior similar to that of NFS clients < 2.6.12, use Samba as Samba maps Windows share mode locks as flock. Since NFS clients > 2.6.12 implement flock by emulating POSIX locks, this will result in conflicting locks.	No Default value	This option is supported in kernels 2.6.37 and later. NOTE: When used together, the 'local_lock' mount option will be overridden by 'nolock'/'lock' mount option.

NFS client mount examples

under cli, 

sudo mount -t nfs -o vers=4 192.168.33.10:/backups /backups
sudo mount -t nfs -o vers=4 192.168.33.10:/www /srv/www

use fstab
192.168.33.10:/backups /backups   nfs   nfsvers=4.1,defaults,nofail,timeo=900,retrans=5,_netdev	0 0
192.168.33.10:/www /srv/www       nfs   nfsvers=4.1,defaults,nofail,timeo=900,retrans=5,_netdev	0 0

Optimizing NFS Performance

Tuning for performance is a loaded question because performance is defined by so many different variables, the most important of which is how to measure performance.

Tunning Options	Target	Description	Recommendation
NFS performance	Synchronous vs asynchronous on client	sync or async on the mount command line or in the file /etc/fstab	To change the option, you first have to unmount the NFS filesystem, change the option, then remount the filesystem.
	Number of NFS daemons (nfsd) on server	One way to determine whether more NFS threads helps performance is to check the data in cat /proc/net/rpc/nfsd (Ubuntu) for the load on the NFS daemons. The output line that starts with th lists the number of threads, and the last 10 numbers are a histogram of the number of seconds the first 10% of threads were busy, the second 10%, and so on. this page explain how to part the contents of /proc/net/rpc/nfsd. Ideally, you want the last two numbers to be zero or close to zero, indicating that the threads are busy and you are not "wasting" any threads. If the last two numbers are fairly high, you should add NFS daemons, because the NFS server has become the bottleneck. If the last two, three, or four numbers are zero, then some threads are probably not being used on Ubuntu 20.04, RPCNFSDCOUNT in /etc/default/nfs-kernel-server file tells you the number of NFS daemons for the server. NFS tips and tricks in addition, for tuning how many threads are needed you could look at : /proc/fs/nfsd/pool_stats	256 on 16C/128GB 64 on 8C/64GB to take effect reboot system
	Block Size Setting on client	Two NFS client options specify the size of data chunks for writing (wsize) and reading (rsize). If you don't specify the chunk sizes, the defaults are determined by the versions of NFS and the kernel being used the best way to check the current chunk size is to run the command on the NFS client and look for the wsize and rsize values. $cat /proc/mounts	edit /etc/fstab to reflect the rsize/wsize
	Timeout and Retransmission on client	On congested networks, you often see retransmissions of RPC packets. A good way to tell is to run the nfsstat -r command and look for the column labeled retrans. If the number is large, the network is likely very congested. If that is the case, you might want to increase the values of timeo and retrans to increase the number of tries and the amount of time between RPC tries. Although taking this action will slow down NFS performance, it might help even out the network traffic so that congestion is reduced. In my experience, getting rid of congestion and dropped packets can result in better, more even performance	timeo can be adjusted of the mount command or by editing the /etc/fstab file on the NFS client to indicate the value of timeo.
	FS-Cache on client	FS-Cache option enable caches NFS client requests on a local storage device, such as a hard drive or SSD, helping improve NFS read I/O: Data that resides on the local NFS client means the NFS server does not have to be contacted. To use NFS caching you have to enable it explicitly by adding the option -o fsc to the mount command or in /etc/fstab: # mount <nfs-share>:/ </mount/point> -o fsc Any data access to </mount/point> will go through the NFS cache unless the file is opened for direct I/O or if a write I/O is performed. The important thing to remember is that FS-Cache only works if the I/O is a read. FS-Cache can't help with a direct I/O (read or write) or an I/O write request. However, there are plenty of cases in which FS-Cache can help. For example, if you have an application that needs to read from a database or file and you are running a large number of copies of the same application, FS-Cache might help, because each node could cache the database or file.
	Filesystem-independent mount options on client	Linux mount command has a number of options that are independent of the filesystem and might be able to improve performance noatime – Inode access times are not updated on the filesystem. This can help performance because the access time of the file is not updated every time a file is accessed. nodiratime – The directory inode is not updated on the filesystem when it is accessed. This can help performance in the same way as not updating the file access time. relatime – Inode access times are relative to the modify or change time for the file, so the access time is updated only if the previous atime (access time) was earlier than the modify or change time.
System tuning	System Memory on server	If you are choosing to use asynchronous NFS mode, you will need more memory to take advantage of async, because the NFS server will first store the I/O request in memory, respond to the NFS client, and then retire the I/O by having the filesystem write it to stable storage. Therefore, you need as much memory as possible to get the best performance.
	Network MTU on both server and client	Changing the network MTU (maximum transmission unit) is also a good way to affect performance, but it is not an NFS tunable. The MTU size can be very important because it determines packet fragments on the network. If your chunk size is 8KB and the MTU is 1500, it will take six Ethernet frames to transmit the 8KB. If you increase the MTU to 9000 (9,000 bytes), the number of Ethernet frames drops to one. A study by Dell a few years back examined the effect of an MTU of 1500 compared with an MTU of 9000. Using Netperf, they found that the bandwidth increased by about 33% when an MTU of 9000 was used Fortunatly most switches can accommodate an MTU of 9000 (commonly called "jumbo packets")
	TCP tuning on the server	The NFS server NFS daemons share the same socket input and output queues, so if the queues are larger, all of the NFS daemons have more buffer and can send and receive data much faster. For read buffer - 256KiB (base 2), echo 262144 > /proc/sys/net/core/rmem_default echo 262144 > /proc/sys/net/core/rmem_max For write buffer - 256KiB (base 2), echo 262144 > /proc/sys/net/core/wmem_default echo 262144 > /proc/sys/net/core/wmem_max To make the values survive reboots, you need to enter them in the proper form in the /etc/sysctl.conf file or /etc/sysctl.d/somefile
NFS management/policy	Subtree checking on server	subtree_check to the exports on the NFS server checks that the file being accessed is contained within the exported directory and force the NFS server to check that the requested file was located within exported folder Many people have the opinion that subtree_check can have a big effect on performance, but the final determination is up to that is performance more important than security for the configuration and your situation For security, it is recommended to export folders which uses separate partition or separate drive to prevent a rogue user from guessing a file handle to anything outside of the filesystem
	Root squashing on server	if you want root to have access to an NFS-mounted filesystem, you can add the option no_root_squash to the file /etc/exports to allow root access. Just be aware that if someone reboots your system to gain root access, it's possible for them to copy (steal) data.

Specify NFS v4.1 or, Better Yet, v4.2

probably:
# mount -o nfsvers=4.2 srvr:/export /mountpoint

or possibly:
# mount -o nfsvers=4,minorversion=2 srvr:/export /mountpoint

Specify Bytes Per Read/Write With rsize and wsize

The mount command options rsize and wsize specify the size of the chunks of data that the client and server pass back and forth to each other. All 2.4 clients currently support up to 32K block transfer sizes.

Put options for static mounts in /etc/fstab. Put options for automounted file systems in the map file.

# grep nfs /etc/fstab
srvr1:/usr/local  /usr/local  nfs  nfsvers=4.2,rsize=131072,wsize=131072  0  0
# tail -1 /etc/autofs/auto.master
/-   auto.local
# cat /etc/autofs/auto.local
/usr/local  -nfsvers=4.2,rsize=131072,wsize=131072  srvr1:/usr/local 

What may be a better way is to use the systemd automounter.
# grep nfs /etc/fstab
srvr1:/usr/local  /usr/local  nfs  noauto,x-systemd.automount,nfsvers=4.2,rsize=131072,wsize=131072  0  0

Mount Options Example

• In Linux kernel 5.3 (and higher), the nconnect option allows multiple TCP connections for a single NFS mount. Note: Currently, the maximum number of concurrent TCP connections is 16.
• In Linux kernel 2.6.25 (and higher), the intr and nointr mount options are deprecated. If you use the hard option on modern Linux kernels, you must use the kill -9 (SIGKILL) command to interrupt a stuck NFS mount.

Condition	Recommended Options
Client with Server-Side Network Lock Manager (NLM) Enabled	mount -t nfs -o rsize=65536,wsize=65536,intr,hard,tcp,rdirplus,readahead=128 server:/path mountpath
Client with Local Locking Enforced	mount -t nfs -o rsize=65536,wsize=65536,intr,hard,tcp,locallocks,rdirplus,readahead=128 \ server:/path mountpath