Tuning Systems for NFS - By Neil Horman
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Tuning Systems for NFS
By Neil Horman
Abstract
Network File System (NFS) has been one of the most
popular forms of filesharing between Linux® and UNIX®
systems. This paper provides a list of the most common
issues users encounter when using NFS and suggestions
for ways to tune systems to avoid some or all of them.
February 2005
Table of Contents
Introduction 2
Issues 2
Performance 2
Limiting file system data that needs to be cached 2
Coalesce data into larger segments 3
Write software with NFS in mind 3
Integrity 4
Looking ahead 5
Copyright © 2005 Red Hat, Inc. All rights reserved. “Red Hat” and the “Shadowman” logo are registered trademarks of Red Hat, Inc. in the US and other countries. Linux is a
registered trademark of Linus Torvalds. All other trademarks referenced herein are the trademarks of their respective owners. WHP73871US 01/05Introduction
Linux and UNIX systems have always been very strong in their ability to
share data with one another. Communication protocols such as UUCP, FTP,
HTTP, and SCP, allow users of these systems to share data in a simple,
secure, and reliable fashion. Network File System (NFS) has been one of the
most popular forms of filesharing between Linux and UNIX systems. NFS
version 4 (NFSv.4) is available as a filesystem option in Red Hat Enterprise
Linux 4.
This communication protocol allows Linux workstations and servers to make
portions of their local file systems accessible to other Linux machines over a
network. By mounting a remote filesystem via NFS, a Linux workstation can
access files in the same manner that an application would access local files.
This makes NFS particularly appealing because it is completely transparent to
all applications. This paper provides a list of the most common issues users
encounter with NFS and suggestions for ways to tune systems to avoid some
or all of these issues.
Issues
While NFS operates under the same semantics as other file systems, it does
not behave as other filesystems do. The fact that NFS operates over a
network creates a performance bottleneck that can become quite severe under
the right circumstances. A network also provides malicious individuals with the
opportunity to access sensitive data without your knowledge. Lastly, while one
of the strengths of NFS is the ability to share data with multiple users, the
implicit shared nature also creates an environment in which data can become
easily corrupted.
Performance
The most prevalent issue people face when implementing an environment
including NFS is the performance it provides. Networks have significantly less
throughput than the busses that transport data to other types of storage (IDE,
SCSI, SATA, etc). The relatively low throughput of the network, plus the fact
that multiple nodes will be accessing NFSbased storage, equals a high
demand for the physical disk on the server, and every ounce of throughput
counts. NFS primarily overcomes its natural performance issues via
aggressive clientside caching, the attributes of which are largely fixed. There
are several measures an administrator can take to influence cache behavior
and affect performance. These measures are detailed below.
Limiting file system data that needs to be cached
One of the ways to improve performance in an NFS environment is to limit the
amount of data the file system must retrieve from the server. This limits the
amount of network traffic generated, and, in turn, improves file system
performance. Metadata to be retrieved on the client and updated on the
server includes:
• access time
• modification time
• change time
• ownership
• permissions
• size
Under most local file systems this data is cached in RAM and written back to
disk when the operating system finds the time to do so. The conditions under
which NFS runs are far more constrained. An enormous amount of excess
network traffic would be generated by writing back file system metadata when
a client node changes it. On the other hand, by waiting to write back metadata,
other client nodes do not see updates from each other, and applications that
rely on this data can experience excess latencies and race conditions.
Depending on which applications use NFS, the attribute cache can be tuned to
provide optimum performance. The following mount options affect how the
attribute cache retains attributes:
• acregmin – The amount of time (in seconds) the attributes of a regular file
must be retained in the attribute cache
• acregmax – The amount of time (in seconds) the attributes of a regular file
may remain in cache before the next access to the file must refresh them
from the server
• acdirmin – Same as acregmin, but applied to directory inodes
• acdirmax – Sam as acregmax, but applied to directory inodes
There are also two settings: actimeo, which sets all four of the above numbers
to the same value, and noac, which completely disables the attribute cache.
By increasing these values, one can increase the amount of time attributes
remain in cache, and improve performance.
Coalesce data into larger segments
One of the most potentially effective measures an admin can make on an NFS
file system is the adjustment of the read and write size. This setting allows the
NFS server to reduce the overhead of clientserver communication, allowing it
to send larger transactions to the NFS server when the server is available. By
default, most NFS clients set their read and write size to 8KB, allowing a read
or write NFS transaction to transfer up to 8KB of file data. This transaction
consists of an NFS read/write transaction request and a set of packets. In
case of a write, the payload is datacarrying packets; if it'
s a read, they are
response packets. By increasing the read and write size, fewer read/write
transactions are required, which means less network traffic and better
performance.
Write software with NFS in mind
Since NFS file systems generate traffic based on the how programs access
filesystem data, it can have a strong impact on how well the file system
performs. It is important that programs access files while keeping them in the
page cache, thus reducing network traffic and latency. Here are a few factors
to consider when preforming file I/O in a program over an NFS file system.
• Avoid unnecessarily opening and closing of files. If you access a file
frequently, opening and closing that file triggers the opentoclose cache
consistency mechanism in the NFS file system. Triggering this mechanism
causes NFS getattr requests to be generated, which can slow performance.
By keeping the file open, traffic is kept to a minimum. Of course there are
times when a program must close the file. One example is in a multiple
accessor environment with appropriate file locking as described in the
section on maintaining file integrity.
• Performing applicationbased readahead. Since NFS file systems have a
preset read and write transaction size, it is beneficial if a program attempts
to preform reads and writes which are of a size as large as or larger than
the underlying read/write size. This allows the file system to be used to its
maximum efficiency (assuming of course, that you intend to use future data
from the file).
Integrity
Maintaining data integrity is often an operation in contradiction to increased
performance. When operating on data sets in a parallel environment (a
common use for NFS), mechanisms must be placed within an application to
properly serialize access to shared data while avoiding corruption. This
serialization often leads to decreased performance as parallel operations on
multiple nodes wait for one another to complete in serialized regions. On NFS
file systems, this serialization often takes the form of POSIXstyle file locking.
While the use of file locks is required in many shared data situations, programs
can be written in such a way as to minimize the use of these locks, and
consequently performance degradation can be kept to a minimum.
Specifically, an application developer must understand the balance between
throughput and latency of an application. An application running on multiple
NFS nodes, which makes use of of shared NFS data, will write in varying
patterns to a file. Since sharing data safely on NFS requires file locks, each
application must decide when to lock the file. If the application needs to make
several small, nonconsecutive writes to a file, the developer may choose to
lock the file before the first write and unlock it after the last. This operation will
result in a relatively short write time for the node holding the lock, but may
result in a long latency for the other nodes while they wait to obtain the lock.
Conversely, a developer may choose to lock and unlock the file before and
after every individual write operation. This approach will lead to a more
responsive application, but will bring down overall average performance. By
understanding the needs of their applications, a developer can select the best
combination of throughput and responsiveness.
Looking ahead
Starting with Red Hat Enterprise Linux 4, NFS version 4 (NFSv4) is available
as a filesystem option. This provides several advantages to the system
administrator. NFSv4 provides two major upgrades that sysadmins will find
useful.
• Security. NFSv4 provides security via RFC2078, also known as GSSAPI.
This interface allows for any implementing security mechanism (known as a
security flavor) to be used to secure NFS communications. At a minimum
NFSv4 implements the kerberos5 security flavor, meaning that all NFSv4
transactions can be encrypted to protect against malicious network users
attempting to read secure files in transit over a network.
• Client delegation system. This mechanism is not unlike opportunistic locks
in the SMB protocol. An NFSv4 server, if it detects that a given NFSv4
client is the only node reading from or writing to a file, the server may then
inform the client that updates to the file are not needed immediately. This
means that the client can cache all read or write activity locally and update
the server only in bulk when the file is closed on the client. This can have a
dramatic boost to performanceeven more so if applications can be
designed to limit multiplenode, simultaneous file access. This feature is
not currently implemented in the NFSv4 server available in Enterprise Linux
4, but is planned for a future release.
For more information on Red Hat Enterprise Linux, please visit
www.redhat.com/software/rhel.
You can also read
