ZFS Recommendations

Because ZFS uses kernel addressable memory, we need to make sure to allow enough system resources to take advantage of its capabilities. We should run on a system with a 64-bit kernel, at least 1GB of physical memory, and adequate swap space.

While slices are supported for creating storage pools, their performance will not be adequate for production uses.

Mirrored configurations should be set up across multiple controllers where possible to maximize performance and redundancy.

Scrubbing should be scheduled on a regular basis to identify problems before they become serious.

When latency or other requirements are important, it makes sense to separate them onto different pools with distinct hard drives. For example, database log files should be on separate pools from the data files.

Root pools are not yet supported in the Solaris 10 6/2006 release, though they are anticipated in a future release. When they are used, it is best to put them on separate pools from the other filesystems.

On filesystems with many file creations and deletions, utilization should be kept under 80% to protect performance.

The recordsize parameter can be tuned on ZFS filesystems. When it is changed, it only affects new files. zfs set recordsize=size tuning can help where large files (like database files) are accessed via small, random reads and writes. The default is 128KB; it can be set to any power of two between 512B and 128KB. Where the database uses a fixed block or record size, the recordsize should be set to match. This should only be done for the filesystems actually containing heavily-used database files.

In general, recordsize should be reduced wheniostat regularly shows a throughput near the maximum for the I/O channel. As with any tuning, make a minimal change to a working system, monitor it for long enough to understand the impact of the change, and repeat the process if the improvement was not good enough or reverse it if the effects were bad.

The ZFS Evil Tuning Guide contains a number of tuning methods that may or may not be appropriate to a particular installation. As the document suggests, these tuning mechanisms will have to be used carefully, since they are not appropriate to all installations.

For example, the Evil Tuning Guide provides instructions for:

Turning off file system checksums to reduce CPU usage. This is done on a per-file system basis:

# zfs set checksum=off filesystem
# zfs set checksum='on | fletcher2 | fletcher4 | sha256' filesystem

Limiting the ARC size by setting set zfs:zfs_arc_max in /etc/system in Solaris 8/07 and later.

If the I/O includes multiple small reads, the file prefetch can be turned off by setting zfs:zfs_prefetch_disable in Solaris 8/07 and later.

If the I/O channel becomes saturated, the device level prefetch can be turned off with set zfs:zfs_vdev_cache_bshift = 13 in /etc/system for Solaris 8/07 and later

I/O concurrency can be tuned by setting set zfs:zfs_vdev_max_pending = 10in /etc/system on Solaris 8/07 and later.

If storage with an NVRAM cache is used, cache flushes may be disabled with set zfs:zfs_nocacheflush = 1 in /etc/system for Solaris 11/06 and later.

ZIL intent logging can be disabled. (WARNING: Don't do this.) Metadata compression can be disabled. (Read this section of the Evil Tuning Guide first-- you probably do not need to do this.)