Veritas Volume Manager

Veritas produces volume and file system software that allows for extremely flexible and straightforward management of a system's disk storage resources. Now that ZFS is providing much of this same functionality from inside the OS, it will be interesting to see how well Veritas is able to hold on to its installed base.

In Veritas Volume Manager (VxVM) terminology, physical disks are assigned a diskname and imported into collections known as disk groups. Physical disks are divided into a potentially large number of arbitrarily sized, contiguous chunks of disk space known as subdisks. These subdisks are combined into volumes, which are presented to the operating system in the same way as a slice of a physical disk is.

Volumes can be striped, mirrored or RAID-5'ed. Mirrored volumes are made up of equally-sized collections of subdisks known as plexes. Each plex is a mirror copy of the data in the volume. The Veritas File System (VxFS) is an extent-based file system with advanced logging, snapshotting, and performance features.

VxVM provides dynamic multipathing (DMP) support, which means that it takes care of path redundancy where it is available. If new paths or disk devices are added, one of the steps to be taken is to run vxdctl enable to scan the devices, update the VxVM device list, and update the DMP database. In cases where we need to override DMP support (usually in favor of an alternate multipathing software like EMC Powerpath), we can run vxddladm addforeign.

Standard Operations

Here are some procedures to carry out several common VxVM operations. VxVM has a Java-based GUI interface as well, but I always find it easiest to use the command line.

Operation	Procedure
Create a volume (length specified in sectors, KB, MB or GB)	vxassist -g dg-name make vol-name length(skmg)
Create a striped volume (add options for a stripe layout)	layout=stripe diskname1 diskname2 ...
Remove a volume (after unmounting and removing from vfstab)	vxstop vol-name then vxassist -g dg-name remove volume vol-name or vxedit -rf rm vol-name
Create a VxFS file system	mkfs -F vxfs -o largefiles /dev/vx/rdsk/dg-name/vol-name
Snapshot a VxFS file system to an empty volume	mount -F vxfs -o snapof=orig-vol empty-vol mount-point
Display disk group free space	vxdg -g dg-name free
Display the maximum size volume that can be created	vxassist -g dg-name maxsize [attributes]
List physical disks	vxdisk list
Print VxVM configuration	vxprint -ht
Add a disk to VxVM	vxdiskadm (follow menu prompts) or vxdiskadd disk-name
Bring newly attached disks under VxVM control (it may be necessary to use format or fmthard to label the disk before the vxdiskconfig)	drvconfig; disks vxdiskconfig vxdctl enable
Scan devices, update VxVM device list, reconfigure DMP	vxdctl enable
Scan devices on OS device tree, initiate dynamic reconfig of multipathed disks	vxdisk scandisks
Reset a disabled vxconfigd daemon	vxconfigd -kr reset
Manage hot spares	vxdiskadm (follow menu options and prompts) vxedit set spare=[off|on] vxvm-disk-name
Rename disks	vxedit rename old-disk-name new-disk-name
Rename subdisks	vxsd mv old-subdisk-name new-subdisk-name
Monitor volume performance	vxstat
Re-size a volume (but not the file system)	vxassist growto|growby|shrinkto|shrinkby volume-name length[s|m|k|g]
Resize a volume, including the file system	vxresize -F vxfs volume-name new-size[s|m|k|g]
Change a volume's layout	vxassist relayout volume-name layout=layout

The progress of many VxVM tasks can be tracked by setting the -t flag at the time the command is run: utility -t taskflag. If the task flag is set, we can use vxtask to list, monitor, pause, resume, abort or set the task labeled by the tasktag.

Physical disks which are added to VxVM control can either be initialized (made into a native VxVM disk) or encapsulated (disk slice/partition structure is preserved). In general, disks should only be encapsulated if there is data on the slices that needs to be preserved, or if it is the boot disk. (Boot disks must be encapsulated.) Even if there is data currently on a non-boot disk, it is best to back up the data, initialize the disk, create the file systems, and restore the data.

When a disk is initialized, the VxVM specific information is placed in a reserved location on the disk known as a private region. The public region is the portion of the disk where the data will reside.

VxVM disks can be added as one of several different categories of disks:

sliced	Public and private regions are on separate physical partitions. (Usually s3 is the private region and s4 is the public region, but encapsulated boot disks are the reverse.)
simple	Public and private regions are on the same disk area.
cdsdisk	(Cross-Platform Data Sharing) This is the default, and allows disks to be shared across OS platforms. This type is not suitable for boot, swap or root disks.

If there is a VxFS license for the system, as many file systems as possible should be created as VxFS file systems to take advantage of VxFS's logging, performance and reliability features.

At the time of this writing, ZFS is not an appropriate file system for use on top of VxVM volumes. Sun warns that running ZFS on VxVM volumes can cause severe performance penalties, and that it is possible that ZFS mirrors and RAID sets would be laid out in a way that compromises reliability.

VxVM Maintenance

The first step in any VxVM maintenance session is to run vxprint -ht to check the state of the devices and configurations for all VxVM objects. (A specific volume can be specified with vxprint -ht volume-name.) This section includes a list of procedures for dealing with some of the most common problems. (Depending on the naming scheme of a VxVM installation, many of the below commands may require a -g dg-name option to specify the disk group.)

Volumes which are not starting up properly will be listed as DISABLED or DETACHED. A volume recovery can be attempted with the vxrecover -s volume-name

If all plexes of a mirror volume are listed as STALE, place the volume in maintenance mode, view the plexes and decide which plex to use for the recovery

vxvol maint volume-name (The volume state will be DETACHED.)
vxprint -ht volume-name
vxinfo volume-name (Display additional information about unstartable plexes.)
vxmend off plex-name (Offline bad plexes.)
vxmend on plex-name (Online a plex as STALE rather than DISABLED.)
vxvol start volume-name (Revive stale plexes.)
vxplex att volume-name plex-name (Recover a stale plex.)

If, after the above procedure, the volume still is not started, we can force a plex to a 'clean' state. If the plex is in a RECOVER state and the volume will not start, use a -f option on the vxvol command

vxmend fix clean plex-name
vxvol start volume-name
vxplex att volume-name plex-name

If a subdisk status is listing as NDEV even when the disk is listed as available with vxdisk list the problem can sometimes be resolved by running

vxdg deport dgname
vxdg import dgname

to re-initialize the disk group. To remove a disk: Copy the data elsewhere if possible. Unmount file systems from the disk or unmirror plexes that use the disk.

vxvol stop volume-name (Stop volumes on the disk.)
vxdg -g dg-name rmdisk disk-name (Remove disk from its disk group.)
vxdisk offline disk-name (Offline the disk.)
vxdiskunsetup c#t#d# (Remove the disk from VxVM control.)

To replace a failed disk other than the boot disk

In vxdiskadm, choose option 4: Remove a disk for replacement. When prompted, chose 'none'for the disk to replace it. Physically remove and replace the disk. (A reboot may be necessary if the disk is not hot-swappable.)

In the case of a fibre channel disk, it may be necessary to remove the /dev/dsk and /dev/rdsk links and rebuild them with

drvconfig
disks

or a reconfiguration reboot.

In vxdiskadm, choose option 5: Replace a failed or removed disk. Follow the prompts and replace the disk with the appropriate disk.

To replace a failed boot disk: Use the eeprom command at the root prompt or the printenv command at the <ok> prompt to make sure that the nvram=devalias and <code.boot-device parameters are set to allow a boot from the mirror of the boot disk. If the boot paths are not set up properly for both mirrors of the boot disk, it may be necessary to move the mirror disk physically to the boot disk's location. Alternatively, the devalias command at the <ok> prompt can set the mirror disk path correctly, then use nvstore to write the change to the nvram. (It is sometimes necessary to nvunalias aliasname to remove an alias from the nvramrc, then nvalias aliasname devicepath to set the new alias, then nvstore to write the changes to nvram.)

In short, set up the system so that it will boot from the boot disk's mirror. Repeat the steps above to replace the failed disk. Clearing a "Failing" Flag from a Disk:

• First make sure that there really is not a hardware problem, or that the problem has been resolved.
• Then, vxedit set failing=off disk-name

Clearing an IOFAIL state from a plex:

• First make sure that the hardware problem with the plex has been resolved.
• Then,

  vxmend -g dgname -o force off plexname
  vxmend -g dgname on plexname
  vxmend -g dgname fix clean plexname
  vxrecover -s volname

VxVM Mirroring

Most volume manager availability configuration is centered around mirroring. While RAID-5 is a possible option, it is infrequently used due to the parity calculation overhead and the relatively low cost of hardware-based RAID-5 devices.

In particular, the boot device must be mirrored; it cannot be part of a RAID-5 configuration.

To mirror the boot disk eeprom use-nvramrc?=true

Before mirroring the boot disk, set use-nvramrc? to true in the EEPROM settings. If you forget, you will have to go in and manually set up the boot path for your boot mirror disk. (See "To replace a failed boot disk" in the "VxVM Maintenance" section for the procedure.) It is much easier if you set the parameter properly before mirroring the disk! The boot disk must be encapsulated, preferably in the bootdg disk group. (The bootdg disk group membership used to be required for the boot disk. It is still a standard, and there is no real reason to violate it.) If possible, the boot mirror should be cylinder-aligned with the boot disk. (This means that the partition layout should be the same as that for the boot disk.) It is preferred that 1-2MB of unpartitioned space be left at either the very beginning or the very end of the cylinder list for the VxVM private region. Ideally, slices 3 and 4 should be left unconfigured for VxVM's use as its public and private region. (If the cylinders are aligned, it will make OS and VxVM upgrades easier in the future.) (Before bringing the boot mirror into the bootdg disk group, I usually run an installboot command on that disk to install the boot block in slice 0. This should no longer be necessary; vxrootmir should take care of this for us. I have run into circumstances in the past where vxrootmir has not set up the boot block properly; Veritas reports that those bugs have long since been fixed.) Mirrors of the root disk must be configured with "sliced" format and should live in the bootdg disk group. They cannot be configured with cdsdisk format. If necessary, remove the disk and re-add it in vxdiskadm. In vxdiskadm, choose option 6: Mirror Volumes on a Disk. Follow the prompts from the utility. It will call vxrootmir under the covers to take care of the boot disk setup portion of the operation. When the process is done, attempt to boot from the boot mirror. (Check the EEPROM devalias settings to see which device alias has been assigned to the boot mirror, and run boot device-alias from the ok> prompt. Procedure to create a Mirrored-Stripe Volume: (A mirrored-stripe volume mirrors several striped plexes it is better to set up a Striped-Mirror Volume.)

vxassist -g dg-name make volume length layout=mirror-stripeCreating a Striped-Mirror Volume: (Striped-mirror volumes are layered volumes which stripes across underlaying mirror volumes.) vxassist -g dg-name make volume length layout=stripe-mirror Removing a plex from a mirror: vxplex -g dg-name -o rm dis plex-name

Removing a mirror from a volume: vxassist -g dg-name remove mirror volume-name

Removing a mirror and all associated subdisks: vxplex -o rm dis volume-name

Dissociating a plex from a mirror (to provide a snapshot):

vxplex dis volume-name
vxmake -U gen vol new-volume-name plex=plex-name (Creating a new volume with a dissociated plex.)
vxvol start new-volume-name
vxvol stop new-volume-name (To re-associate this plex with the old volume.)
vxplex dis plex-name
vxplex att old-volume-name plex-name
vxedit rm new-volume-name

Removing a Root Disk Mirror:

vxplex -o rm dis rootvol-02 swapvol-02 [other root disk volumes
/etc/vx/bin/vxunroot