Table of Contents

1. Introduction
2. "Failed" versus "failing" disks
3. Making an emergency backup of the disk group configuration
4. Has the disk been overwritten by another logical volume manager solution?
5. Determining if a disk can be reattached
6. Verifying that a disk is readable to the operating system
7. Have the paths to the disk been disabled?
8. Restoring the disk group configuration using vxconfigrestore
9. Restoring the disk group configuration manually, using UDIDs and Disk IDs
10. Restarting the volume

1. Introduction

(Back to top)

This article contains a procedure for troubleshooting the "failed," or "failed was" status, as reported by vxdisk.

A "failed" status is a record of a disk that is no longer accessible. This is often caused by sustained I/O errors to the disk that prevents it from being read by the operating system (OS). It may also be the result of corruption within the Veritas private region.

The private region is the portion of the disk where Veritas stores records about the disk group, such as disks, volumes, subdisks and plexes. This can be contrasted with the public region, which contains the actual volumes, including user data.

2. "Failed" versus "failing" disks

(Back to top)

The status of "failed" should not be confused with a status of "failing." This article primarily discusses the "failed" status, as reported by vxdisk. For information on troubleshooting the "failing" status, see http://www.symantec.com/docs/TECH61915.

3. Making an emergency backup of the disk group configuration

(Back to top)

Before making any further changes, use vxconfigbackup to create an emergency backup of the private region for the remaining disks in the affected disk group.

Vxconfigbackup does not back up the actual data that is contained within the volumes. Instead, it backs up the Veritas private region configuration database that resides on the disks, along with some information about the disks themselves. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.

If vxconfigbackup is not available, vxprivutil can be used to dump a copy of the configuration database.
 

More details about vxconfigbackup and vxprivutil, including syntax and examples, can be found in this article:

"Using vxconfigbackup and vxprivutil to back up the disk group configuration of the Veritas private region"
http://www.symantec.com/docs/S:TECH201329

4. Has the disk been overwritten by another logical volume manager solution?

(Back to top)

If vxdisk shows that the disk type includes the words "LVM" or "ZFS," then the disk may have been overwritten by another logical volume manager (LVM) solution. It is also possible that there is a problem with the SAN zoning which may have caused disks to be presented to the wrong systems. Before making any further changes, ensure that the disk is not supposed to be zoned to another system.

To bring a disk back into its original, Veritas disk group, the disk must first be removed from the control of the other LVM solution and then initialized for Veritas, using vxdisksetup. Refer to the documentation for the appropriate vendor for information about removing a disk from the control of their LVM solution.

5. Determining if a disk can be reattached

(Back to top)
 

Vxreattach is used to restore the original disk media name and reattach the disk back to the disk group. It can normally only be used if the status of the disk is "online" (see Figure 1).

Run vxreattach, using the "-c" argument, to determine if a disk can be reattached to the disk group.


Figure 1 - Using vxreattach, with the "-c" argument, to check if a reattach is possible

If vxreattach -c returns a disk group and disk media name, without returning any errors, proceed with reattaching the disk (Figure 2). If a reattach is not possible, a V-5-2-238 error will appear.

Figure 2 - Using vxreattach to reattach a disk to the disk group

6. Verifying that a disk is readable to the operating system

(Back to top)

Use native OS commands to confirm that the OS can read the disk, including the disk label.

• Use commands, such as prtvtoc, fdisk, lspv or diskinfo to read the disk label.
• Use dd to read a block from the disk.

Veritas depends on the OS device drivers to communicate with disks. If the OS is unable to read a disk, Veritas will also fail to read it. If a disk does not have a label, or the label has been corrupted, Veritas will not recognize the disk. Completing these steps will assist with identifying the source of a disk outage.
 

7. Have the paths to the disk been disabled?

(Back to top)

Use vxdmpadm to determine the status of the paths to the disks (Figure 3).

Veritas will disable a path if serious or sustained I/O errors occur. When all paths to a disk are disabled, the server will be unable to read or write to the volume. If a path has been disabled, review the syslog for events that are reported by "vxdmp," or "scsi" for I/O errors.

Although a path can be re-enabled using "vxdmpadm enable," vxdmp should automatically evaluate the status of a path in five minute intervals using a scsi inquiry. If the query is successful, the path is automatically re-enabled. If a path remains disabled beyond this interval, it is possible that I/O errors are still being detected, warranting further investigation. Paths are not automatically re-enabled If the diskgroup has been disabled, or if vxesd is stopped. The behavior of vxdmp in response to disabled paths can be modified via the DMP tunables, which can be viewed using "vxmpadm gettune."
 

Figure 3 - Example of a disabled path, as reported by vxdmpadm

8. Restoring the disk group configuration using vxconfigrestore

(Back to top)

If a vxreattach is not possible, use vxconfigrestore to recover the disk group.

Vxconfigrestore does not restore the actual data that is contained within the volumes. It only restores the Veritas configuration database that is located within the private region of the disks. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.
 

9. Restoring the disk group configuration manually, using UDIDs and Disk IDs

(Back to top)

If using vxconfigrestore is not possible, another method for recovering the disks is to compare the UDID or Disk ID attributes of the disks with the records that are contained with the private region configuration database.
 

10. Restarting the volume

(Back to top)

Once the original disk has been added back to its disk group, additional steps may be needed to recover the volume. Use vxprint to determine the current status (Figure 4).

• For mirrored volumes:
  • If at least one plex was not affected by the outage, the other plexes should be resynchronized when they are reattached to the volume.  It may be necessary to use vxrecover to initiate this process (Figure 6).
  • If all plexes were affected by the outage, it may be necessary to manually review each plex to determine which contain the most recent updates.

WARNING: Do not simply force-start a mirrored volume. This may cause a plex that contains old, or corrupt, blocks to overwrite a plex that contains up-to-date data. A procedure for manually determining the most up-to-date mirror plex can be found in this article:

"Manually determining which mirror plex contains the most recent data and then resynchronizing"
http://www.symantec.com/docs/S:TECH202503

• For non-mirrored volumes:
  • It may be necessary to manually restart the volume using vxvol after adding the disk back to the disk group (Figure 5).

Figure 4 - Using vxprint to determine the status of a volume

Figure 5 - Using vxvol to start a volume and using vxprint to review any changes in the status of the volume

Figure 6 - Using vxrecover to finish the recovery, or start the resynchronization, of a volume

keywords: failed, failing, failed disk, failed disks, failing disk, failed disks,