A rare data loss condition involving online (hot) catalog backups has been identified in Veritas NetBackup (tm) 6.0 environments where the IDIRSTRUCT configuration file contains a value of 1.

Article:TECH46433

Created: 2006-01-25

Updated: 2006-01-25

Article URL http://www.symantec.com/docs/TECH46433

Article Type
Technical Solution

Product(s)

Show all

Environment

Show all

Languages

Show all

Problem

A rare data loss condition involving online (hot) catalog backups has been identified in Veritas NetBackup (tm) 6.0 environments where the IDIRSTRUCT configuration file contains a value of 1.

Solution

Introduction:

A rare data loss condition involving online (hot) catalog backups has been identified in NetBackup 6.0 environments where the IDIRSTRUCT configuration file contains a value of 1.

In NetBackup versions 2.0 and earlier, all client images were stored directly in the client's folder within the images directory. This legacy method is considered "version 1" of the images database structure:

Version 1
Image files are stored in the images folder, directly under the client name:

UNIX Example:

/usr/openv/netbackup/db/images/nbu1/test_1139250156_FULL

Windows Example:

C:\Program Files\NetBackup\db\images\nbu1\test_1139250156_FULL

In versions after NetBackup 2.0, functionality was added to NetBackup to store client images within "ctime" numbered folders that reside in the client's image directory. This is considered "version 2" of the images database structure:

Version 2
Image files are stored within "ctime" numbered folders under within the client directory:

UNIX Example:

/usr/openv/netbackup/db/images/nbu1/1139000000/test_1139250156_FULL

Windows Example:

C:\Program Files\NetBackup\db\images\nbu1\1139000000\test_1139250156_FULL

This ctime folder (seen in Version 2) was added to improve performance when browsing client images and became the default for a new NetBackup (post Version 2.0) installation. Installations that were upgraded from NetBackup 2.0 and earlier could be converted to version 2 through a manual process. Alternatively, the image database could simply have been left in the version 1 format, maintaining that version 1 format as the server is upgraded to newer versions of NetBackup.

In a NetBackup 6.0 installation, the version of the image database can be confirmed by checking the contents of the IDIRSTRUCT configuration file and noting its value:

UNIX:

Type the following on the Master Server:

cat /usr/openv/netbackup/db/IDIRSTRUCT

Windows:

In notepad, open the following file on the Master Server:

<install_path>\NetBackup\db\IDIRSTRUCT

If the IDIRSTRUCT file contains a "1", image files are placed in the root of the client directory, as seen in Version 1. If the IDIRSTRUCT file contains a "2", image files are placed in a <client_name>/<ctime> directory, as seen in Version 2.

Image data residing in the root of the client directory (as seen in Version 1) will not be backed up by a hot catalog backup. Therefore, if an IDIRSTRUCT setting of "1" is in use, client image data will not be properly protected by a hot catalog backup.

Note: This issue is likely to be very rare and should not occur unless the NetBackup environment has inherited an IDIRSTRUCT value of 1 from NetBackup 2.0 or earlier, or if the IDIRSTRUCT file has been manually modified. It is not recommended to modify the IDIRSTRUCT file under any circumstances, unless directed to do so by Support.

What is Affected:
This issue affects Master Servers on all supported platforms running the following NetBackup Enterprise Server / NetBackup Server releases:

NetBackup 6.0
NetBackup 6.0, Maintenance Pack 1 (MP1)
NetBackup 6.0, Maintenance Pack 2 (MP2)

How to Determine if Affected:
Data loss has been known to occur if ALL of the following conditions are true:

The NetBackup Master Server is running one of the affected versions of NetBackup, mentioned above.
The IDIRSTRUCT configuration file contains a value of 1.
The "Hot" catalog backup method is being used.
A catalog recovery is necessary and hot catalog backup(s) are the only available catalog backups from which to recover.

Formal Resolution:
This issue is resolved in NetBackup 6.0, Maintenance Pack 3 (MP3), currently available for download at the below link:
http://support.veritas.com/menu_ddProduct_NBUESVR_view_DOWNLOAD.htm

Until this formal resolution can be implemented, Symantec strongly recommends following the workaround documented in the next section of this TechNote, and then perform a full backup.

Workaround:
There are two separate possible workarounds to prevent this issue. Workaround 1 is recommended, as it resolves the Hot catalog backup issue and is also likely to increase performance.

1. Convert to version 2 of the images database. Perform these steps on the master server when no jobs are active:

UNIX Master Servers

- If any client images are compressed, decompress all client images when no jobs are active. It is important to note the available disk space on the volume that contains the NetBackup catalog, to help determine if enough free space is available to decompress the catalog. All client images can be decompressed all at once, or a single client at a time.

Note: If insufficient disk space is available to perform the decompression, either make use of the second workaround until a formal resolution is available, or call Symantec Enterprise Technical Support for NetBackup for assistance in manually relocating the image information.

To decompress the images for all clients:

# /usr/openv/netbackup/bin/admincmd/bpimage -decompress -allclients

It is also possible to decompress images for a single client at a time. This will better ensure that ample disk space will be available to decompress all clients:

# /usr/openv/netbackup/bin/admincmd/bpimage -decompress -client <client name>

(repeat this command for all client names)

Note: If catalog compression is used, decompressing the images for all clients is crucial in order to ensure that ALL catalog information is converted properly! For further information on catalog compression, please refer to the System Administrator's Guide, found below, in the Related Documents section.

- If no compression is used, or if all client images have been decompressed, stop the NetBackup daemons:

# /usr/openv/netbackup/bin/goodies/netbackup stop

- Convert the image directory structure to version 2:

# /usr/openv/netbackup/bin/bpdbm -converti2

- Start NetBackup daemons:

# /usr/openv/netbackup/bin/goodies/netbackup start

- Re-compress the image catalog if desired:

# /usr/openv/netbackup/bin/admincmd/bpimage -compress -allclients

Windows Master Servers

Note: No decompression is needed for Windows Master Servers.

- Stop NetBackup Services:

From the <install_path>\NetBackup\bin directory, type:

bpdown -v -f

- Convert the image directory structure to version 2:

From the <install_path>\NetBackup\bin directory, type:

bpdbm -converti2

- Start NetBackup Services:

From the <install_path>\NetBackup\bin directory, type:

bpup -v -f

2. Use cold/offline catalog backups, instead of hot catalog backups. For detailed information on configuring offline catalog backups, please refer to the System Administrator's Guide, found below, in the Related Documents section.

Note: In the very rare case that a catalog recovery is necessary and the only catalog backup available is a hot catalog backup that has encountered this issue, do not recover the hot catalog backup. If the entire catalog is lost, one option would be to perform an import of each backup media and/or disk image that may contain client images. If this is not feasible, please contact Symantec Enterprise Technical Support for NetBackup for further instruction.

Warning: If a catalog recovery has been performed using a hot catalog backup which has exhibited this issue, please stop all NetBackup services on the Master Server and contact Symantec Technical Services, as there may be a resulting database inconsistency that could lead to loss of client backup data.

Symantec strongly recommends the following best practices:

1. Always perform a Full backup prior to and after any changes to your environment.

2. Always make sure that your environment is running the latest version and patch level.

If you have not received this TechNote from the Symantec Email Notification Service as a Software Alert, please subscribe at the following link:
http://maillist.support.veritas.com/subscribe.asp