A rare potential for data loss exists in secondary copies of NDMP, FlashBackup and True Image Recovery (TIR) backups for all supported operating systems using NetBackup (tm) 5.1 through 5.1 Maintenance Pack 3A. Please note: The original backup image is not affected.

Article:TECH42496  |  Created: 2005-01-04  |  Updated: 2005-01-04  |  Article URL http://www.symantec.com/docs/TECH42496
Article Type
Technical Solution

Product(s)

Environment

Issue



A rare potential for data loss exists in secondary copies of NDMP, FlashBackup and True Image Recovery (TIR) backups for all supported operating systems using NetBackup (tm) 5.1 through 5.1 Maintenance Pack 3A. Please note: The original backup image is not affected.

Solution



What is Effected
This issue is known to affect the following NetBackup versions on all supported operating systems:

-NetBackup (tm) Enterprise Server 5.1, 5.1 Maintenance Pack 1 (MP1), 5.1 MP2, 5.1 MP3 and 5.1 MP3A
-NetBackup (tm) Server 5.1, 5.1 Maintenance Pack 1 (MP1), 5.1 MP2, 5.1 MP3 and 5.1 MP3A

NetBackup Master Server and Media Server installations are affected.  

How to Determine if Affected
Where ALL of the following three conditions are met, there is a rare potential for data loss to occur in secondary copies of backup images.

1. A duplication is performed against an NDMP, FlashBackup, or True Image Recovery (TIR) enabled backup image.

2. The duplication is local to one media server, running both the read and write operations.
3. The duplicate image (Copy number 2 or higher) is NOT multiplexed to tape.  The "Images on Media" report can be used to determine if an image is multiplexed.
NOTE:  This issue affects only the duplicate images.  The data contained in the original backup image is not affected by this issue.  Be aware that NDMP backups do not support the multiplexing feature, and therefore, will always meet condition three above.
A duplication job meeting all of these conditions has the potential to write the TIR fragment incorrectly on the duplicate image.  In addition to TIR enabled backups, it is suspected that FlashBackup, FlashBackup-Windows and NDMP backup types could also be affected by this issue.  While these backup types do not use the TIR feature itself, they do utilize the TIR fragment written in the backup image.  If affected, these backup types may show symptoms other than those noted below for TIR backups.
Because TIR data is also stored within the NetBackup image catalog on disk, the issue will only manifest when the TIR fragment must be read from the duplicate image.  Reading the TIR fragment from the image is required if the image is imported, or if an incremental backup image is read after the TIR data has been "pruned" from the catalog.  Full backups will retain TIR data in the catalog on disk until the image expires.
In a scenario where the affected duplicate image has been imported, the NetBackup catalog files for that image will be corrupted.  As a result the following error may occur when attempting to browse the image in the Backup Archive and Restore interface:
ERROR: database system error
A Verify job run against the image may report the following:
7/14/2005 11:47:40 AM - Error bpverify(pid=996) Verify of policy Policy_Name, schedule_name (client_1121348244) failed, database system error.
The greatest concern is for environments in which the original copy is expired quickly, such as when using a Virtual Tape Library (VTL), Disk Storage Unit (DSU) or Disk Staging Storage Unit (DSSU) to stage backups for duplication to tape.  The original backup to disk is written correctly, but may expire soon after the second and subsequent copies are made.  There is then a potential for data loss.

Please Note:  While NetWare Media Server backups rely on the NDMP protocol, they are not impacted by this issue.


Workarounds
Symantec strongly suggests implementing one of the following temporary workarounds, until the formal resolution is released and can be applied to your environment (refer to the Formal Resolution section, below, for release information):

1. To prevent impacting additional duplicate images, create the following as a zero length file named "NOSHMDUP"on all affected Media servers:

UNIX: /usr/openv/netbackup/NOSHMDUP
Windows: <install_path>\NetBackup\NOSHMDUP

This can be accomplished with the touch command on UNIX, or by creating an empty text file on Windows, by using either Notepad or Edit, and removing the .txt extension.  Be aware that file extensions are typically hidden, by default, on Windows platforms.

Stop and restart of the NetBackup daemons or services to put this setting into effect.  This file will prevent NetBackup duplication jobs from using a shared memory buffer to read and write the data.  Once in place, any duplicate images created will no longer be impacted by this issue and a full backup should be performed.

Be aware that this configuration may have a small impact on duplication performance.  As a result, adjustments to the available time window for scheduled duplications may be necessary.  


2. If data must be recovered from an affected image, restore from the original backup.  If the original copy is unavailable, please contact Symantec Enterprise Technical Support.  In many cases, the data can be at least partially recovered.



Formal Resolution
This issue will be formally resolved with NetBackup 5.1 Maintenance Pack 4 (MP4), scheduled for release in Q4 of 2005.  When NetBackup 5.1 MP4 is released, it will be available at the following link:      http://support.veritas.com/menu_ddProduct_NBUESVR_view_DOWNLOAD.htm .
Once 5.1 MP4 is applied to your environment:
1.  Perform a full backup.  
2.  If the touch file "NOSHMDUP" in the Workaround, listed above, was implemented, that touch file should be removed at this time from each Media Server.

If you have any questions or concerns about this issue, please contact Symantec Enterprise Technical Support.   To contact Symantec Enterprise Technical Support, refer to the following link:  http://support.veritas.com/phonesup/phonesup_ddProduct_.htm
 

Symantec strongly recommends the following best practices:
1.  Always perform a Full backup prior to and after any changes to your environment.
2.  Always make sure that your environment is running the latest version and patch level.

If you have not received this TechNote as a Software Alert, from the Symantec Email Notification Service for Software Alerts, please subscribe at the following link:    http://maillist.support.veritas.com/subscribe.asp

If you would like to continue to receive updates regarding this particular issue, you may subscribe to the reference document. Symantec recommends subscription to ensure that you always have the latest information available for your immediate review. To initiate a subscription, click on the "Subscribe to this document" link in the document itself or visit the following URL:    http://maillist.support.veritas.com/notification_services.asp?doc=277732





Legacy ID



277732


Article URL http://www.symantec.com/docs/TECH42496


Terms of use for this information are found in Legal Notices