A potential for data loss has been discovered where data restored to a UNC CIFS share on an EMC Celerra appliance results in Status 0, but files restored will have incorrect size and may be corrupt/unusable.

Article:TECH60710  |  Created: 2008-01-02  |  Updated: 2013-10-24  |  Article URL http://www.symantec.com/docs/TECH60710
Article Type
Technical Solution

Product(s)

Environment

Issue



A potential for data loss has been discovered where data restored to a UNC CIFS share on an EMC Celerra appliance results in Status 0, but files restored will have incorrect size and may be corrupt/unusable.

Solution



Introduction:
A potential for data loss has been discovered where data restored to a UNC CIFS share on an EMC Celerra appliance results in Status 0, but restored files have incorrect size and may be corrupt/unusable.  The backup image is in tact, but the restore operation may restore files in a corrupt form due to this issue.

This issue occurs due to a permissions condition on the NAS device that prevents NetBackup from opening the file at a crucial point of the restore operation process to finish writing it.  This issue does not affect restores performed to a UNC path that target a Windows host.

Information on NTIO
NTIO is a Windows component that provides overlapped direct I/O operations.  NTIO pre-allocates a percentage of the file size in locked memory and often increases performance when used by NetBackup.  

In versions prior to NetBackup 6.0 MP6, the minimum file size for NTIO operation is 8MB.  Beginning with 6.0 MP6, NTIO now operates with all files 32KB and greater.


What is Affected:
The following NetBackup Client versions are affected when running on a supported Windows platform:
-  NetBackup Server/Enterprise Server 6.0 GA through 6.0 Maintenance Pack 7 (MP7)
-  NetBackup Server/Enterprise Server 6.5 through 6.5.2A

Versions prior to 6.0 are also affected by this issue.  Please note that these versions have reached the End of Engineering Support date and no future patches are planned for those versions.  It is recommended that affected users of these older versions apply one of the workarounds below, then upgrade to NetBackup 6.x as soon as the formal resolution is available.


How to Determine if Affected:
Data loss has been known to occur if ALL of the following conditions are true:
- The client is running one of the NetBackup versions listed above on a supported Windows Operating System.
- NTIO is enabled (default)
- Data restored from this client to a UNC CIFS share on an EMC Celerra appliance.
- The NetBackup Client service is not running with the DOMAIN\Administrator account (equivalents do not qualify)*.
- This restore includes files larger than minimum NTIO file size mentioned above, resulting in a restore of corrupt data.
- The corrupt data goes unnoticed and is required at a later time, but the required backup set(s) have become expired and are overwritten.

Note:  As mentioned above, it is important to note that if data is restored in a corrupt state due to this issue, then backed up again, the corrupt data in the backup set will not be able to be restored properly, even with a workaround in place.  

This issue has been observed when restoring data to an EMC Celerra appliance, but it may be possible for this to occur when restoring to a UNC path on other appliance types.  However, this has not been observed or reproduced.

* This issue has not been observed when running the NetBackup Client Service with the DOMAIN\Administrator account specifically.  However, a domain admin equivalent, or account that is a member of the domain administrator group other than "Administrator," will exhibit this problem when used for the client service if all other conditions are met.


Unaffected data:
The following files are not affected by NTIO.  Therefore, they are not affected by this issue:
- Encrypted files
- Offline files
- Reparse point files
- Sparse files


Formal Resolution:
The formal resolution to this issue is tentatively scheduled for inclusion in the following NetBackup releases:
- NetBackup Server/Enterprise Server 6.5.3
- NetBackup Server/Enterprise Server 6.0 Maintenance Pack 8 (MP8)

As releases become available, refer to the Related Documents section below, or visit the following Symantec Enterprise Support Web site:    http://www.symantec.com/business/support/downloads.jsp?pid=15143

Applying the formal resolution will prevent a restore operation from reporting a status 0 (successful) when files are restored with the incorrect size due to this issue; instead, restores affected by this issue will fail with Status 5 (the restore failed to recover the requested files) or Status 1 (the requested operation was partially successful).

Because this issue involves permissions, a workaround must still be followed to ensure that restores are performed correctly.  Implement one of the three workarounds listed in the "Workaround" section below to restore data correctly in an affected environment.  Symantec recommends implementing Workaround A, as this directly addresses the issue with the least impact to performance.  Even though a workaround allows data to be restored correctly, it is recommended to also apply a formal resolution if the environment is exposed to this issue, to prevent future restores of this type to a UNC CIFS share from incorrectly reporting a Status 0.

Please be sure to refer back to this document periodically, as any changes to the status of the issue will be reflected here. A link to the release update or patch download will be added to this document when it becomes available. Please note that Symantec reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Symantec's plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.


Workaround:
One of the following workarounds must be applied to properly restore data affected by this issue.  Symantec recommends Workaround A for the least impact to performance.

Workaround A
Execute the following steps on the Windows client performing the restore:
1. Determine the Domain Account that the NetBackup Client Service starts with:
 
In this example, the Domain is "INOV" and account name is "acct02"

2. Right-Click Computer Management (Local) and select Connect to another computer:
 

3. Specify to connect to the Celerra instance, in this example the host name is "Celerra01c"
 

4. Expand System Tools > Local Users and Groups > Groups and right-click on the group named Backup Operators and select Add to Group.
 

5. From the Select Users, Computer, or Groups window, inside the Enter the object names to select field, type in the domain and username of the account and click Check Names.
 

6. Make sure the name appears within the Backup Operators group Members list:
 

7. Restart the NetBackup Client Service so these changes can take effect.

8. Run a restore of a file via the CIFS share to the NAS device.   Ensure that the restored file size(s) are correct and that the data is intact.  


Workaround B
Disabling NTIO on the Windows client is another option to allow the backup sets to be restored as they were backed up.
To disable NTIO (which is enabled by default), a registry key and value must be created as follows:
1.  Run regedt32
2.  Create the following key:
HKEY_LOCAL_MACHINE\Software\VERITAS\NetBackup\CurrentVersion\Config\NTIO
3.  Under this key, create a DWORD value named "UseNTIO".
4.  Give this a value of "0".  A value of 0 disables NTIO.

Note: This procedure must be done for each affected client.  To resume NTIO usage, set the "UseNTIO" value to 1 or delete the NTIO key.  Ensure that NTIO is not enabled during any restores until a fix can be applied.  While NTIO is disabled, a moderate decrease in performance may be noticed.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly.  Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.


Workaround C
Direct the restore to a location that is not a UNC CIFS share (such as a local file system, or UNC on a Windows host) and ensure that the client service account has permissions to write to that destination.  This will work around the issue, but is recommended to perform Workaround A or B, or apply a formal resolution so the client in question is not left in a configuration where corrupt restores can still be performed in the future due to this issue.  



Best Practices:
Symantec strongly recommends the following best practices:
1. Always perform a Full backup prior to and after any changes to your environment.
2. Always make sure that your environment is running the latest version and patch level.



How to Subscribe to Software Alerts:
If you have not received this TechNote from the Symantec Email Notification Service as a Software Alerts, please subscribe at the following link:
 http://maillist.entsupport.symantec.com/subscribe.asp


Supplemental Materials

SourceError Code
Value1
DescriptionNetBackup Error 1: the requested operation was partially successful

SourceError Code
Value5
DescriptionNetBackup Error 5: the restore failed to recover the requested files

SourceETrack
Value1272570
DescriptionETrack (NetBackup) 1272570: restored file size is bigger then what was backed up.


Legacy ID



304383


Article URL http://www.symantec.com/docs/TECH60710


Terms of use for this information are found in Legal Notices