Setting the configuration task performance tuning feature for multiple I/O threaded mirror operations can cause data loss when using Storage Foundation for Windows (SFW)

Article:TECH87524  |  Created: 2009-01-14  |  Updated: 2010-01-18  |  Article URL http://www.symantec.com/docs/TECH87524
Article Type
Technical Solution

Product(s)

Environment

Issue



Setting the configuration task performance tuning feature for multiple I/O threaded mirror operations can cause data loss when using Storage Foundation for Windows (SFW)

Solution



Overview

Symantec has identified a critical issue with potential for data loss in Storage Foundation for Windows (SFW) when performing mirror operations.

The configuration task performance tuning feature of SFW allows you to set the number of I/O threads to perform mirror-related operations, such as creating a mirror or adding a mirror. This is configured in the Veritas Enterprise Administrator (VEA) console under Control Panel > Storage Agent > Task Settings > Configuration Task Performance Tuning.

By default all parameters are single-threaded, but under non-default settings a potential for data loss has been observed as a result of NTFS file system or SQL Server database corruption.

What products are affected?
This issue is present in the following versions:
  • Storage Foundation for Windows 5.0 GA
  • Storage Foundation for Windows 5.0 GA RP1a
  • Storage Foundation for Windows 5.0 GA RP2
  • Storage Foundation for Windows 5.1
Storage Foundation for Windows 5.1 AP1

How to determine if affected

The issue can manifest itself in one or both of the following ways:
  • NTFS file system driver logs a Windows System Event reporting that the file system is corrupt and unreadable, for example:
Event ID 55: The file system structure on the disk is corrupt and unusable.
and/or
  • SQL Server database consistency check (DBCC checkdb) fails to complete successfully (e.g reports Torn-pages), Event ID messages include:
Event ID 823: The operating system returned error <error> to SQL Server during a write at offset <offset> in file '<file>'
Event ID 824: SQL Server detected a logical consistency-based I/O error

Note:  Although not reported to Symantec, it is conceivable that the consistency of other applications (e.g. Microsoft Exchange databases) may be affected if mirrored using multiple I/O threads. It is therefore recommended that the fix below should be applied to any system that is mirrored using multiple threads.

Workaround:
Set all parameters on the Configuration Task Performance Tuning tab to 1 (single threaded).

Formal Resolution
A private fix to address this issue has been released and is attached below.

Note:  This fix has not been fully tested. It is recommended that the fix be evaluated in a test environment before implementing it in your production environment. When the fix is incorporated into a Storage Foundation for Windows maintenance release, the resulting Hotfix or Service Pack must be installed as soon as possible. Symantec Technical Services will notify you when the maintenance release (Hotfix or Service Pack) is available.

To resolve this issue, apply the patch linked to this article by clicking on the "Download Now" option.

WARNING: Remove any existing mirrors created using multiple I/O threads prior to applying the fix. Recreate the mirrors only after applying the fix.

Additional Patch Information
Note the following recommendations should you decide to apply this fix:
  • Perform a FULL system backup, including any existing application data before and after applying the fix
Ensure application data is consistent using the appropriate tools (e.g. Windows Chkdsk utility, Eseutil, SQL Database consistency check)
For SFW 5.0 GA and SFW 5.0 GA RP1a:  Ensure that your system is upgraded to 5.0 RP2 before applying the fix.


Best Practices:

Symantec strongly recommends the following best practices:  
  • Always perform a full backup prior to and after any changes to your environment.
  • Always make sure that your environment is running the latest version and patch level.
  • Perform periodic "test" restores.
Subscribe to technical articles.

How to Subscribe to Email Notification Service:

If you have not received this from the Symantec Technical Support Email Notification Service, please click on the following link and subscribe to Software Alerts, or to this article, ID

Attachments

1591716_323096.zip (9.3 MBytes)

Supplemental Materials

SourceETrack
Value1591716
DescriptionPossible data corruption when mirroring using multiple I/O threads


Legacy ID



323096


Article URL http://www.symantec.com/docs/TECH87524


Terms of use for this information are found in Legal Notices