A potential for data loss can occur when archiving to an EMC Centera Storage Partition with Collections enabled.

Article:TECH135807  |  Created: 2010-01-22  |  Updated: 2010-01-22  |  Article URL http://www.symantec.com/docs/TECH135807
Article Type
Technical Solution

Product(s)

Environment

Issue



A potential for data loss can occur when archiving to an EMC Centera Storage Partition with Collections enabled.

Solution



Introduction
There is a potential for data loss with the known affected Enterprise Vault versions outlined below, in the What is Affected section, when configured with the following:
  • Data is archived to an EMC Centera storage partition.
  • Collections are enabled on the partition.
  • The following Windows System Environment Variable is set to 0 (disabled):
FP_OPTION_STREAM_STRICT_MODE

This issue occurs when a saveset of a specific size (exactly 256K or multiples of) causes any subsequent savesets to not have their streams appended to the blob even though entries are recorded in the Clip.  This results in the actual size of the data being less than the total referenced saveset size in the clip.


What is Affected
This issue is known to exist only when the following conditions have been met:
  • One of the following versions of Symantec Enterprise Vault is installed:
Enterprise Vault for File System Archiving (FSA) 8.x
Enterprise Vault for Lotus Domino 8.x
Enterprise Vault for Microsoft Exchange 8.x
Enterprise Vault for Microsoft SharePoint 8.x
Enterprise Vault for SMTP 8.x
  • Archiving to an EMC Centera partition that is configured to use Collections
  • The Environment Variable FP_OPTION_STREAM_STRICT_MODE exists on the server that is running the Enterprise Vault Storage Service

How to Determine if affected
There is a potential for data loss to occur if all the following conditions are met:
  • Archiving to an EMC Centera partition that is configured to use Collections.  (See 'How to check if Collections is enabled')
  • The Environment Variable FP_OPTION_STREAM_STRICT_MODE exists on the server that is running the Enterprise Vault Storage Service. (See 'How to check if FP_OPTION_STREAM_STRICT_MODE Environment Variable exists')

How to check if "Collections" is enabled
1. Open the Vault Admin Console and expand Vault Store Groups.
2. Click the Vault Store that contains the Centera partition.
3. In the right hand pane right-click the Vault Store Partition and click Properties.
4. On the Collections tab check if 'Enable collection' is enabled. (Figure 1)

Figure 1
 

How to check if FP_OPTION_STREAM_STRICT_MODE Environment Variable exists
1. Open the Windows System Properties.
2. Click on the Advanced tab, then Environment Variables.
3. Check under System variables for the Variable:
FP_OPTION_STREAM_STRICT_MODE

If the FP_OPTION_STREAM_STRICT_MODE Environment Variable is set to 0 the following Event may be seen when attempting to add an item to the Index or attempting to Verify using EVSVR:

Event Type:             Error
Event Source:          Enterprise Vault
Event Category:       Storage Crawler
Event ID:  7083
Description: Retrieval of saveset failed.
Archive Id: 17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local
Transaction Id: A154921088BD807C0DC4D08657679581
Extended information: Saveset decompression failed.   Error:17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local   Context:A154921088BD807C0DC4D08657679581      [0xc0041aa7]
Internal Ref: Vault/RI

If either condition above is met, follow the steps outlined below in the To Determine the Scope section to generate a log file that will list the affected items.


To Determine the Scope

Using EVSVR to scan a Vault Store Partition:

1. Install the Hotfix in the Formal Resolution section of this TechNote.

2.  Open EVSVR.exe from the Enterprise Vault  program directory (i.e C:\Program Files\Enterprise Vault).

3.  At the EVSVR prompt type "Edit" and hit Enter.

4.  In the EVSVR Operations GUI, select the Centera Partition to scan.

5.  Under Operations to Perform, choose Verify, ArchiveObjects, and SavesetValid.

6.  Click "Save As" to save the settings as a .xml file.

7.   At the EVSVR prompt type "Load".

8.  Browse to the location of the xml file previously saved and click "Open".

9.  At the EVSVR prompt type "Start".

Note:  There is no progress indicator for EVSVR, when the scan is complete the EVSVR window will display the message "processing has stopped".

10.  Once complete, view the EVSVR log file (located by default under \Enterprise Vault\Reports\EVSVR) and look for an entry like the following to determine the items that are affected:

Collection Identity: 2365, DVS RelativeFileName: PF0OAC1O7R5M5eE1D7CUSI78ESOG4320TEMSND0N487E285728R6A
Decompression failed [0xc0041aa7]                                                                      

11. To verify if there is data loss, the BLOB data must be extracted from the Centera Clip using the instructions in this document:
 http://entsupport.symantec.com/docs/315392

12.  Use EVSVR to extract the contents of the BLOB to a folder:
   a. Open EVSVR.exe from the Enterprise Vault program directory (i.e C:\Program Files\Enterprise vault).
   b. At the EVSVR prompt type "Interactive" and hit Enter.
   c. At the prompt use the following syntax to specify the Vault Store or Archive ID and the Saveset or Transaction ID:

ExtractSaveset <path to blob file> [-o <OutputFolder>]
<OutputFolder> - The folder to put retrieved files into.

Example:
EVSVR Command> ExtractSaveset C:\folder\PF0OAC1O7R5M5eE1D7CUSI78ESOG4320TEMSND0N487E285728R6A.Blob45 -o C:\Test

Note:
The blob file name proceeds the Decompression failed [0xc0041aa7] error in the EVSVR log file and is given a file extension when extracting from JCAS.

After EVSVR completes, the extracted files will be found in the output folder specified.

13.  Open the .CDF file that was generated when following the instructions in Step 11.

14.  There should be a dvs in the output folder for every SavesetId listed in the CDF.  If there is not a dvs file for each SavesetId then data loss has occurred.

Alternatively view the CDF file and confirm whether the sum of the size of all the eclipblob entries are greater than the last SavesetDataOffset value when using the option for Device Level Sharing.
If Device Level Sharing is not enabled then the single eclipblob size entry is greater than the last SavesetDataOffset.

Example:
 <SymantecCollectionV1.5SavesetInformation VaultId="17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local" SavesetId="201006237310342~201006231624290000~Z~A15492BE4D0D3E1AE461D78C245EA6B1" ShareType="0" SavesetDataOffset="1496874" SavesetDataSize="12922" />
 <SymantecCollectionV1.5SavesetInformation VaultId="17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local" SavesetId="201006237310361~201006231624300000~Z~A158347BD1431FAC577B07C782A98451" ShareType="0" SavesetDataOffset="1509796" SavesetDataSize="12764" />
 </SymantecCollectionV1.5SavesetsInformation>
 <SymantecCollectionV1.5SavesetsData>
     <eclipblob md5="2S75BQ7IR52NIx2SM3MAN4F6VSFG5C78HI504D7O3HEDGOUOV6QFG" size="363733" offset="0" />
 </SymantecCollectionV1.5SavesetsData>

If the SavesetDataOffset size plus the SavesetDataSize size is greater than the eclipblob size data loss has occurred.  From the example above,

SavesetDataOffset = 1509796
SavesetDataSize = 12764
eclipblob size should be = 1522560

The actual eclipblob size is 363733 which means data loss has occurred.


Formal Resolution:
This issue is formally resolved in the following Hotfix.  This hotfix will ensure all savesets regardless of size have all streams appended to the blob and recorded in the clip.  However, the hotfix will not be able to recover the items that were affected.

NOTE:  Hotfix 357554 below is not compatible with Hotfix 355829 and Hotfix 357607.  Symantec Corporation plans to address the hotfix conflict in the next service pack release.

Enterprise Vault 8.0 - Enterprise Vault 8.0 SP3
Upgrade to Enterprise Vault 8.0 SP4:    http://fileconnect.symantec.com
Download Hotfix:  http://entsupport.symantec.com/docs/357888

Enterprise Vault 8.0 SP4
Download Hotfix:    http://entsupport.symantec.com/docs/357888


Workaround:
If the hotfix in the Formal Resolution section cannot be installed, the following workaround must be applied immediately.  

Remove the Environment Variable FP_OPTION_STREAM_STRICT_MODE from all Enterprise Vault servers running the Enterprise Vault Storage Service.
a. Open the Windows System Properties.
b. Click on the Advanced tab, then Environment Variables.
c. Click on FP_OPTION_STREAM_STRICT_MODE under System variables and click Delete.

Symantec Corporation recommends to disable Collections (See 'How to disable Collections on the Centera partition') due to the Collector thread faulting which may generate the following errors in the Event Viewer log.

Event Type:             Error
Event Source:          Enterprise Vault
Event Category:       Storage File Watch
Event ID:  6760
Description:
Error from EMC Centera FPLibrary
Function call: FPStream_Close(-)
Status: FP_STREAM_BYTECOUNT_MISMATCH_ERR (-10045)
Reason: CloseProc: mStreamLen
Pool Address: X.X.X.X


Event Type:             Error
Event Source:          Enterprise Vault
Event Category:       Storage File Watch
Event ID:  6952
Description:
A EMC Centera Collector thread encountered an error and has stopped.

Reason: Error from EMC Centera FPLibrary   Function call:    Status:  ()   Reason:    Pool Address:       [0xc0041a68]
PartitionEntryID: 17AF0EF2E2F4C324EA6F6DF5ED087B2391q10000evsite.domain1.local
Method: CCollectorEMCCentera::CollectStore

The error message will be generated for each thread that faults, resulting in degradation of Collection performance.  There are a total of ten collector threads processing at one time, once all ten threads have faulted the Collection process will stop. If the above errors are generated and disabling collections is not an option refer to the article:

How to remove the affected item from the Collections folder on the Enterprise Vault (EV) server to resolve Event ID 6760.
 http://entsupport.symantec.com/docs/357068


How to disable Collections on the Centera partition
a. Stop the archiving task that is archiving to the Centera Collections folder.
b. Confirm the collections folder contains zero files. Folders may exist in the collections folder but there cannot be any files within that folder.
c. If files exist, allow Collections to process those files.
d. Open the Vault Admin Console and expand Vault Store Groups.
e. Click the Vault Store that contains the Centera partition.
f.  In the right hand pane right-click the Vault Store Partition and click Properties.
g. On the Collections tab uncheck 'Enable collection'.
h. Restart the Enterprise Vault Storage service and then start the archiving task.


Subscribe to this TechNote:
Subscribe to this TechNote for any updates that are made to this article, by clicking on the following link:
 http://maillist.entsupport.symantec.com/notification.asp?doc=356790


Symantec Strongly Recommends the Following Best Practices:
1. Always perform a FULL backup prior to and after any changes to your environment.
2. Always make sure that the environment is running the latest version and patch level.
3. Subscribe to technical articles for updates.


How to Subscribe to Software Alerts
If this TechNote was not received from the Symantec Email Notification Service as a Software Alert, please subscribe at the following link:
 http://maillist.entsupport.symantec.com/subscribe.asp

Supplemental Materials

SourceETrack
Value2066441
DescriptionFP_OPTION_STREAM_STRICT_MODE can result in dataloss if disabled resulting in a Collections backlog


Legacy ID



356790


Article URL http://www.symantec.com/docs/TECH135807


Terms of use for this information are found in Legal Notices