A potential for data loss can occur when archiving to an EMC Centera Storage Partition with Collections enabled.

Article:TECH135807  |  Created: 2010-01-22  |  Updated: 2014-11-17  |  Article URL http://www.symantec.com/docs/TECH135807
Article Type
Technical Solution

Product(s)

Issue



 A potential for data loss can occur when archiving to an EMC Centera Storage Partition with Collections enabled.


Error



If the FP_OPTION_STREAM_STRICT_MODE Environment Variable is set to 0 the following Event may be seen when attempting to add an item to the Index or attempting to Verify using EVSVR:

Event Type:             Error
Event Source:          Enterprise Vault 
Event Category:       Storage Crawler 
Event ID:  7083
Description: Retrieval of saveset failed. 
Archive Id: 17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local 
Transaction Id: A154921088BD807C0DC4D08657679581 
Extended information: Saveset decompression failed.   Error:17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local   Context:A154921088BD807C0DC4D08657679581      [0xc0041aa7] 
Internal Ref: Vault/RI 

If either condition above is met, follow the steps outlined below in the To Determine the Scope section to generate a log file that will list the affected items.
Event Type:             Error
Event Source:          Enterprise Vault
Event Category:       Storage File Watch
Event ID:  6760
Description:
Error from EMC Centera FPLibrary
Function call: FPStream_Close(-)
Status: FP_STREAM_BYTECOUNT_MISMATCH_ERR (-10045)
Reason: CloseProc: mStreamLen
Pool Address: X.X.X.X


Event Type:             Error
Event Source:          Enterprise Vault
Event Category:       Storage File Watch
Event ID:  6952
Description:
A EMC Centera Collector thread encountered an error and has stopped.

Reason: Error from EMC Centera FPLibrary   Function call:    Status:  ()   Reason:    Pool Address:       [0xc0041a68]
PartitionEntryID: 17AF0EF2E2F4C324EA6F6DF5ED087B2391q10000evsite.domain1.local
Method: CCollectorEMCCentera::CollectStore

The error message will be generated for each thread that faults, resulting in degradation of Collection performance.  There are a total of ten collector threads processing at one time, once all ten threads have faulted the Collection process will stop. If the above errors are generated and disabling collections is not an option refer to the article:

Workaround:
If the hotfix in the Formal Resolution section cannot be installed, the following workaround must be applied immediately.  

Remove the Environment Variable FP_OPTION_STREAM_STRICT_MODE from all Enterprise Vault servers running the Enterprise Vault Storage Service.

 

a. Open the Windows System Properties.
b. Click on the Advanced tab, then Environment Variables.
c. Click on FP_OPTION_STREAM_STRICT_MODE under System variables and click Delete.
 
Symantec Corporation recommends to disable Collections (See 'How to disable Collections on the Centera partition') due to the Collector thread faulting which may generate the following errors in the Event Viewer log.

Cause



How to remove the affected item from the Collections folder on the Enterprise Vault (EV) server to resolve Event ID 6760.
http://www.symantec.com/docs/TECH65517


How to disable Collections on the Centera partition 

a. Stop the archiving task that is archiving to the Centera Collections folder.
b. Confirm the collections folder contains zero files. Folders may exist in the collections folder but there cannot be any files within that folder.
c. If files exist, allow Collections to process those files.
d. Open the Vault Admin Console and expand Vault Store Groups.
e. Click the Vault Store that contains the Centera partition.
f.  In the right hand pane right-click the Vault Store Partition and click Properties.
g. On the Collections tab uncheck 'Enable collection'.
h. Restart the Enterprise Vault Storage service and then start the archiving task.
 
There is a potential for data loss with the known affected Enterprise Vault versions outlined below, in the What is Affected section, when configured with the following:
  • Data is archived to an EMC Centera storage partition.
  • Collections are enabled on the partition.
  • The following Windows System Environment Variable is set to 0 (disabled):
FP_OPTION_STREAM_STRICT_MODE
 
This issue occurs when a saveset of a specific size (exactly 256K or multiples of) causes any subsequent savesets to not have their streams appended to the blob even though entries are recorded in the Clip.  This results in the actual size of the data being less than the total referenced saveset size in the clip.


What is Affected
This issue is known to exist only when the following conditions have been met:
  • One of the following versions of Symantec Enterprise Vault is installed:
Enterprise Vault for File System Archiving (FSA) 8.x
Enterprise Vault for Lotus Domino 8.x
Enterprise Vault for Microsoft Exchange 8.x
Enterprise Vault for Microsoft SharePoint 8.x
Enterprise Vault for SMTP 8.x
  • Archiving to an EMC Centera partition that is configured to use Collections
  • The Environment Variable FP_OPTION_STREAM_STRICT_MODE exists on the server that is running the Enterprise Vault Storage Service

How to Determine if affected
There is a potential for data loss to occur if all the following conditions are met:
  • Archiving to an EMC Centera partition that is configured to use Collections.  (See 'How to check if Collections is enabled')
  • The Environment Variable FP_OPTION_STREAM_STRICT_MODE exists on the server that is running the Enterprise Vault Storage Service. (See 'How to check if FP_OPTION_STREAM_STRICT_MODE Environment Variable exists')
 
How to check if "Collections" is enabled
1. Open the Vault Admin Console and expand Vault Store Groups.
2. Click the Vault Store that contains the Centera partition.
3. In the right hand pane right-click the Vault Store Partition and click Properties.
4. On the Collections tab check if 'Enable collection' is enabled. (Figure 1)
 
Figure 1
 
 
How to check if FP_OPTION_STREAM_STRICT_MODE Environment Variable exists
1. Open the Windows System Properties.
2. Click on the Advanced tab, then Environment Variables.
3. Check under System variables for the Variable:
FP_OPTION_STREAM_STRICT_MODE
 


To Determine the Scope

Using EVSVR to scan a Vault Store Partition:

1. Install the Hotfix in the Formal Resolution section of this TechNote.

2.  Open EVSVR.exe from the Enterprise Vault  program directory (i.e C:\Program Files\Enterprise Vault).

3.  At the EVSVR prompt type "Edit" and hit Enter.

4.  In the EVSVR Operations GUI, select the Centera Partition to scan.

5.  Under Operations to Perform, choose Verify, ArchiveObjects, and SavesetValid.

6.  Click "Save As" to save the settings as a .xml file.

7.   At the EVSVR prompt type "Load".

8.  Browse to the location of the xml file previously saved and click "Open".

9.  At the EVSVR prompt type "Start".

Note:  There is no progress indicator for EVSVR, when the scan is complete the EVSVR window will display the message "processing has stopped".

10.  Once complete, view the EVSVR log file (located by default under \Enterprise Vault\Reports\EVSVR) and look for an entry like the following to determine the items that are affected:

Collection Identity: 2365, DVS RelativeFileName: PF0OAC1O7R5M5eE1D7CUSI78ESOG4320TEMSND0N487E285728R6A
Decompression failed [0xc0041aa7]                                                                      

11. To verify if there is data loss, the BLOB data must be extracted from the Centera Clip using the instructions in this document:
  http://www.symantec.com/docs/TECH65517

12.  Use EVSVR to extract the contents of the BLOB to a folder:
   a. Open EVSVR.exe from the Enterprise Vault program directory (i.e C:\Program Files\Enterprise vault).
   b. At the EVSVR prompt type "Interactive" and hit Enter.
   c. At the prompt use the following syntax to specify the Vault Store or Archive ID and the Saveset or Transaction ID:

ExtractSaveset <path to blob file> [-o <OutputFolder>]
<OutputFolder> - The folder to put retrieved files into.

Example:
EVSVR Command> ExtractSaveset C:\folder\PF0OAC1O7R5M5eE1D7CUSI78ESOG4320TEMSND0N487E285728R6A.Blob45 -o C:\Test

Note:
The blob file name proceeds the Decompression failed [0xc0041aa7] error in the EVSVR log file and is given a file extension when extracting from JCAS.

After EVSVR completes, the extracted files will be found in the output folder specified.

13.  Open the .CDF file that was generated when following the instructions in Step 11.

14.  There should be a dvs in the output folder for every SavesetId listed in the CDF.  If there is not a dvs file for each SavesetId then data loss has occurred.
 
Alternatively view the CDF file and confirm whether the sum of the size of all the eclipblob entries are greater than the last SavesetDataOffset value when using the option for Device Level Sharing.
If Device Level Sharing is not enabled then the single eclipblob size entry is greater than the last SavesetDataOffset.

Example:
 <SymantecCollectionV1.5SavesetInformation VaultId="17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local" SavesetId="201006237310342~201006231624290000~Z~A15492BE4D0D3E1AE461D78C245EA6B1" ShareType="0" SavesetDataOffset="1496874" SavesetDataSize="12922" />
 <SymantecCollectionV1.5SavesetInformation VaultId="17BE0A710B6D1154E8762890BABC129251110000evsite.domain1.local" SavesetId="201006237310361~201006231624300000~Z~A158347BD1431FAC577B07C782A98451" ShareType="0" SavesetDataOffset="1509796" SavesetDataSize="12764" />
 </SymantecCollectionV1.5SavesetsInformation>
 <SymantecCollectionV1.5SavesetsData>
     <eclipblob md5="2S75BQ7IR52NIx2SM3MAN4F6VSFG5C78HI504D7O3HEDGOUOV6QFG" size="363733" offset="0" />
 </SymantecCollectionV1.5SavesetsData>

If the SavesetDataOffset size plus the SavesetDataSize size is greater than the eclipblob size data loss has occurred.  From the example above,

SavesetDataOffset = 1509796
SavesetDataSize = 12764
eclipblob size should be = 1522560

The actual eclipblob size is 363733 which means data loss has occurred.

Solution



This issue has been addressed in the following release:
 

Enterprise Vault 8.0 SP4 - Release Details
http://www.symantec.com/docs/TECH76851

 


Supplemental Materials

SourceETrack
Value2066441
Description

FP_OPTION_STREAM_STRICT_MODE can result in dataloss if disabled resulting in a Collections backlog



Legacy ID



356790


Article URL http://www.symantec.com/docs/TECH135807


Terms of use for this information are found in Legal Notices