Which versions of Symantec Enterprise Vault (tm) can this Hotfix be applied?
 

This Hotfix can only be applied to the following versions of Enterprise Vault:
 

 

Enterprise Vault 8.0 SP4, Build 1991
 

 

What issues does this hotfix resolve?

 

ETrack 1506974
 

An updated version of the diagnostic utility that now supports Enterprise Vault V8 Savesets can be used to identify all the duplicate Savesets and then delete them leaving only the last archived instance from each set of duplicate Savesets. Further to the duplicate cleaner the shortcut processing has been enhanced to attempt to fix up shortcuts that are "broken" by the EVDuplicateCleaner tool.
 

This functionality is optional and can be activated via a regkey.
 

 

Etrack 1846761/1872448/1872925/1808828
 

Several SQL Stored procedures remedy the observed SQL performance issues and allows for missing folders to be recreated if necessary (refer to step 7 in the installation steps).
 

 

Etrack 1978274/1978277
 

EVSVR now correctly determines that CIFS Savesets have been migrated to Centera where files containing these CIFS Savesets are still found in the CIFS partition.
 

 

Etrack 1976971
 

Enhancements for handling items on legalhold during Dumpster expiry processing.
 

 

Etrack 1984396
 

- Ensure that when a saveset is recombined, missing attachments are ignored. This will allow the item to be indexed and also to be retrieved.
 

- Prevent items from being archived if any of the attachments are not stored properly.
 

 

Etrack 2002124
 

If DWORD: HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Storage\EnableMissingIndexRecipientEvent is set to a non-zero value, an event will be generated if the number of recipients passed to indexing is less than the number of recipients on the original item when items are ingested using the CM API.
 

Event Type: Error
 

Event Source: Enterprise Vault
 

Event Category: <variable>
 

Event ID: 7227
 

 

Etrack 2002354
 

If actual content is missing because a Centera blob is missing then attempt to reconstruct attachment by inserting converted content in instead.
 

 

Etrack 2034460
 

1. Enhancements for checking the Centera blob sizes to prevent unnecessary memory allocations when saveset recombination encounters missing attachment blobs. Using EVSVR to retrieve the saveset with missing attachments from the Centera via "DumpSaveset" command will no longer appear to hang but will now complete with an indication that it failed to get the recombined saveset (due to fingerprint validation failure). The retrieved saveset will not be available in the specified EVSVR Output folder Recombined sub-folder. However the recombined saveset (including the converted content for the missing attachment blob) will be available in the EVSVR Output folder Parts sub-folder.
 

 

2. Enhancements to allow for retrieving Lotus Notes item attachments that were originally archived to a Centera Vault Store partition archive with missing attachment blobs. For missing attachments the associated converted content will (if available) be retrieved instead.
 

 

Etrack 2045764
 

Enhancements for supporting Delegate sender types when processing SMIME signed emails.
 

 

Etrack 2067158
 

Recalling large files when using an EMC Centera storage device using Enterprise Vault (EV) File System Archiving (FSA) can cause the StorageOnlineOpns process to increase in memory usage. This causes the StorageOnlineOpns process on the EV server to increase in memory usage until recalls will no longer process properly. The following events can be seen in the EV Event Log on the EV server.
 

 

Event ID: 6882 Unable to complete retrieval request | |Reason:| Not enough storage is available to complete this operation.
 

Event ID: 6621 An exception has occurred. |Type: CStoreAccessor::RetrieveLargeFileBytes |Reference: /e |Reason: Exception occurred. [0x80020009] |
 

 

Etrack 2066441
 

We make sure that if a saveset being collected is 256KB (or multiples of) in size then we write that whole file to the clip and move onto the next saveset.
 

 

Etrack 2085909
 

Enhancements for handling MAPI message items with invalid Original Message Class MAPI custom property value types.
 

 

Which file(s) does this Hotfix replace?

 

 

AgentsCommon.dll	8.0.4.2020
ArchServer.dll	8.0.4.2020
AutoStorageOnlineOpns.dll	8.0.4.2020
EVDuplicateCleaner.exe	8.0.4.2020
EVDuplicateCleaner.xml
EVNoteStream.dll	8.0.4.2020
EVStgAPI.dll	8.0.4.2020
evsvr.exe	8.0.4.2020
EVSVRDLL.dll	8.0.4.2020
ExchangeItemsU.dll	8.0.4.2020
Hotfix_2069692-DirectoryUpdate.sql
Hotfix_2069692-VaultStoreUpdate.sql
IndexServer.exe	8.0.4.2020
StorageDelete.exe	8.0.4.2020
StorageOnlineOpns.exe	8.0.4.2020
StorageOnlineOpnsps.dll	8.0.4.2020
StorageRuntime.dll	8.0.4.2020
VaultStoreDB_DuplicateTransaction.sql
VaultStoreDB_EVSVR.sql
XMLDataStream.dll	8.0.4.2020

 

 

How to install this Hotfix?

If experiencing the issue referenced in article 357007 in the Related Documents section apply the cumulative hotfix then apply the hotfix in article 357007.


NOTE: CUSTOMERS MUST TAKE A BACKUP OF THE ENTIRE ENVIRONMENT (All databases and file system data) PRIOR TO APPLYING THE HOTFIX

1. On each Enterprise Vault server, stop all Enterprise Vault Services and verify that all EV related processes have stopped by using Windows Task Manager to confirm this. Take note especially that all Storage related processes have stopped, such as StorageServer.exe and StorageFileWatch.exe

2. On each Enterprise Vault Server take a backup of the following files (if present) and deposit the backup in a separate location (not under the Enterprise Vault installation path):

   AgentsCommon.dll
   ArchServer.dll
   AutoStorageOnlineOpns.dll
   EVNoteStream.dll
   EVSVRDLL.dll
   EVStgAPI.dll
   evsvr.exe
   ExchangeItemsU.dll
   IndexServer.exe
   StorageDelete.exe
   StorageOnlineOpns.exe
   StorageOnlineOpnsps.dll
   StorageRuntime.dll
   XMLDataStream.dll

   On EnterpriseVaultDirectory Database and any Vaultstore Database make a copy of the following Stored Procedures:
EnterpriseVaultDirectory Database
=================================
UpdateArchiveFolder
UpdateArchiveFolderPath
UpdateDescendentArchiveFolderPaths

VaultStore Database
===================
uspcu_Vault
uspi_Saveset
uspu_Saveset_JournalUpdate


3. On each Enterprise Vault Server copy the following files from the hotfix to the Enterprise Vault installation directory:

   AgentsCommon.dll
   ArchServer.dll
   AutoStorageOnlineOpns.dll
   EVNoteStream.dll
   EVSVRDLL.dll
   EVStgAPI.dll
   evsvr.exe
   ExchangeItemsU.dll
   IndexServer.exe
   StorageDelete.exe
   StorageOnlineOpns.exe
   StorageOnlineOpnsps.dll
   StorageRuntime.dll
   XMLDataStream.dll
   VaultStoreDB_EVSVR.sql

   Files for SQL
   =============
   Hotfix_2069692-DirectoryUpdate.sql
   Hotfix_2069692-VaultStoreUpdate.sql

4. Register the StorageOnline server as follows (run from the Enterprise Vault directory):
StorageOnlineOpns /regserver

5. Register EVStgAPI.dll COM interface changes and proxy stub dlls as follows (on 32-bit systems):
C:\windows\system32\regsvr32.exe "C:\Program Files\Enterprise Vault\EVStgAPI.dll"
C:\windows\system32\regsvr32.exe "C:\Program Files\Enterprise Vault\StorageOnlineOpnsps.dll"
  C:\windows\system32\regsvr32.exe "C:\Program Files\Enterprise Vault\AutoStorageOnlineOpns.dll"
(Note: on 64-bit systems, use C:\windows\SysWOW64\regsvr32.exe "...")

6. Run the Hotfix_2069692-DirectoryUpdate.sql script against the Directory database.
7. Run the Hotfix_2069692-VaultStoreUpdate.sql against each vault store database
- In order to activate folder creation, alter the call to uspcu_Vault by setting the input variable @CanCreateNewVault to 1
- This line is marked by the tag "<EDIT>"
- Deactivate folder creation once the desired folders have been created by setting the input variable @CanCreateNewVault to 0.

8. Restart all Enterprise Vault services and tasks.

9. To uninstall the fix, stop all the Enterprise Vault services, restore the original files, repeat steps 4 & 5 above.

10. Restore the following SQL Stored Procedures:

EnterpriseVaultDirectory Database
=================================
UpdateArchiveFolder
UpdateArchiveFolderPath
UpdateDescendentArchiveFolderPaths

VaultStore Database
===================
uspcu_Vault
uspi_Saveset
uspu_Saveset_JournalUpdate

11. Restart all the services and tasks.


For EVSVR Usage:
================
**The following instructions represent an outline only and does not constitute a comprehensive recovery procedure that will cater for every scenario resulting from the use of EVSVR.**
1. Place the entire Vault Store Group(s) being repaired into BACKUP MODE.
** Failure to do so can affect the running of EVSVR and result in serious data loss**

If the Enterprise Vault services are not currently running then in order to place the Vault Store Group being repaired into BACKUP MODE then ONLY the Enterprise Vault Admin Service and Enterprise Vault Directory Service should be started.  DO NOT START the Enterprise Vault Storage Service.


2. With the entire Vault Store Group being repaired still in BACKUP MODE, restart ONLY the following Enterprise Vault Services needed for the EVSVR repair operation:

   Enterprise Vault Admin Service
   Enterprise Vault Directory Service
   Enterprise Vault Indexing Service (and all Enterprise Vault Indexing Services that relate to the archives in the Vault Store Group being repaired).
   * typically the Enterprise Vault Storage Service is not required and should therefore not be started

3. With the entire Vault Store Group being repaired still in BACKUP MODE, perform the EVSVR operation Repair>RecreateMissingDBReferences. If you use deferred indexing (FSA), have a backlog of index operations outstanding on any archives, or any index rebuilds currently running, you may wish to deselect the "Require Index Entries" checkbox, as this checkbox controls whether database records are repaired based on the existence of index entries for the items.

4. With the entire Vault Store Group being repaired still in BACKUP MODE, after successful completion of the EVSVR operation Repair>RecreateMissingDBReferences repair operation, now perform the EVSVR operation Verify>Complete.

   Any errors detected should be investigated. Depending on the nature of the errors it may be advisable to contact Support before proceeding.
   
5. With the entire Vault Store Group being repaired still in BACKUP MODE, consider removing some or all records from the relevant Vault Store database WatchFile and WatchSISPartFile tables depending on how old the restored copy of the Vault Store database was and whether or not Collections or Migrations are enabled. If the possibility exists that some records in the restored WatchFile table have since then been collected then it is recommended that the WatchFile and WatchSISPartFile records be removed. For further information, contact technical support.

6. All archive pending items in mailboxes will need to be cancelled and reverted to normal state.

7. Only when the state of the repaired databases have been backed up and their state is deemed to be acceptable should the remaining Enterprise Vault services be started and the system taken out of BACKUP MODE.

Known limitations of EVSVR
==========================
While every effort has been made to recover content from the converted content, certain document types do not lend themselves to recovery so well.
For example: Excel spreadsheets cannot be reliably recovered as the converted content contains only a "preview" of the spreadsheet.

EVDuplicateCleaner - Additional installation steps and Usage
============================================================

Files for SQL
=============
VaultStoreDB_DuplicateTransaction.sql

Files to be installed in Enterprise Vault Installation Directory
================================================================
EVDuplicateCleaner.exe
EVDuplicateCleaner.exe.config


Installation Instructions
=========================
1. Close Outlook (if running) on the EV server.
2. Stop all Enterprise Vault services.
3. Backup the original versions of the files listed above if they exist.
4. Copy the supplied files listed above into the EV installation folder.
5. Duplicate Cleaner tool:
- Ideally this utility (EVDuplicateCleaner.exe) should be run from a storage server.
However it can in theory also run from any client machine as long as ECM API related and other dependent assemblies are present and that client machine has connectivity to the SQL server hosting Directory and Vault Store databases.
- Edit the EVDuplicateCleaner.exe.config setting the configurations values as detailed in the following section.  
- Run the SQL script provided (VaultStoreDB_DuplicateTransaction.sql) with the hotfix ON EACH VAULTSTORE DATABASE TO BE PROCESSED.
6. Restart the Services.
- The Duplicate Cleaner utility can be run from the command prompt.

Execution/running Instructions for the Duplicate Cleaner tool
===============================================================

IMPORTANT: *** THIS UTILITY MUST BE RUN UNDER THE VSA ACCOUNT.***

The following are the various settings (in EVDuplicateCleaner.exe.config) that should be correctly configured before running this utility.
1. DirDBSQLServer - Mandatory, the name of the SQL Server hosting the Directory database.
2. MaxAllowedFailures - Maximum number of errors after which it will stop processing (default = unlimited) .
3. SavesetChunkSize - Number of duplicate savesets to fetch in one call to directory database (default = 10000).
4. ErrorToLogFile - Do we need to log errors to output log file (default = yes).
5. MaxDuplicateSavesetsToReport - Maximum number of duplicated savesets that will be processed in 'Report Mode' (default = 100000).
6. TryToOverrideDumspter- If dumpster is enabled then this decides if we want items to be hard deleted (i.e. not going to dumpster but deleted directly). This override will work only from V75 onwards. (default = true)
7. SkipLegacySavesets - If there are no legacy savesets then set this to True to skip even bothering looking for them.

The following are the supported commands for running this utility.

EVDuplicateCleaner.exe Execute[E,Ex,Exe,...,Executiv] <VaultStoreName> [<ArchiveEntryId>]
EVDuplicateCleaner.exe Report[R,Re,Rep,...,Repor] <VaultStoreName> [<ArchiveEntryId>]
EVDuplicateCleaner.exe Summary[S,Su,Sum,...,Summar] <VaultStoreName> [<ArchiveEntryId>]

Details of various commands
1. Execute - In this mode EVDuplicateCleaner.exe will delete all duplicate savesets. VaultStoreName is mandatory, ArchiveEntryId may also be provided in order to process one single archive.
2. Report - The output log will contain a list of duplicate savesets (by default maximum 100000) and a count of how many duplicate items each one has.
3. Summary - Will just output summary of duplicates per archive based on the supplied inputs.


Limitations of the Duplicate Cleaner tool
===========================================

1. This utility uses the following attributes of a saveset to identify duplicates

Legacy (pre-Version 8.0) savesets:
ArchivePointIdentity, IdPartition, VaultIdentity (folder identity), IdChecksumHigh, IdChecksumLow, IdDateTime (creation date), IdUniqueNo
So the items which may have same binary content may not be treated as duplicate if they are not on the same partition, not in the same folder or do not have the same unique number.

Enterprise Vault Version 8.0 savesets:
ArchivePointIdentity,VaultIdentity (folder identity), IdDateTime (creation date), IdUniqueNo, SisPartCount
In addition to the above, each item's content fingerprint is checked. Items with different content fingerprints will not be treated as duplicate.
It should be noted that due to the reduced number of attributes that can be used from within the Vaultstore Database for the V8 savesets when determining the list of possible duplicate savesets there is a higher possibility for false positives to occur. Actual processing of the duplicate items will potentially result in less items being removed once the validation of the fingerprint value contained within the Index has been performed.

2. Maximum number of duplicate savesets processed in 'report mode' will be limited to value configured through 'MaxDuplicateSavesetsToReport' setting  
(default = 100000).

3. This utility will not delete duplicate items that are on legal hold or on other compliance devices or if for similar reasons (retention category) delete is prohibited.

4. Running the utility and removing duplicates may have the effect of temporarily "breaking" shortcuts in user mailboxes as the saveset id that is contained in the shortcut may be incorrect if it points to a now removed duplicate.
This can be fixed by using the modified Shortcut Processing - see below.

5. It may take the index for each archive processed a while to catch up if lots of duplicates are removed - so you may still "appear" to have lots of duplicates in the web app search even after running the tool. Once the index has caught up these will disappear.

Enhancements to Shortcut Processing (Duplicate Cleaner tool)
============================================================
!!BEFORE running this tool turn off the Journal mailbox task.!!

In certain conditions the duplicate removal tool may have the effect of breaking shortcuts.
After running EVDuplicateCleaner users may complain that they can no longer retrieve items from their mailbox shortcuts.
This does not affect archive explorer or the web search app, and shortcuts will still be expired according to site policy, however if users really require their shortcuts repaired then the following regkey can be used to activate the new "shortcut fixup" functionality:

Create the following registry value:
HKLM\SOFTWARE\KVS\Enterprise Vault\Agents\FixOrphanedShortcut  - DWORD; set to 1 to enable new function.

*******************************************************************************************************************************************
* BE SURE TO REMOVE THE REGVALUE ONCE THE MAILBOXES HAVE BEEN FIXED UP - otherwise mailbox processing performance may be adversely affected *
*******************************************************************************************************************************************

Also: if the mailboxes contain duplicates of meeting requests, be sure to add "IPM.Schedule*" to the message classes supported by the site and
then check the same message class in the policy for the mailbox in question.

This regvalue has the effect of FORCING the new shortcut processing to use more intensive checks for orphaned shortcuts.
If a shortcut is deemed to be "orphaned" before it is finally removed a check is made to see if a duplicate saveset could exist that this shortcut can be fixed to point to. If that is the case then the shortcut will be fixed up to point to the duplicate of the original saveset. If there is any doubt about the duplicate saveset then the shortcut will not be fixed.
If no duplicate can be found then the shortcut will be deleted as orphaned.

Report Mode:
------------
To see the effect of the new shortcut processing BEFORE actually performing shortcut procesing run the mailbox archiving task in "Report" mode against any mailboxes you wish to fixup.
A report will be generated per-mailbox in the Reports folder beneath the EV install folder. It details either what would happen or what happened during shortcut processing.