General Storage Process Flow

Article:HOWTO84800  |  Created: 2013-06-13  |  Updated: 2013-06-13  |  Article URL http://www.symantec.com/docs/HOWTO84800
Article Type
How To



The intent of this document is to provide a general overview of the storage process flow of an item when archived by Symantec Enterprise Vault (EV).

General Process:

1. Connection is made to the container which contains the message objects. (Exchange User Mailboxes, Exchange Journal Mailboxes, Domino NSFs). 
2. The item is given a Saveset value within the Saveset Table, which includes the item's Transaction ID, which is unique to the message object.
3. Item is written to Storage.
4. The  unique Transaction reference (TransactionID) is added to the JournalArchive and WatchFile tables, within the Vault Store Database (VSDB).
5. A 'hash' is generated for the item and a reference pointer is inserted into the FingerPrint Database (FPDB) for Single Instance Storage (SIS).
6. The Item and the item's related SIS Part references are considered Secured, typically after a successful backup.
7. Once secured, the original item is 'post processed' (Item is removed or a shortcut of the item is generated).

The Vault Store 'Remove safety copies' settings, Partition Storage type and the individual Partition Collection tab settings will determine how the item is treated by storage and within the Vault Store Database (VSDB) and Fingerprint Database (FPDB):

a. Storage Types (Differences between NTFS and Centera)

i. NTFS :

-   The item is separated into individual SIS parts (DVS file, DVSSP file(s) and DVSCC File(s))
    SISParts include : Message Header, Message Body, individual attachments..

-   Each piece is referenced through the Fingerprint database (FPDB) member tables for Single Instance Sharing (SIS).

ii. Centera :

- The item is kept as a single file (DVS file)

- The item is separated (as 'blobs') on the Centera level where EMC will perform its own sharing if configured within the Partition properties. 
   Since sharing is performed at the device level, 'fingerprinting' of the item is not preformed and the FPDB is not utilized.

b. Vault Store 'Remove safety copies' settings

i.  NTFS :

After backup
* Item is set to Pending, which does not change the content of the message.

1. An entry is added to the JournalArchive and WatchFile Tables in the VSDB.

The following records are added:
        JournalArchive : IndexCommited = 0 and BackupComplete = 0
        Watchfile           : ItemSecured = 0

2. Entries are added to the Fingerprint Database to track the ItemSecured level for the individual SIS Parts of the message.

3. After the backup of the NTFS partition completes, and EV is moved out of Backup Mode:

a. Fingerprint item references are updated for the backed up partition to mark the SIS Parts as Secured.
b. JournalArchive.BackupComplete is set to 1.

NOTE: JournalArchive.BackupComplete is only set to 1 once all Shared VSDB's that may have SIS Parts in the VSDBs for the referenced DVS item, in the JournalArchive table, have all had their SIS Parts set to secured in the FPDB.

4. Once the item is added to the archive's index, IndexCommited is set to 1.

5. Once the BackupComplete is set to 1, and the Server is set to a Backup state and back, the Watchfile.ItemSecured is set to 1.

6. The item is now considered stored and backed up and the item is sent for post processing. 

Note: Indexing of an item is not required for the post processing of the item to proceed. 

 

After backup (immediate for Journaling)
Note: See above for 'After backup'.  Behavior changes if identified as a Journal mailbox.
* Item is set to Pending, which does not change the content of the message.

1. An entry is added to the JournalArchive and WatchFile Tables in the VSDB.

The following records are added:
        JournalArchive : IndexCommited = 0 and BackupComplete = 0
        Watchfile           : ItemSecured = 0

2. Entries are added to the Fingerprint Database to track the ItemSecured level for the individual SIS Parts of the message and set to secured.

3. The Watchfile.ItemSecured entry is set to 1

4. Once the item is added to the archive's index, IndexCommited is set to 1

5. Once EV has a successful response from storage that the item has been written to the device, the item will be post processed.

Note: Indexing of an item is not required for the post processing of the item to proceed. 

Immediately after archive
Note: See above for 'Immediately for Journaling'. 

ii.  Centera :

After backup
* Item is set to Pending, which does not change the content of the message.

1. An entry is added to the JournalArchive and WatchFile Tables.

The following records are added:
        JournalArchive : IndexCommited = 0 and BackupComplete = 0
        Watchfile           : ItemSecured = 0

2. EMC returns that the item has been stored to the Primary node AND replicated to the Replica node, BackupComplete is set to 1.

Note: While the item is in Staging area (Collections) BackupComplete will be 0.

3. Once the item is added to the archive's index, IndexCommited is set to 1

4. Once the BackupComplete is set to 1, the Watchfile.Itemsecured is set to 1.

5. The item is now considered stored and Backed up and the item is sent for post processing.

Note: Indexing of an item is not required for the post processing of the item to proceed. 

After backup (immediate for Journaling)
* Item is set to Pending, which does not change the content of the message.
Note: See above for 'After backup'.  Behavior changes if identified as a Journal mailbox.

1. Item is sent to the Primary Node. 

2. Centera returns that the item was stored successfully and provides EV a C-Clip reference.

- If Collections are not enabled, an entry is created in SavesetStore referencing the SavesetIdentity and C-Clip.
- The BackupComplete attribute for the saveset(s) is set to 1 after the item is stored to the Primary node.

3. The item will be post processed. 

Note: The 'Immediately for Journaling' and 'Immediately after archive' does not wait for replication of the item to the Replica Node.

Immediately after archive

Note: See above for immediately for Journaling.

 

c. Partition Collection Tab settings

i. NTFS :

1. When enabled, the scheduled process will trawl through the NTFS partition for items eligible for collection.

2. These items are then moved from their original locations to a CAB file.
    Note: Items will be separated to individual files (DVS, DVSSP and DVSCC)

3. The Saveset transactions in the Saveset table are given a CollectionIdentity value.

4. A new entry is added to the Collection table for the CollectionIdentity (RelativeFileName = location and name of CAB file).

ii. Centera :

1. When enabled, a Collections location, or temporary staging location is used.

2. The StorageFileWatch process will generate 10 Collector Threads which will scan this location for files.
Note: If a failure occurs against one of these Collection Threads, and the thread fails, no additional threads are generated until the Storage Service is restarted.

3. After StorageFileWatch determines 100 items are eligible(or 10 MB in size), a call is sent to the Centera API providing this data.

4. The CollectionIdentity is added to the Vault Store Collection Table.

5. The Saveset record(s) in the Saveset Table are updated with a CollectionIdentity.

6. Once EMC returns that the storage request is successful, Centera will provide EV with the associated unique C-Clip.

7. The Collection Table is updated with the C-Clip as the RelativeFileName.

 

Detailed example of process flow:

* Using Immediately after backup and with Centera Collections

1. The Archiving Task connects to the mailbox/nsf.
2. Items are determined if eligible to be archived.
3. The items are changed to Pending in the container and copies the item to the Staging/Collections area as a single DVS.
4. When Storage Service Starts, 10 EMC Centera Collector Threads are generated.
5. Housekeeping is performed on staging location to confirm items in Saveset Table with no CollectionIdentity/RelativeFileName are present physically in the Staging area.

Notes:
    a. If the Staging area is extremely backlogged (50GB+), the housekeeping process can take time to finish as more items are being added to staging which it is being scanned.
    b. If the Staging area does not decrease in size, this can be indicative of Housekeeping not completing. See Related Article TECH61544 for details on the EMCCenteraCollectionHousekeepFiles registry value.
    c. If there are 10 Storage Errors in the Event log referencing that the Collector thread stopped, even though Storage Service is running, the staging area is no longer being scanned. (Workaround: Restart Storage Service)

- When a discrepancy is found, the saveset entry is 'deallocated' where the DVS file or the Saveset entry is removed (Depending on which reference was missing).
- If a Transaction is deallocated, the original item is reverted back to the original message class and will be re-archived on the next trawl and placed to the the Staging area.

6. Once Housekeeping has finished scanning the entire Staging area, StorageFileWatch sends the DVS files in 100 item increments to Centera.

- An entry is created in the Collection Table for the CollectionIdentity and an Empty RelativeFileName
- The 100 Saveset Transactions in Saveset Table are updated with a CollectionIdentity value

7. EMC Centera returns a success or failure to store the items and reference them to a C-Clip.

- If the Storage request fails to Centera, a reattempt is made.

8. SQL Changes performed:

a. JournalArchive.BackupComplete is set to 1
b. Watchfile.ItemSecured is set to 1
c. Collection.RelativeFileName is set with the C-Clip value.

9. The item is considered stored to the device and EV will begin post process of item(s).

Note: Only when all the steps above are successful will EV send a successful storage response and the item(s) will be post processed.




Article URL http://www.symantec.com/docs/HOWTO84800


Terms of use for this information are found in Legal Notices