How the Enterprise Vault NTFS Collector works

Article:TECH63406  |  Created: 2008-01-18  |  Updated: 2013-12-03  |  Article URL http://www.symantec.com/docs/TECH63406
Article Type
Technical Solution

Product(s)

Environment

Issue



In order to improve backup performance, Enterprise Vault provides a Collector function which moves numerous archives objects into central collection files.  This document discusses how the Enterprise Vault NTFS Collector functions.


Solution



Overview:

The Enterprise Vault (EV) Collector, a component of the Storage File Watch process, is a file management software which is responsible for:
  • The collection of multiple saveset files (.DVS, .DVSSP and .DVSCC) into collection files (.CAB)
  • The recall of saveset files from collection files when archived items are retrieved
  • The deletion of temporarily recalled saveset files extracted from collection CABs
  • Managing collection files in response to deletion of items
Enabling Collections
 
The Collector function is enabled on each Vault Store Partition individually.  This is performed within the Vault Admin Console (VAC):
 
1. Within the VAC, expand the EV Site - Vault Store Groups.
2. Select the Vault Store in question.
3. Right-click on the Partition - Properties.
4. Select the Collections Tab.
5. Select the option 'Use collection files'.
 
Notes:
a. Once Collections are enabled for an NTFS Partition, this feature cannot be disabled.
b. Collections will decrease the total number of files on a partition, which will increase backup performance.
c. Native archived data on a partition are in a compressed format and moving the data into collection (CAB) files will not decrease the space used.
 
Once the EV Collector has been enabled for an NTFS vault store partition the Collector runs once per day, based on the configured time window.
 
The following options are configurable within the Partition Properties, Collections Tab.
  • Start at: Local time at which collecting will start (Default = 10:00 AM).
  • End at:  Local time at which collecting will finish (Default = 4:00 PM). EV will stop all collection threads at this time or when no more files are avaIlable to collect, whichever occurs first.
  • Maximum collection file size: Maximum size of the CAB file generated (Default = 10 megabytes)
  • Age at which files will be collected: Age of files on storage (Default = 10 Days)
Notes:
a. The 'Run Now' option within the Collections Tab will begin the collection process immediately, completing once no files are eligible to collect.
b. The 'Run Now' may be interrupted by a restart of the Enterprise Vault Storage Service.
c. The collection component utilizes the StorageFileWatch process to scan the partition for eligible work.
 
File collection

EV vault store partitions that are hosted on an NTFS storage location, organize the storage of saveset files based on the sent/received date of the item archived by Enterprise Vault. These saveset files are organized across a hierarchy of folders rather than into one single folder. 

The EV Collector parses the DAY (EV 7.5 and previous) or MONTH-DAY (EV 8.0 and later) directory of this structure and utilizes configurable rules to package the saveset files within the DAY directory into collection files (*.CAB files). 
 
Storage Structure
In EV 8.0 and above, the NTFS storage structure has changed:
 
EV 2007 (7.5) and previous
          \PartitionName\Year-Folder\Month-Folder\Day-Folder\Hour-Folder
(Format: \PartitionName\YYYY\MM\DD\HH)
 
EV 8.0 and later
\PartitionName\Year-Folder\Month-Day-Folder
(Format: \PartitionName\YYYY\MM-DD\
 
When Collections scan the NTFS partition for eligible items to Collect, the following is performed:
 
On the partition:
  • The created date of each DAY directory is scanned and compared to the current date on the local system to determine whether that directory may contain files old enough for collection. 
    • If the difference is greater than the “Age at which files will be collected” setting for the partition, this indicates to the collector that the files under the day directory may be eligible for collection.
    • The collector will begin to parse the files for the modified date attribute to determine eligibility for collection.
  • During the eligibility check, the modified date of the files are compared to the current date on the local system to determine if these files meet the criteria for collections.
    • If the difference is greater than the ‘Collect files older than’ setting for the partition, this indicates to the collector that the files are eligible for collection.
  • The collector will compress the eligible files into a collection (CAB) file which is placed in the day directory. The collection of the saveset files are based on the following rules:
    • During each daily run, the collector will add eligible saveset files to new collection (CAB) files.
    • If there are fewer than 15 files that are eligible for collection, the collector does not perform a collection of these files.
    • Per partition, there is a configurable, maximum size for each collection (CAB) file. The default size is 99 MB for EV 6.0 SP 2 and versions prior.  6.0 SP3 and higher default to 10 MB with a maximum of 99 MB.
    • When creating a collection, if one of these rules are met, the files to be collected will roll over into another collection (CAB) file. This process will result in the generation of multiple collection (CAB) files within a single day directory.
  • When a Collection file is created, the file name format is ‘Collection#.CAB’, where # is an incremented numerical identifier for each collection file within a partition.

Within SQL:

  • Archived items are referenced within the Vault Store Database, within the dbo.Saveset Table.  When an archived items is placed within a Collection, the numerical identifer assigned to the CAB file is assigned to the individual items within the Savset Table as the item's 'CollectionIdentity'.
  • The 'CollectionIdentity' value is then added to the Vault Store Database, within the dbo.Collections Table.  This table is then populated with the properties of the Collection file, including the location and name (RelativeFileName value)
  • The SQL components are referenced when a retrieval request is performed in order to identify the location of the file(s) which are required by the retrieval/restore process.

Collected item retrievals 

When an item is retrieved from NTFS storage, the following will be performed:

a. A check is performed within the Vault Store Saveset Table to identify the Partition where the item is located (PartitionEntryIdentity).
b. If a CollectionIdentity is recorded, a subsequent check against the Collections table is performed to identify the CAB location.
c. Storage will browse to the physical Partition location to attain the file(s) associated with the archived item in the following order:

i.  A check of the original storage location (prior to being collected) of the items in .DVS format.
ii. A check of the original storage location (prior to being collected) of the items in .ARCHDVS format.
iii.A check for the CAB location is performed, based on the RelativeFileName SQL entry for the associated CollectionIdentity.
iv. When the requested files are located within the collection file, the files are temporarily extracted to the file's original location as an .ARCHDVS file.

Note: When performing an export of an archive, this process can potentially cause the partition location to temporarily decrease in free space, as the files will be in two locations on the partition.

d. If utilizing the Migrator option to move older collections to an alternate location, an additional check for an *.ARCHCAB file in the original collection file location. If a the *.CAB or *.ARCHCAB file is not present, a request to retrieve the collection is performed.  This process will place the collection file in the original collection file location utilizing an *.ARCHCAB extension.

See the following articles for additional information on utilizing secondary storage Migration:

About configuring Enterprise Vault for collection and migration

Deletion of temporarily retrieved files

Items retrieved from collection files will be extracted to the original file location with an *.ARCH* preface to the extension (Ex. *.ARCHDVS, *.ARCHDVSSP,  *.ARCHDVSCC and *.ARCHCAB). When the Collection process scans each partition, a check is performed against each *.ARCH* file on the partition for the Last Accessed Date attribute.  If the Last Accessed Date attribute is greater than 24 hours from the current time, the Collector, using the StorageFileWatch process, will issue a delete of the 'expired' file(s).

Managing collection files due to deletions

The EV Collector process is responsible for permanently removing archived data from collection files after expiry or user deletions are performed.  Initially when a collected archived items is deleted, the physical data will still be retained within the physical collection file until the collection is analyzed by the Collector process.  This function is called the Sparse Collections process. See the following article for details on the Sparse Collections process:

How do Sparse Collections and the SparseCollectionPercentage setting work?

 




Legacy ID



308660


Article URL http://www.symantec.com/docs/TECH63406


Terms of use for this information are found in Legal Notices