Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Enterprise Vault Engineering Blog

Amazon Simple Storage Service (S3) Migrator Facts in Enterprise Vault

Behind-the-scene architecture details, feature usage & Troubleshooting tips about Amazon S3 as secondary storage Mirgator of data in Symantec Enterprise Vault
Created: 24 Jul 2014 • Updated: 24 Jul 2014 • 3 comments
Manoj Chanchawat's picture
0 0 Votes
Login to vote

In the blog, I will introduce details about Amazon S3 cloud migrator feature & its usage in Enterprise Vault. This will help you to understand architecture, feature details, how to use it as secondary migrator in Enterprise Vault & if any problem, how to start troubleshooting it.  

In Enterprise vault world, what is use of Amazon S3 cloud migrator?

  • Customer can use Amazon S3 as a secondary storage location in the cloud to store data which is CAB files, created by the Enterprise Vault file collection software, which in turn contains *.DVS* files.
  • In Enterprise vault 11, the feature name has been changed from Symantec Enterprise Vault Cloud Storage Connector (Amazon S3; Rackspace or AT&T Synaptic) to Symantec Enterprise Vault Cloud Storage Secondary Migrator.

Details of Feature Usage

  • Enables Migration of archived data to Amazon S3 cloud storage.
  • Enables Retrieval of archived data from Amazon S3 cloud storage.
  • Enables Expiration of archived data from Amazon S3 cloud storage.

What’s the minimum requirement?

  • The Enterprise Vault 10.0.1 or later installation supports migration to Amazon S3 storage
  • Amazon S3 account
  • At least one Amazon S3 bucket to store data.

Some interesting facts\issues highlighted via this article:

  • Amazon Simple Storage Service (S3) store Multi Region support: Enterprise Vault started with support for Amazon Simple Storage Service (S3) by store the buckets storage based in US region. Enhancing this capability further, now with Enterprise Vault 10.0.4 onwards, customer can now choose other geographical regions as well where Amazon Simple Storage Service stores the buckets that they create using the Administration Console (VAC). A different Amazon S3 store region is an option to customers to optimize latency, minimize costs, or address regulatory requirements. Objects stored in one Amazon S3 store region never leave that region unless customer explicitly transfer them to another Amazon S3 store region.
  • With Enterprise Vault 10.0.4, list of Amazon S3 store regions supported via Enterprise Vault are: US Standard, US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), South America (Sao Paulo). The Advanced tab of the vault store partition properties includes the Amazon bucket region parameter, which provides various region options.
  • With Enterprise Vault 10.0.4 we support Amazon S3 instances across the world using both HTTP and HTTPS

Oh that’s great, what’s the architecture?

  • The architecture is based on time tested Symantec’s OST framework. Well in Simple terms, here is how EV will communicate with Amazon S3 for storing, retrieval and expiration:

Architecture.jpg

Backend of this architecture

  • Answer is: Enterprise Vault Storage Streamer API:

The Enterprise Vault Streamer exposes interfaces that benefit the communication of Enterprise Vault with third party Storage System Vendors components that store, retrieve and delete Enterprise Vault content stream to and from a storage system.

As shown in the pictorial representation, the Vendor’s Streamer Plugin implements all the interface methods exposed by EV Streamer API. This facilitates Enterprise Vault, which in turn loads the Vendor’s Streamer Plugin DLL and calls appropriate method to read and write data to the Vendor’s Storage System. The Streamer storage could be presented as a primary or secondary storage in the Enterprise Vault Administration Console.

backend Architecture.png

Worth noting here: There are other devices that also make use of EV Streamer API. Please check Compatibility Guide for supported devices as well as related blogs on Streamer API devices.

Troubleshooting Tips?

  • Verify the correctness of Amazon S3 configuration parameters and/or Server connectivity issues by clicking the ‘Test’ button on Partition’s Advanced tab.
  • Set LogLevel parameter to ‘Everything’ for verbose logging of Amazon S3 OST Plugin.
  • Set Dtrace on StorageFileWatch.exe to monitor migration activity (data upload and empty cab deletion).
  • Set Dtrace on EVStgOfflineOps.exe to monitor retrieval or restore activity.
  • Set Dtrace on StorageManagment.exe to validates the configured settings and monitor ‘Test’ button functionality.
  • For further refinement, set Dtrace filter on “OST Streamer” keyword to view Amazon S3 OST Plugin specific log messages.

Troubleshooting Scenarios & Solutions:

  • Scenario 1 – Amazon Region: Enterprise Vault version 10.0.4 onwards, when a customer choose a regions where Amazon Simple Storage Service stores the buckets, it should immediately reflect in the configuration page as well as in the configuration file (perf file). Just in case, if the new region changes, to store the buckets, are not reflected (for example from US region to Sydney region in the Advance configuration tab on EV), it may result in bucket to be created in US region only.

Solution: To solve such issues, one needs to make sure that the right configuration is entered in Enterprise Vault (Advanced tab of the vault store partition properties). Second, make sure you have right account\credential for the new region. Third, check the integrity and location of bucket by logging into Amazon S3 account via browser and verifying bucket and location.

  • Scenario 2 – CURL Proxy Settings:  Many customers prefer to use and configure proxy with Amazon S3 in Enterprise Vault. The CURL proxy settings (with or without SSL) can be entered in Enterprise Vault in Advanced tab of the Vault Store Partition properties. If there are any issues in this, the way customer can troubleshoot is by narrowing it down if this is an EV configuration issue or in general network issue. In browser, one can type https://s3.amazonaws.com and check if it is hitting correctly with same proxy settings (as entered in the Advanced tab of the vault store partition properties). If results vary in both cases, then it may be a pointer as which part the issue is.

For example, in case if you get HTTP error 400 (For example something like this in dtrace:  EV:M     OST Streamer: [TID:7584] [Plugin] <= Recv header, 0000000026 bytes (0x0000001a)|0000: HTTP/1.1 400 Bad Request|), it’s worth to check who is throwing HTTP error 400 – configured proxy or the Amazon cloud (you can use wireshark or similar tool to find that).

Second, it’s worth checking which proxy type is used – .

What’s not supported?

  • As of now, Enterprise Vault current implementation of Amazon S3 plugin, does not support SSE (Server side encryption).

Need more information? – Look for these Technote

http://www.symantec.com/business/support/index?page=content&id=TECH206024 

Both these issues related to configuration, has been addressed in 10.0.4

Hope you found this information useful. Comment if you want to know anything further on this topic. smiley. Thanks for reading this blog and sharing your valuable feedback. 

GLOSSARY

TERM

DEFINITION

 VAC

Enterprise Vault Administration Console for Configuring and Managing Archiving Target, Policies and Vault Stores.

Amazon S3

Amazon  Simple Storage Service

Content Stream

Data in the form of bytes of stream to be stored.

OST Adapter

OST Adapter  - a.k.a EV Streamer Plugin for OST

It’s a COM dll that implements all the methods exposed by IEVContentStreamer interface and acts as a bridge between Enterprise Vault Server and OST layer

DVS File

Digital Vault Saveset. DVS file is a single piece of archived content and the associated ownership. It is a fully self contained storage structure and is itself in compressed format. Every single file or email you successfully archive should have a corresponding .DVS file. Note: A DVS file is mainly stored on Primary Storage

CAB File

A CAB file is collection of one or more .DVS files. A CAB file merely clubs DVS files and doesn’t further compress them. Purpose of creating a CAB file is to migrate it to cheaper secondary storage.

Comments 3 CommentsJump to latest comment

W a t's picture

Will you please provide a feature so that customers can add to the list of S3 regions that EV can use? It would be nice to be able to add new regions as they come online and/or non public regions. Thanks.

0
Login to vote
Manoj Chanchawat's picture

Hi Wat, 

Thanks for your feedback. Our engineering and PM team will note this down for possible future enhancement. As we discussed, currently, the regions are are hard-coded. So cant be added on the fly. But we will see how we can enhance this feature more in future. 

Thank you!!

Best Regards,

Manoj

+1
Login to vote
Chris Harrison's picture

Its probably worth mentioning that these type of feature requests, enhancements etc. can be logged here as this is where PM will look:

https://www-secure.symantec.com/connect/archiving-and-ediscovery/ideas

Chris

Symantec Enterprise Vault Engineering

Customer Focus Team

+1
Login to vote