Add Size Based Load Balancing to BP Vault Batches

Currently the Symantec suggested method for accomplishing size-based load balancing is to use Storage Lifecycle Policies, which required additional licenses, or to do the duplication manually, bypassing Vault completely. Vaulting is only used to do ejects and print the reports thus devaluing a very useful and expensive add-on product to NetBackup.

As Vault woks presently, it creates batches based on retention level and media server(s) designated to perform the duplications. In doing so it tends to single monolithic duplicate jobs for the most commonly used retention levels while other batches are of miniscule size in comparison. This tends to leave one very long running large job using a single set of resources while other sets of resources remain available and unused. Additionally this single large batch normally will exceed the prescribe time window for duplications and thus be terminated long before it would be able to finish unhindered.

The goal first proposed option would be to look at the total size of all batches that will be assigned to a media server for duplication against the total number of assigned resources for that media server. It would not limit the number of batches to be run but would add additional batches to ensure that each destination duplication resource is assigned an approximately equal amount of space to duplicate.

I see the second option as being a further control setting for the first to allow more granular control on the size of the duplication batches. If a single batch exceeds the size and has more than a single image then images are removed from the bottom of the list until the size is below that specified. The images that are removed are put into a new batch.

A possible third option would be to set a hard limit on the number of batches for a specific media server and have Vaulting figure out how to evenly spread the images across the set number of batches.

Example: Multiple retention level batches.
Take a set of 24 images of 50 GiB each with retention levels 3, 6, and 7 having 2 images each. The media server is assigned a total of three write drives with an assigned duplication window of 10 hours.

NetBackup Vault, without the proposed options, will create four batch jobs, three of size 100 GiB and one of 900 GiB for a total of 1.2 TiB worth of duplications or 400 GiB per write drive. Assuming 15 MiB/sec (~50 GB/hour) per tape drive, it would take 18 hours to duplicate the single batch of 900 GiB leaving one assigned tape drive idle for 16 hours and the other for 14 hours. Thus when the duplication window ends in 10 hours and the duplications in progress are halted, 8 images still remain to be duplicated in the next cycle.

With the option to load balance the backups, the three batches of 100 GiB each would remain, but the one 900 GiB batch would be split up into three batches of approximately 300 GiB. Disregarding tape load and seek times, this would complete the duplication of the multiple batches in 8 hours as well as fully utilizing the three assigned tape drives. This leaves 2 hours of room available for this media server to handle additional duplication requirements.

In summary the options proposed herein would aid in fully utilizing the hardware resources to ensure that all image are duplicated in a timely manner thus ensuring business continuity in the even of a disaster. As I understand it, this is the main reason for the Vault product being in existence and these proposed options would greatly benefit this product.