Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Backup and Recovery Community Blog

Virtual Tape Libraries, Tape Capacity and Media Management

Created: 26 Oct 2011 • Updated: 26 Oct 2011 • 2 comments
Mark Harvey's picture
+1 1 Vote
Login to vote

Background:

NetBackup does not treat media and the contents of data (images) written to Virtual Tape Libraries (VTLs) any differently to real Tape Libraries.

A mile high view of the process after the 'expiry date' of each NetBackup image has passed :
- Removes the image from the image database
- Decrements the number of valid images on the media in question in the media database (a table within the EMM db since NetBackup 6).
- Once the number of valid images in the media db reaches zero, the 'assigned' field in the volume database (a table within the EMM db since NetBackup 6) is cleared
   Optionally the tape is moved back into the 'Scratch' pool (if configured).
   Note: It is best practice is to configure a Scratch pool.

Nothing in the above process touches either the media or library. For "real" libraries, the media can be out of the library and offsite (again this is best practice), and the above cleanup process completes successfully.

The Problem:

As VTLs use disk space to store the data, VTLs have a finite capacity. Often VTLs are configured where the amount of disk space is over-committed for the virtual media. This can result in the VTL filling up and unable to accept more data.

As NetBackup expires images, the now available (from NetBackup perspective) media still consumes valuable disk space on the VTL.

Note: Over-commit disk space is a simple calculation of the number of virtual media times the size of each media exceeding the total disk capacity of the VTL.

NetBackup may select new (as yet unwritten) media for new backups (instead of the previously expired media) resulting in failed backups due to the over-commit of VTL disk space.

The Solution:

Write a small amount of data to the beginning of each newly 'unassigned' media.

The VTL now 'knows' the media only contains x KB of data instead of the previous GBytes/TBytes of data. The VTL will then be able to reclaim this space for other media.

NetBackup best practice to 'write a few KB of data' is to use the 'bplabel' command and relabel the media AFTER the media has expired. Care needs to be taken with the bplabel as it does not check if media has valid data (incorrect use of bplabel can overwrite valid media resulting in the inability to restore data - you have been warned).

NetBackup hook to simplify this process is the 'media_deassign_notify' script.

- Promote the media_deassign_notify from the goodies directory (i.e. move from netbackup/bin/goodies/media_deassign_notify to netbackup/bin)
- Modify media_deassign_notify script to call 'bplabel' command with appropriate options to suit your environment/setup.

ONLY label media in the 'Scratch' pool.

For the protection of your existing backups and your sanity, configuring the 'Scratch' pool is highly recommended for this process.
   Why: Media is moved from the Scratch pool once assigned (i.e. media now contains valid image).
           The bplabel command will fail (because the media is now in a different pool).
           This protects against the race condition of other backup jobs attempt to utilise the newly released media
           within the window of 'media deassigned' and your bplabel command being called and executed.

Parameters passed to media_deassign_notify are:

 "-----------------------------------------" 
"         MEDIA ID:  $1"
"LEGACY MEDIA TYPE:  $2"
"          BARCODE:  $3"
"     ROBOT NUMBER:  $4"
"       ROBOT TYPE:  $5"
"----------------------------------------"

The modifications to the 'media_deassign_notify' script should also check the robot number and only re-label virtual media.

Examples:

Identify 'scratch' pool:

 /usr/openv/volmgr/bin/vmpool -list_scratch
Scratch Pools
=============
Scratch

Where 'Scratch' is the name of the scratch pool.

Identify 'robot number'

 # /usr/openv/volmgr/bin/tpconfig -d
Id  DriveName           Type   Residence
      Drive Path                                                       Status
****************************************************************************
0   STK.T10000B.004      hcart  TLD(1)  DRIVE=5
      /dev/nst4                                                        UP
1   STK.T10000B.005      hcart  TLD(1)  DRIVE=4
      /dev/nst3                                                        UP
2   STK.T10000B.006      hcart  TLD(1)  DRIVE=3
      /dev/nst2                                                        UP
3   STK.T10000B.007      hcart  TLD(1)  DRIVE=2
      /dev/nst1                                                        UP
4   STK.T10000B.008      hcart  TLD(1)  DRIVE=1
      /dev/nst0                                                        UP

Currently defined robotics are:
  TLD(1)     robotic path = /dev/sg8

Robot number : '1' - Note the "TLD(1)" reference to above output.

Query media details (identify which pool, if assigned and media type)

 vmquery -m SYM003
================================================================================
media ID:              SYM003
media type:            1/2" cartridge tape (6)
barcode:               SYM003TA
media description:     ---
volume pool:           Scratch (6)
robot type:            TLD - Tape Library DLT (8)
robot number:          1
robot slot:            3
robot control host:    mhmedia
volume group:          000_00001_TLD
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               Fri 24 Jun 2011 01:15:21 PM EST
assigned:              ---
last mounted:          Tue 18 Oct 2011 11:48:04 AM EST
first mount:           Fri 24 Jun 2011 01:30:25 PM EST
expiration date:       ---
number of mounts:      3
max mounts allowed:    ---
================================================================================

The above media is 'deassigned' - Note the 'assigned:' field is blank. Also assigned media will contain an extra field of "status: 0x0" as the last entry.

bplabel example

 # bplabel -m SYM003 -d hcart -p Scratch
Media is already NetBackup format, media id = SYM003, overwrite it y/n (n)? y

Note: Add the '-o' switch to automatically answer 'y' to any questions.

Comments 2 CommentsJump to latest comment

Mark Harvey's picture

A simple example of a modified media_deassign_notify script.

The following example to be added towards the end of the existing media_deassign_notify script.

(Use at own risk)

 ## Customise to environment ##
# VTL robot number
VTL=1
# Name of scratch pool
SCRATCH=Scratch

## End of customization ##

MEDIAID=$1
MT=$2
ROBOTNUM=$4

DENSITY=`grep "^$MT " /usr/openv/var/global/external_densities.txt | awk '{print $6}' | sed -e 's/"//g'`

if [ $ROBOTNUM == $VTL ]; then
        /usr/openv/netbackup/bin/admincmd/bplabel -m $MEDIAID -d $DENSITY -o -p $SCRATCH
        echo ${dateStr} "Labeled $MEDIAID "        >> $OUTF
fi

exit 0

  bplabel takes the media type as a string.

The media_deassign_notify script is passed the media type as a 'number'.

The simple regular expression searches the external_densities.txt for the number and extracts the corresponding string.

0
Login to vote
AbdulRasheed's picture

Hi Mark,

  There is no one who can talk about VTL than you! Looking forward to hearing more stories.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-4
Login to vote