Video Screencast Help

Duplicate full backup and delete original?

Created: 28 Jun 2012 | 16 comments

The reason is that we are just implimenting GRT backups of Sharepoint, which requires disk storage. Until we get some new hardware, I have a very small amount of disk storage (enough for one of said backups). So, ideally, I'd like to run the normal backup on a daily schedule and then have it automatically duplicate to tape and delete the copy on disk, so that the backup can run the next day.

So, is there a way that I can automate a process to duplicate image that's on disk and delete the original once the duplication is complete? I was looking at the bpduplicate command (which we already use in batch files called by scheduled tasks to duplicate other images) and the duplication part is easy, but I can't figure out how to get rid of the old image, short of doing it manually each day.

Comments 16 CommentsJump to latest comment

sazz.'s picture

Yes this can be done and the best way to do this is using SLP or use the duplicate option in VAULT management. In both these options there is a option where you can delete the original copy once it is duplicated..

 

smckelvey's picture

SLP isn't availabe in BasicDisk is it? And NetbackupVault requires additional licensing?

sazz.'s picture

You are right, the only option is to do them manually either using bpexpdate or the other option is in the GUI go to the Catalog options and search with primary copy and copy the contents in Excel, then select copy 2 and copy the content in Excel. Do the Vlookup in excel to find out the images that are duplicated and those images can be expired either using GUI option in Catalog or via bpexpdate.

Marianne's picture

I agree - the only 'automated' methods are SLP and NBU Vault.

If you have DSSU configured, disk will also be cleaned out when High Water mark is reached, provided that backups have been successfully duplicated.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

watsons's picture

If you have only BasicDisk, I second Marianne's suggestion to use DSSU - it's automated in a way when HWM is reached. You don't even have to script it. It does make sense because you can still have the disk copy after the duplication, and only get cleaned up when disk is going to fill up. Just need to watch out for duplication error from time to time.

smckelvey's picture

Interesting. I wasn't familiar with it, nor could I find much info. But, I think I got it set up now. Couple of questions/oncerns, which I suppose i can answer through testing, but in case you know the answer:

  • High/low water marks- Is it possible that if I make the Low Water Mark lower, can the media server delete the files in real time so said backup can continue w/o error. It seems like the backup is if it hits the High Water Mark regardless.
  • Once the duplication is complete, is the image on the Disk Storage Unit supposed to expire/cleanup/delete or anything? I did a successful duplication using DSSU, but the original image (on disk) was still in the Catalog as the primary, and the files were still on the disk space.
  • Lastly, isn't this really a "backup to tape" that's simply been staged on disk first? I'm not sure what the difference would be but Netbackup documentation seems very specific about GRT backups and restores being done to/from Disk. But this might be something that I won't know definitively until I test.

Thanks in advance. As always, you have some really good info and suggestions.

watsons's picture

 

1) You mean making default Low Water Mark (LWM) value lower? Of course you can, the only rule is "HWM > LWM". 

 

2) No, when duplication is done, it won't cleanup/expire/delete your image. It will start to do so when your HWM is reached. For example HWM is 90%, LWM is 70%, you keep doing backup (to DSU) and duplication (to tape) daily, nothing happens, one day the DSU  is 90% filled up, the image cleanup job now kicks in to select and cleanup the oldest images. How much data would it select to cleanup? That will be determined by LWM, so cleanup will select up to a point that DSU is back to 70% filled. So the automation is not based on duplication completion, but the disk usage/consumption.

 

3) Yes, it's a staging kind of setup and yes it's applicable to all type of data.

smckelvey's picture

what about a script using bpduplicate, then maybe bpexpdate?

 

Regardless, maybe someone can help me out with this bpduplicate script, at least I can automate that. I'm trying to make 2 copies of the image. Both new copies will reside on the same storage unit, but in different Volume Pools. I've been playing with it, and this is the closest I've gotten to working, where it starts writing one of the copies (to "Full", but I'm not sure if that's coincidence), but the other immediately generates a status 167.

bpduplicate -number_copies 2 -dstunit balt-srvr-nbu-lto,balt-srvr-nbu-lto -dp FULL,offsite_full -policy SharePoint_GRT -hoursago 11 -L c:\Duplicate_Scripts\Dup_SP_GRT_log.txt

Here's the output of the log file.

15:56:22 Duplicate started 6/28/2012 3:56:21 PM
15:56:22 Activity monitor job id = 308698
15:56:22 Copy 1 activity monitor job id = 308699
15:56:22 Copy 2 activity monitor job id = 308700
15:56:26 INF - Destination storage unit balt-srvr-nbu-lto on host balt-srvr-nbu
15:56:26 INF - Destination storage unit *NULL* on host balt-srvr-nbu
15:56:27 INF - Duplicating policy SharePoint_GRT schedule Weekly_Full backup id wtp-sp2010-fe1_1340902715 copy 1 created on 06/28/2012 12:58:35 on source path E:\SharepointBackup
15:56:28 INF - no storage units available for use (213), cannot continue with copy 3
15:56:32 INF - Waiting for mount of media id 000113 on server balt-srvr-nbu for writing.
15:56:34 INF - Beginning duplicate on server wtp-nbumedia2.wtplaw.com of client wtp-sp2010-fe1, reading file E:\SharepointBackup\wtp-sp2010-fe1_1340902715_C1_F1.

 

Marianne's picture

> bperror -S 167 -r
cannot find requested volume pool in EMM database
A backup to a nonrobotic drive was tried and the tape manager (bptm) cannot find or add the specified volume pool.

Verify the Media and Device Management volume configuration. Check the NetBackup Problems report for
 more information about the error. For detailed troubleshooting information, create a bptm debug log
 directory and retry the operation. Check the resulting debug log.

 

If FULL volume pool is fine, check exact spelling and case of offsite_full.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

smckelvey's picture
  • High/low water marks- Is it possible that if I make the Low Water Mark lower, can the media server delete the files in real time so said backup can continue w/o error. It seems like the backup is if it hits the High Water Mark regardless. NO, Backup will fail until there is enough space again.
  • Once the duplication is complete, is the image on the Disk Storage Unit supposed to expire/cleanup/delete or anything? I did a successful duplication using DSSU, but the original image (on disk) was still in the Catalog as the primary, and the files were still on the disk space. NO.
  • Lastly, isn't this really a "backup to tape" that's simply been staged on disk first? I'm not sure what the difference would be but Netbackup documentation seems very specific about GRT backups and restores being done to/from Disk. But this might be something that I won't know definitively until I test. This will work for GRT, but as stated in the manual, images on tape will need to be duplicated back to Disk and it must be the Primary copy in order to Restore from it.
smckelvey's picture

I'm having some trouble getting this to behave the way I think it should or would like it to. Any experienced or knowledgable input is appreciated.

This is only a temporary setup, until we get some new Disk Storage (but that could be a little while), but I needed to get it running so we can do GRT backups of Sharepoint until that time. So, no need to recommend more space, etc; as this is all I have to work with for the time being.

  • A BasicDisk storage unit: 341GB of attached disk space on Media Server.
  • 1 backup takes approximately 180GB, so I only have enough space for 1 backup (sometimes 2 can squeeze in)

I'd like to get it so that Netbackup will free up the space and continue to back up w/o any intervention, but I'm not sure if that's possible.

  • From what I've read the Low Water Mark (LWM) doesn't matter in BasicDisk, correct? So we can leave that out of the equation.
  • See screen shots for DSSU configuration.
  • The backup to disk runs successfully. Then the Duplication to tape runs per my schedule and is successful. After which it appears as though I have 2 Primary copies (one on tape, one on disk).
  • The problem is that I don't understand when or how the HWM comes into play, because I don't see any behavior related to it:
  1. Per this KB, "the high water mark is used to trigger the deletion of images that have been relocated." So, shouldn't the image and/or data removed from the Catalog and Storage unit, leaving me with only 1 primary copy on tape?
  2. And even when the next scheduled backup runs, it doesn't remove data or delete the previous image from disk.
  3. And if this behavior is because of retention times, I don't see anyway to differentiate between the tape and disk copy, as the only setting appears to be the retention configured in the backup policy.

watsons's picture

Not sure whether you have gone through all we had posted in regards to your questions earlier...

Anyway, you've given an example (which is good), supposed we can talk about it. However, I should say there is not much we can play around because the storage is just not big enough to hold even 2 full copies of your full backup. 2 full copies would take up 360GB and is already filling up your disk!

- Let's assume you have 1TB of available disk in your basicDisk, and you're only doing full backup for this case, forget about incremental backup for now.

- Day1 : Backup 180GB, assume it's duplicated to tape by end of day. Disk storage usage is now 18%
- Day2 : Backup another 180GB, duplicate to tape again. Usage = 36%, nothing would happen.
- Day3 : Backup another 180GB, duplicate to tape again. Usage = 54%, now... this is where something would happen, since your HWM is 50%, and now disk usage is more than HWM, image cleanup would start.

and how would cleanup knows when to stop, that will be up to the LWM. In your case:

- Cleanup deletes day1 image, and disk usage is reduced to 36% , but still > LWM. That means more cleanup is needed.
- Cleanup deletes day2 image, and disk usage = 18%, still > LWM. More cleanup needed.
- Cleanup deletes day3 image, only then disk usage is < LWM, and cleanup would stop. Now you have an empty disk for have backup run again on Day4.

The LWM value of 10% in this case is not a very good setting as it will cleanup all images - leave you no disk copy, so a 20% would be better IMO. So the value should be set based on your total image size (per day), and its retention (and other factors, but let's keep it simple here). 

If you then notice, 341GB is really hard for you set the HWM & LWM here because of the quick usage and no room for 2nd backup. Consider adding more diskspace into it before changing to use this.

smckelvey's picture

Yes, I've been following up on each post, which is how I arrived at my last set of questions, which all remain unanswered.

  • As I said, I need to work within the confines of my current environment for the time being. It's not ideal, but it's temporary, and I see no reason that it shouldn't work. Recommendations for more space are pointless in this discussion.
  • You keep referring to the LWM, for which I've already asked for confirmation of the fact that it is NOT applicable to BasicDisk. If you are saying that it is, can you please expound (perhaps the literature is only for previous versions, or a scenario that I missed, etc). Otherwise, let's leave it out of the discussion.
  • And as I said, I've don't 2 backups to said diskspace, after the first exceeded the HWM. Both images duplicated successfully as well. But no images were expired nor was any disk space recovered that I can tell.

So, my questions are the same.

  1. Per this KB, "the high water mark is used to trigger the deletion of images that have been relocated." So, shouldn't the image and/or data removed from the Catalog and Storage unit, leaving me with only 1 primary copy on tape?
  2. by "deletion of images" (above), does that mean it will "Expire" the image and delete the associated files, or will it only delete the files, or.....?
  3. If HWM is the only trigger for deletion, why isn't it deleting in my example?
  4. If Retention time and HWM play a role in the process, how do I create a different retention between the tape and disk copy (the only setting appears to be the retention configured in the backup policy)?
watsons's picture

I tested my basicdisk, much to my surprise, the image cleanup just works like how I understand it - not sure if it's a coincidence. But the image cleanup just got trigger when HWM is exceeded, and it stops once LWM is met! Here is a snippet of my image cleanup job detail:

07/16/2012 11:03:28 - Info bpdm (pid=30546) initial volume /bscdisk: Kbytes total capacity: 50330044, used space: 43095820, free space: 7234224
07/16/2012 11:03:39 - Info bpdm (pid=30546) Removed 3318488 Kbytes
07/16/2012 11:03:39 - Info bpdm (pid=30546) ending volume /bscdisk: Kbytes total capacity: 50330044, used space: 39777332, free space: 10552712

My HWM = 85%, LWM = 82%

 

watsons's picture

Oh, sorry about that.. guess I got the wrong perception on LWM. I am going to test this in my environment... and hopefully get some insight on it.

sazz.'s picture

I do agree with Watson and tested the configuration.

My environment:

NBU 6.5

Basic Disk - used D drive for staging

 

So, my questions are the same.

  1. Per this KB, "the high water mark is used to trigger the deletion of images that have been relocated." So, shouldn't the image and/or data removed from the Catalog and Storage unit, leaving me with only 1 primary copy on tape?

 

As per the policy schedule the data will be written to the basic disk. Once the backup completed the DSSU policy will trigger as per the schedule you have in the storage unit for staging. If for some reason the DSSU policy is not getting triggered the backups will fill the basic disk and eventually fail with no space avilable. The data on the basic disk will not be deleted or expired unless it get duplicated through DSSU policy. The data will remain on the disk as per the policy retention that you have mentioned. If your DSSU policy runs properly it will duplicate the data from the basic disk to the tape but the data will be only deleted from disk depending upon the HWM setting.

 

2)by "deletion of images" (above), does that mean it will "Expire" the image and delete the associated files, or will it only delete the files, or.....?

Once the data is duplicated to tape through DSSU policy and HWM is triggered it will delete the images from basic disk. Now you only have primary copy on tape and disk data doesn't exist on catalog or anywhere, Its gone completely.

3)If HWM is the only trigger for deletion, why isn't it deleting in my example?

Please test it I have tested it for past 3 days and works like charm. At one point I have set the LWM to 30% and it immediately deleted the last image on the basic disk. If you still think it doesn't work for you please post the output of your policy details, storage unit

4)If Retention time and HWM play a role in the process, how do I create a different retention between the tape and disk copy (the only setting appears to be the retention configured in the backup policy)?

The retention will only be available in the policy. Lets see you have a 100 GB disk and you are doing a retention of 1 week. The backup will run on basic disk and then as per DSSU policy it will be duplicated to tape. Now you have 2 copies, primary on disk and secondary on tape. The primary copy on disk will not be deleted till HWM gets triggered, till that time you can restore from disk instantly. Though the image on disk will be expired after 1 week.

 

Hope this makes things a bit clear.