Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Netbackup expired the images without duplicating them - Need help in importing them from Data Domain

Created: 09 Oct 2013 • Updated: 22 Oct 2013 | 11 comments
This issue has been solved. See solution.

Hi,

I was manually duplicating the backup images on Data domain (3 weeks retention) to tape with extended retention ( 2 months) yesterday. But the duplication failed as the tape drive paths were down. When I checked today, I see that the images on Data Domain are expired! sad. I am not sure if this was expired after the duplication job. But Netbackup shouldn't expire the images, if it is not duplicated!

I hope the images still reside in Data domain storage and need help in importing those images back from the Data Domain. Please let me know the procedure to do this.

Below is the error that I received while duplicating:

"C:\Program Files\Veritas\NetBackup\bin\admincmd\bpduplicate.exe"  -dstunit MEDIASERVER-hcart-robot-tld-2 -dp PBSG_Offsite_ORA-DB -owner *ANY* -rl 4 -fail_on_error 0 -set_primary 0 -primary -number_copies 1 -L "C:\Program Files\Veritas\NetBackup\Logs\user_ops\CORPPVT\abhasura\logs\jbpDup-20131008154419.log" -Bidfile "C:\Users\abhasura\AppData\Local\Symantec\NetBackup\20131008154419.bid" -M MASTER_SERVER

15:44:22 Duplicate started Tue Oct  8 15:44:22 2013
15:44:22 Activity monitor job id = 13796408
15:44:24 INF - Destination storage unit MEDIASERVER-hcart-robot-tld-2 on host MEDIASERVER
15:44:24 INF - Duplicating policy POLICY_NAME1 schedule Default-Application-Backup backup id CLIENT-NAME_1381236815 copy 1 created on 10/08/2013 07:53:35 on source path @aaaaq
15:44:25 INF - Waiting for mount of media id PNP070 on server MEDIASERVER for writing.
15:44:27 INF - Beginning duplicate on server MEDIASERVER of client CLIENT-NAME, reading file @aaaaq.
16:21:58 INF - host MEDIASERVER backup id CLIENT-NAME_1381236815 read failed, media manager killed by signal (82).

16:21:58 INF - host MEDIASERVER backupid CLIENT-NAME_1381236815 write failed, error requesting media (tpreq) (98).

16:21:58 INF - Duplicate of backupid CLIENT-NAME_1381236815 failed, error requesting media (tpreq) (98).

16:21:58 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236528 copy 1 created on 10/08/2013 07:48:48 on source path @aaaaq
16:24:33 INF - Waiting for mount of media id PNP098 on server MEDIASERVER for writing.
16:24:34 INF - Beginning duplicate on server MEDIASERVER of client CLIENT-NAME, reading file @aaaaq.
16:52:40 INF - Waiting for mount of media id PNP098 on server MEDIASERVER for writing.
17:22:00 INF - host MEDIASERVER backup id CLIENT-NAME_1381236528 read failed, media manager killed by signal (82).

17:22:00 INF - host MEDIASERVER backupid CLIENT-NAME_1381236528 write failed, extended error status has been encountered, check logs (252).

17:22:00 INF - Duplicate of backupid CLIENT-NAME_1381236528 failed, extended error status has been encountered, check logs (252).

17:22:01 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236585 copy 1 created on 10/08/2013 07:49:45 on source path @aaaaq
17:22:02 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:02 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:02 INF - Duplicate of backupid CLIENT-NAME_1381236585 failed, extended error status has been encountered, check logs (252).

17:22:02 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236619 copy 1 created on 10/08/2013 07:50:19 on source path @aaaaq
17:22:03 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:03 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:03 INF - Duplicate of backupid CLIENT-NAME_1381236619 failed, extended error status has been encountered, check logs (252).

17:22:03 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236646 copy 1 created on 10/08/2013 07:50:46 on source path @aaaaq
17:22:04 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:04 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:04 INF - Duplicate of backupid CLIENT-NAME_1381236646 failed, extended error status has been encountered, check logs (252).

17:22:04 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236669 copy 1 created on 10/08/2013 07:51:09 on source path @aaaaq
17:22:05 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:05 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:05 INF - Duplicate of backupid CLIENT-NAME_1381236669 failed, extended error status has been encountered, check logs (252).

17:22:05 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236697 copy 1 created on 10/08/2013 07:51:37 on source path @aaaaq
17:22:06 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:06 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:06 INF - Duplicate of backupid CLIENT-NAME_1381236697 failed, extended error status has been encountered, check logs (252).

17:22:06 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236730 copy 1 created on 10/08/2013 07:52:10 on source path @aaaaq
17:22:07 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:07 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:07 INF - Duplicate of backupid CLIENT-NAME_1381236730 failed, extended error status has been encountered, check logs (252).

17:22:07 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236745 copy 1 created on 10/08/2013 07:52:25 on source path @aaaaq
17:22:08 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:08 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:08 INF - Duplicate of backupid CLIENT-NAME_1381236745 failed, extended error status has been encountered, check logs (252).

17:22:08 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236761 copy 1 created on 10/08/2013 07:52:41 on source path @aaaaq
17:22:08 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:08 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:08 INF - Duplicate of backupid CLIENT-NAME_1381236761 failed, extended error status has been encountered, check logs (252).

17:22:08 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236778 copy 1 created on 10/08/2013 07:52:58 on source path @aaaaq
17:22:09 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:09 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:09 INF - Duplicate of backupid CLIENT-NAME_1381236778 failed, extended error status has been encountered, check logs (252).

17:22:09 INF - Duplicating policy POLICY_NAME2 schedule Default-Application-Backup backup id CLIENT-NAME_1381236791 copy 1 created on 10/08/2013 07:53:11 on source path @aaaaq
17:22:10 INF - All compatible drive paths are down but media is available (2009), cannot continue with copy 2
17:22:10 INF - write host MEDIASERVER: extended error status has been encountered, check logs (252)
17:22:10 INF - Duplicate of backupid CLIENT-NAME_1381236791 failed, extended error status has been encountered, check logs (252).

17:22:10 INF - Status = no images were successfully processed.
 

Thanks,

Ajeesh B

 

 

Operating Systems:

Comments 11 CommentsJump to latest comment

mph999's picture

If the images are on disk, when they expire the data is deleted from the disk, so no way to import.
If you are using manual duplication, then NBU will expire the original image when it gets to the expire time, if it has benn duplicated or not. This is normal - look at it this way, how did NBU know you hadn't duplicated it.

Perhaps you though that as the job you ran failed, NBU would be aware of this and not delete the images perhaps - ok, fair point, but NBU isn't able to think and make decisions like that (as far as I am aware. no backup software can do this).

For future reference, the best way to handle duplications is to use SLP.

SLP will NOT delete the backup image until the duplication has completed successfully. SLP (Storage Lifecycle Policies) would have prevented the issue you have had.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
SOLUTION
huanglao2002's picture

It's most like duplicate target storage error.You can try to run a backup job use target storage unit.

Marianne's picture

But Netbackup shouldn't expire the images, if it is not duplicated!

What makes you think that?

There is no rule in NBU that says 'all backups must be duplicated'.

You selected a retention level in the schedule of the backup policy. This is the only rule that is applied. When the backup image expires, NBU removes all catalog entries and instructs the Storage Server to delete the expired images.

As per Martin's excellent post, the only way to prevent source images from getting expired before duplicate is made, is to create SLPs. All backups and duplications must be done via SLP to have this functionality.

You specify retention levels for backup and duplicate in SLP.
When the backup is done, NBU assigns infinite retention to this copy. When duplication has completed successfully, the actual retention is assigned to the backup copy.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Ajeesh's picture

Hi Martin & Marianne

I know SLP is the best option for duplication. But in my case, we keep this particular backup (database backup) only in Data domain for 3 weeks. If the DBA's request, we duplicate specific client backups to tape storage with an extended retention as per their request. This request comes once/twice in a month. So we don't use SLP perform a daily duplication.

The backup ran on 8-Oct-13 with 3 weeks retention. So it is not supposed to be expired on the next day itself (Sorry, I forgot to mention this in my first post). The only thing I did on the backup images is to duplicate them manually to tape with extended retention (2 months). But duplication failed, and the copy1 image is expired the next day itself.!

There is no manual expiration of images done by anyone. So if it is a netbackup bug or something, I was just hoping that it still exists in Data domain (as the expiration date is not yet over) and I can some how import the images directly from the Data domain - I heard of this from someone. But I dont know how to check in DD to see if the images exists and to import them.

 

Regards,

Ajeesh

Marianne's picture

I have NEVER seen NBU expiring images by itself long before expiration date. And I am talking from 14 years of experience managing own backups as well as supporting hundreds of customers with small and large installations.

If you have logging enabled (admin and bpdbm folders on the master as a start) we may be able to trace what happened to these backup images. Failed duplication job is highly unlikely to have caused the images to be expired.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Ajeesh's picture

Hi Marianne,

I too never thought this will happen. I suspect this is due to the duplication as it happened after the duplication on the images.

What happened is happened. Do you have any idea on how to import the images from Data domain (First I need to check if it exists there). If it is an issue with the Netbackup catalog, there are chances that the images are still in the DD. Please help me on finding this.

 

Regards,

Ajeesh

mph999's picture

You just run a phase 1 / phase 2 import as you would for tape, but you choose disk from the list (as opposed to tape).

If yu have catalog inconsistencies, that could cause images to expire before they should - however, this isn't a bug as such, it's the NBDB having problems.

The best option is to log a call and run NBCC to ensure they are not any further issues.

Like Marianne, apart from above, I've not known NBU to expire images 'by accident'.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Ajeesh's picture

In admin log, I am not able to see anything related to the image expiration. Just can see the duplication failure alerts in the admin logs as I got while duplicating the images (in my first post). And bpdbm log is not enabled.

mph999's picture

Loads of TNs around about imports, he is one (tape but applies to disk) ....

http://www.symantec.com/docs/TECH72429

As mentioned, I don't think you will have any luck, but hey, worth checking.
There are times importing images is possible from disk. For example, you might physically move a disk device to another NBU environment, and simply import the images. Also, it is possible to expire an image from the NBU catalog but not delete the actual fagments on disk (usually this happens automatically but there is an option on bpexpdate to stop it, -nodelete )

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Ajeesh's picture

Hi Martin,

Thank you for your inputs on this.

I already tried importing the images (phase1) and couldn't find any images for this. :(

Thanks everyone.

Regards,

Ajeesh B

Marianne's picture

I suspect this is due to the duplication as it happened after the duplication on the images.

There is no automatic expiration linked to duplication. There is not even a bpduplicate option that says 'expire original'.

What were your exact duplication parameters/options?

IMHO - there is NO way that NBU did this.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links