Some T/N:

http://www.symantec.com/docs/TECH86295

http://www.symantec.com/docs/TECH152577

Both T/N indicate a defect Image as the root cause.

I've not opened the TNs posted by Nicolai, but as he mentions a defective image I image they suggest to run bpdbm -consistency 2.
You could also check the bpdbm log, this will probably show the files that are being problematic.

When header files are 'corrupted' they can sometimes be manually repaied, but ofteen the only solution is to delete them (and the corresponding .f file) Sometime they are compleyly corrupted /unreadable but if you can see the FRAG line the 9th field shows the media - so there is the option to run a phase 1 / 2 import and recreate the catalog information for these images.

Martin

Thanks guys. You'll have to forgive me if this is a dumb question, but does if i delete the files per the to two TN articles, does that remove the image from Netbackup as well?

I ask because they are "yearly" backups from years past, so I don't want to lose them if I can help it. Each would have 2 copies (one Offsite). 

Those ARE the image files. 

Please locate this file in your post - LAN-CLIENTS-NIGHTLY_1177744582_FULL
copy the file to LAN-CLIENTS-NIGHTLY_1177744582_FULL.txt and post as File attachment.

If you delete the files, NBU will not know what is on the tape. You remove the ability to 'search'/view the files in the BAR gui. I think you will also create a catalog inconsistancy (an NBCC issue), as the number of images on the tape in the image db will not match what is in the media db. I said before to reimport the missing images, but I can't think if this would fix the issue, or if it would simply +1 image to the number of images in the image db and therefore still leave you with a mis-match. Hmm ... would have to test this to be sure. You can try and expire it using bpexpdate -d 0 -backupid acct-srvr-img_1177744582

If the file is damaged this might fail, but worth ago, as it will automatically update the media db, and so the number of images showing on the tape (in the media db) will be correct. Expiring the complete tape might work but do not do this , as if the tape contains images that have spanned from other tapes, it will expire these as well I think.  Having just said that if you run bpimmedia on the media id, then if only one image shows up, then you could do this.

Even if an inconsistency is introduced, it's not a big deal, it can be sorted out with NBCC, or just left, as he tape will still expire when it gets to it's expire time - I can't see that any data loss would happen.  

I'll have to x2 check on the above notes, but what can be safely said ...

1.  If you delete the image + the .f file the tape will be expired from the media db if there are no other images on the tape when the cleanup job runs (every 12 hrs by default) therefore just physically write protect this tape first.

2. If bpimmedia shows only one image, then delete the files, expire the tape (bpexpdate -d 0 -m <media id> )  - before you do this, create a unused volume pool and straight after expiring the tape, move it to this unusued pool (stops it being reused if it isn't write protected) and then you can take you time reimporting it.

martin

I am curious to see output of 'bpimagelist -backupid acct-srvr-img_1177744582 -L'

The header file that you have posted above looks fine. There may be something wrong with .f file (LAN-CLIENTS-NIGHTLY_1177744582_FULL.f)? 

PS: do you have catalog compression enabled?

I do see one issue in the catalog file ... despite all of the fragments one of the last lines says this:

ESTIMATED_KBYTES 0

yet near the top it says this:

KBYTES 32295310
 

Also, if you add up the copy 1 fragments it comes to 12210979 but if you add up the copy 2 fragments it comes to 16084331

Adding those two together comes to 28295310

I also see that the original job re-tired 6 times so looks like it had issues

As it looks like copy 1 and copy 2 still exist it may be that you have the opportunity to try and do something with this image - perhaps try duplicating another copy from the copy 1?

Things do not seem to look right though!

Hope thsi helps

Well, I'm no expert on Catalog stuff by any means, so I just figured I'd TRY a couple of things on a whim:

1. Although I'm not quite sure what this did or if it was supposed to do anything, I ran bpdbm -consistency . It generated the output, but I don't see anything in there that helps. I'm attaching it for what it's worth.
2. I ran "Verify" from the GUI. All ended with Status 191. The details read " Error bpverify(pid=5508) Verify of policy LAN-CLIENTS-NIGHTLY, schedule Yearly_full (wtp-isd_1177744583) failed, the database contains conflicting or erroneous entries."
3. I did an "Initiate Import" from the GUI, which was successful. But then when I tried a "Verify" again, it failed with a 191 Status again.

Marianne, attached is the bpimagelist. The only compression setting with which I'm familiar is in the Policy settings, and those are greyed out. But most of the regular backup Policies had Compression enabled.

This looks to perhaps be the problem.

check directory of client <wtp-isd>
________________________________________________
checking image file <LAN-CLIENTS-NIGHTLY_1177744583_FULL>
checking files file <LAN-CLIENTS-NIGHTLY_1177744583_FULL.f>
.f file exists, and is compressed

Can you post up the file :

...netbackup\db\images\wtp-isd\1177000000\LAN-CLIENTS-NIGHTLY_1177744583_FULL

Martin

Interestingly enough, it's backing up successfully with a Status of 0, and those errors are gone from the BPBKAR logs. I didn't do anything new after my last post.

However, the first child job is now ending with a Status of 1. The error in the details is:
Error bpdbm(pid=6388) error staging NBAZDB backup to C:\Program Files\VERITAS\NetBackupDB\staging  

Haven't had any luck yet, if anyone has any ideas.

Still getting the same behavior: Original problem ceased, but now the first child job, says "Successful" but has a status of 1.

The errors in the Job Details are:

10/30/2012 12:10:07 PM - Info bpdbm(pid=6504) staging relational database files for catalog backup     
10/30/2012 12:10:07 PM - Info bpdbm(pid=6504) staging BMRDB backup to C:\Program Files\VERITAS\NetBackupDB\staging      
10/30/2012 12:10:08 PM - Error bpdbm(pid=6504) error staging BMRDB backup to C:\Program Files\VERITAS\NetBackupDB\staging     
10/30/2012 12:10:08 PM - Error bpdbm(pid=6504) error staging NBAZDB backup to C:\Program Files\VERITAS\NetBackupDB\staging     
10/30/2012 12:10:08 PM - Info bpdbm(pid=6504) staging NBDB backup to C:\Program Files\VERITAS\NetBackupDB\staging      
10/30/2012 12:11:07 PM - Info bpdbm(pid=6504) done staging NBDB backup to C:\Program Files\VERITAS\NetBackupDB\staging     
the requested operation was partially successful(1)
 

From digging around through tech notes, etc, I looked at the vxdbms.conf (see attached) and noticed those two databases don't have file paths. Is that right? Anyone have any input? Is this even a problem?

Also attached is the databases.conf. and the BPDBM log (you can see the exact times above but the errors were around 12:10PM).

FYI. I ended up opening a support ticket. There still hasn't been too much progress. although they seem to think that both databases somehow got partially configured and don't need to be there. We were able to remove the BMRDB from vxdbms.conf, which made one of the two errors go away. The normal steps to removed NBAZDB were not successful so we are still working on that.

I'll update this when I have more info.