GRT Dublication Error 191 with netbackup 7.0.1
Hi all
i have a problem dublicating my GRT Backup for exchange to tape drives
the job ends with code 191
also having [Error bpduplicate(pid=6620) db_IMAGE() failed: database system error (220) ] in the job log
and
[22:18:57.671 [6800.6732] <32> extend_list_add_files: File: ../nbe_cat_image.cpp Line: 405: invalid input data, Function: NBE_CatImage::addNewNode
22:18:57.671 [6800.6732] <2> extend_list_add_files: get_file_rec() error: 1 image: C:\Program Files\Veritas\NetBackup\db\images\Exchange\1316000000\exchange_Information_Store_1316717178_FULL.f nfiles: 0]
in bpdbm log
also
[23:07:12.692 [4832.6332] <2> bpduplicate: Duplicating policy exchange_Information_Store schedule Full backup id exchange_1316717178 copy 1 created on 09/22/2011 20:46:18 on source path F:\exchange_Full
23:07:12.708 [4832.6332] <2> set_job_details: Tfile (57912): LOG 1316898432 16 bpduplicate 4832 Duplicate of backupid exchange_1316717178 failed, database system error (220).
23:07:12.708 [4832.6332] <16> bpduplicate: Duplicate of backupid exchange_1316717178 failed, database system error (220).]
in admin log
The server have 50 GB Free Disk Space , only 70% of memory is utilized , 15% of processor is utilized and 25 % of network is utilized during Dublication
so i do not think that it's resource allocation problem
note
http://www.symantec.com/business/support/index?page=content&id=TECH143796 does not apply to me (all my servers runs netbackup 7.0.1)
Please Help
this is the job log and for the bpdbm, bpcd and admin logs it's too large to be attached
9/24/2011 8:07:24 PM - begin Duplicate
9/24/2011 8:07:24 PM - requesting resource vnb-hcart-robot-tld-0
9/24/2011 8:07:24 PM - requesting resource vnb-hcart-robot-tld-0
9/24/2011 8:07:24 PM - granted resource C046L4
9/24/2011 8:07:24 PM - granted resource IBM.ULT3580-TD4.001
9/24/2011 8:07:24 PM - granted resource vnb-hcart-robot-tld-0
9/24/2011 8:07:24 PM - granted resource C052L4
9/24/2011 8:07:24 PM - granted resource IBM.ULT3580-TD4.000
9/24/2011 8:07:24 PM - granted resource vnb-hcart-robot-tld-0
9/24/2011 8:09:23 PM - Info bpdbm(pid=1640) cataloguing file number 10000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Saleh [saleh]/Top of Information Store/Junk E-mail/Breaking medications rocks again! <000000000e4d0922b890d14c8fed0bccfb35ec0d0700abcc3bd4bcdabb65571c45284250e919000001c500020000f45595b8fdce8b438ae4254794e6681d002e52781ee60000>
9/24/2011 8:09:30 PM - Info bpdbm(pid=1640) cataloguing file number 20000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Adel [adel]/Recoverable Items/Deletions/PO Receipt _Purchase Order(4) _(KAL_Prod_SOB) <00000000726adcf9498f554ea5e27cbeaa7128160700964658674e54197667d9cbbe27ab589f000000005a740000964658674e54194eb7d9cbbe27ab589f000000ddff080000>
9/24/2011 8:09:34 PM - Info bpdbm(pid=1640) cataloguing file number 30000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/soha [soha]/Recoverable Items/Deletions/FW: بيان Øضور وانصرا الدورات وبيان التأخيروØركة السائقين عن يوم21~19~12011 <00000000d948e15961ee9b4898989773a11dd38b0700db5c747a8c97518441161652c89a727d0000000079bc0000db5c747a8c97514281161652c89a727d0000035ff55b0000>
9/24/2011 8:09:38 PM - Info bpdbm(pid=1640) cataloguing file number 40000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/khaled [khaled]/Recoverable Items/Deletions/Emailing: نقل Ù…Øمد صابر Ù…Øمد ابراهيم.jpg <0000000008f111ca5228af469e534018e8222cae0700921c2c82ee46a24fb48899781fe4dc91a000000007a760000921c2c82ee46a24fb42da781fe4dc91a0000036e87340000>
9/24/2011 8:09:42 PM - Info bpdbm(pid=1640) cataloguing file number 50000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/samir [samir]/Deferred Action/<No Subject> <00000000b5592f5abd3f324f8c64ddf78b27f6400700f45595b8fdce8b438ae4432794e6681d0039e33551610000f45595b8fdce8b438ae4254794e6681d00400a5674a00000>
9/24/2011 8:09:46 PM - Info bpdbm(pid=1640) cataloguing file number 60000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/AbdElaal [abdelaal]/<No Subject> <00000000fe6f13e79842594486f0e99c7fb768c207006f7a5aa6692f1e4fb51a732d6a1430aa00000049f3a40000e6a390a428399340894bfda80dfaa500000002828a150000>
9/24/2011 8:09:51 PM - Info bpdbm(pid=1640) cataloguing file number 70000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Anwar [anwar]/Top of Information Store/Inbox/(791)_Purchase Order Receiving <00000000c386df18cbdf45499f24e4beac46089b0700f45595b8fdce8b438543254794e6681d002063f95cd80000f45595b8fdce8b438ae4254794e6681d0037704fbf630000>
9/24/2011 8:09:57 PM - Info bpdbm(pid=1640) cataloguing file number 80000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Elshehaby [Elshehaby]/Top of Information Store/Junk E-mail/DHL Office. You need to get a parcel NR.9324 <0000000078e01d5d6eb2324b86c61fd5b5d6fef507005f0971338642dd651243fdea7d96109400004c2f00020000f45595b8fdce8b438ae4254794e6681d0037702cb81a0000>
9/24/2011 8:10:03 PM - Info bpdbm(pid=1640) cataloguing file number 90000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Mahmoud [Mahmoud]/Top of Information Store/Inbox/FW: Sites survey drawings <00000000c105fe4efa9c12469361d3134a0186300700f45595b8fdce8b438723254794e6681d002e4ab856430000f45595b8fdce8b438ae4254794e6681d00401be135a60000>
9/24/2011 8:10:08 PM - Info bpdbm(pid=1640) cataloguing file number 100000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Ahmed [Ahmed]/Top of Information Store/Deleted Items/Invitation: Company - Audit update and follow up (Feb 24 12:00 ZE2 in Company Premises) <000000006abdf0132e5cd41183680004ac3dd9210700e167e309dd5bd41183661234ac3dd92100000000704f0000f45595b8fdce8b438ae4254794e6681d00370343696b0000>
9/24/2011 8:10:59 PM - Info bpdbm(pid=1640) cataloguing file number 200000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Basant [basant]/Top of Information Store/Inbox/RE: sherifa el adawi's Online Recommendation <00000000b93d7cfb5ef6fb48b9a40b58610b7f1107001feda8819b14454295fccd0998a2836b0000004cc8470000f45595b8fdce8b438ae4254794e6681d003fae23dead0000>
9/24/2011 8:11:53 PM - Info bpdbm(pid=1640) cataloguing file number 300000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Hala [hala]/Top of Information Store/Inbox/FW: انزال ثلاجات كابي <00000000772938382a45d61184760004ac3dd9210700e167e309dd5bd41903660004ac3dd921000000ccd4d800002b4fdf313bf120469210d0b419c7223b000003606cf90000>
9/24/2011 8:12:38 PM - Info bpdbm(pid=1640) cataloguing file number 400000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Karim [karim]/Recoverable Items/Deletions/Delivered: FW: Missing items <000000000028224d71b8f64b8c640cc55afe3f7307004b2a2f2df230b24098018c5abef49d4b000000007a4c00004b2a2f2df230b2439e018c5abef49d4b0000036096b50000>
9/24/2011 8:13:18 PM - Info bpdbm(pid=1640) cataloguing file number 500000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Mai [may]/Top of Information Store/Inbox/Executive assistant <000000004af80de3cdfc7648a42efdac25e9ea3e07001feda8819b14454295857d9298a2836b0000015b2c37000078377bf53002dd4486e6e5d0d30ce39c000003608db80000>
9/24/2011 8:14:06 PM - Info bpdbm(pid=1640) cataloguing file number 600000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Mohamed [mohamed]/Top of Information Store/Sent Items/Fakous Site <00000000cd728e17e4f3ca4892ffd8919cafd171070066b468314840154a8c9876f6cd07b6d900000094392900001cef67955f8df244b787bdcae97eaedb0039bf5e0fb20000>
9/24/2011 8:14:47 PM - Info bpdbm(pid=1640) cataloguing file number 700000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Mostafa [mostafa@company.loc]/Top of Information Store/Old Psts/Inbox/Marketing Budget Comparison <000000003e3b612e44e2914ebfc7c3a006b77ea107001feda8819b14454295fc877298a2836b00000158d30a000010b8847394022f4d9a5dab74eb3ff0b7000011c229da0000>
9/24/2011 8:15:30 PM - Info bpdbm(pid=1640) cataloguing file number 800000 path /Microsoft Exchange Database Availability Groups/exchange.company.loc/Microsoft Information Store/CompanyDB/Database/Reda [reda]/<No Subject> <0000000064e992314e610540ae5ddec4d81c241a0700f45595b8fdce8b438ae4209094e6681d0029e6ec26c70000520bf4db3f29b54797e1e67f11ec778a000001495d370000>
9/24/2011 8:15:54 PM - Error bpduplicate(pid=6620) db_IMAGE() failed: database system error (220)
9/24/2011 8:15:54 PM - Error bpduplicate(pid=6620) Status = no images were successfully processed.
9/24/2011 8:15:54 PM - Error bpduplicate(pid=6620) Duplicate of backupid exchange_1316717178 failed, database system error (220).
9/24/2011 8:15:54 PM - Error bpduplicate(pid=6620) Status = no images were successfully processed.
9/24/2011 8:15:55 PM - end Duplicate; elapsed time: 00:08:31
no images were successfully processed(191)
Comments
I have same issue...
I've got same issue on 7.1.0.1 (EEB 2412409 installed).
Duplicationg GRT images does fail as above, and I cannot browse my mailboxes proper.
With NBU 7.0.1 on exch-server (7101 on master/medias) I couldn't browse any objects.
(Error after looong time, ERROR : database system error).
Now upgraded exch-server to 7.1.0.1, and can browse down to see the users, but cannot open their mailboxes. (samme error as above)
Support is engaged, we went through config. last week... and havn't heard anymore from them.
As we all are part of the product, we're in here to improve it.
Similar Issue - Partly Solved
I have had the same issue with a customer and this has been partly resolved by means of EEB's
It is casued by a bpdbm timeout of 5 minutes which was hard coded along with a similar bpcd issue.
EEBs are now available (or at least have been proivided to myself) for 7.1.0.1 and 7.1.0.2.
Due to using N5200 appliances as Media Servers (They are at V2 which actually reports as NBU 7.1.0.5) we have the Master at 7.1.0.2 and Exchange Clients at 7.1.0.1, both with the EEBs applied.
GRT backups work fine and do now expand properly in the BAR GUI, although each expansion can take up to 24 minutes. So 48 minutes before you get as far as seeing the e-mails.
However the duplication of GRT backups still fails with a 191 after exactly 5 minutes so we are investigating which timeout this relates to.
I will write back if we get this cracked.
The EEBs just contain the bpdbm and bpcd files.
EEB2486657 for 7.1.0.1
EEB2517601 for 7.1.0.2
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
Update
Hi All
The EEBs have resolved the issue - the bpcd gets installed on Master, Media and Client, the bpdbm gets installed on the Master
Basically, if it will take more than 5 minutes to read the information you may get this error - using the Media Server as the GRT Proxy Host will also help - using De-Dupe will make the error more likely
Also, if using a GRT Proxy Host for GRT duplications it MUST be a Windows server
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
where did you find these EEBs?
Hi
sorry, I don't have a lot of experience in NBU so I am looking for you r help since i am having the same issue in NBU 7.0.1
could you please tell me what did you do exactly? did you only istalled the EEBs?
do these EEbs contain the bpdbm and the bpcd daemons packages?
where to find these EEBs? could you please attach them here?
many thanks
To get hold of the EEBs
To get hold of the EEBs (Emergency Engineering Binaries) you need to log a ticket with support who will make them available to you should your issue match. The EEBs contains release notes about installation.
Master/Media NBU7.1.0.3 on W2k8 - VMware, SQL and Exchange agents, MSDP
OCA 7.1.0.3 on W2k8R2
PureDisk 6.6.1.2 + EEB20
but why this problem happens?
thanks Lee,
really I dont understand why that happend? everything here running fine and the problem happend out of sudden.. my configuration is as follow:
I am having two mailbox servers (clustered); mailbox01 and mailbox02. I am having a GRT policy that I use to backup individual mailboxes. for this GRT policy I created a storage unit DAG-GRT unit that contains two staging disks each disk is mounted ( connected) to its server (i.e. disk01 is connected to mailbox01 and disk02 is for mailbox02). when i run the GRT policy,backup images are created successfully from both server to the staging disks, but duplication is always failing on mailbox02 disk; i.e. I can't relocate my mailbox02 image to tapes! meanwhile duplication is always successful for mailbox01.
master server is a Solaris 10 X64
NBU master server version: 7.0.1
clients version on mailbox01 and mailbox02: 7.0.1
appriciating your help anyone.
Has it ever worked for
Has it ever worked for mailbox02?
It may be that mailbox02 is just larger than mailbox01 to causes the timeout.
On the other hand maybe you have something specified wrong on the mailbox02 host properties?
Have you specified a valid mailbox name (preferably that relates to the NetBackup Client Service running on mailbox02)
Have you specified a GRT Proxy Host for mailbox02? If so is it a Windows Server? Does it have NFS installed etc.
If the duplication fails immediately then it is a configuration error, if it fails after about 5 minutes you need the EEBs
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
yes it worked before on
yes it worked before on mailbox02, and you are right; mailbox02 is approximately 20 GB larger than mailbox01 but I am not sure if the failure really happens after 5 minutes..when I run the duplication manually and go to the duplication "job monitor" I see the following messages:
1- the duplication begins
2- requesting resource
3- mounting tape
4- bprm starting the file cataloguing
5- database system error 220
6- no image were succesfully processed
7- exit status 191
regarding the following questions :
1- "On the other hand maybe you have something specified wrong on the mailbox02 host properties?"
> i think that my files are similar for mailbox01 and mailbox02.. but to be sure about that..I can attach any file that can help, could you please tell me which files may help in here?
"Have you specified a valid mailbox name (preferably that relates to the NetBackup Client Service running on mailbox02)"
> backup were running fine before. so i dont think think we have an invalid maibox name
"Have you specified a GRT Proxy Host for mailbox02? If so is it a Windows Server? Does it have NFS installed etc."
> no proxy is used for our configuration
thank you again for your help mark
One another thread installing
One another thread installing Microsoft KB304101 has just sorted out a similar problem:
https://www-secure.symantec.com/connect/forums/exchange-mbx-disk-tape-duplication-job-failing-cannot-read-image-disk-invalid-argument
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
what appears to be a solution
dears,
many thanks for your help, we solved this problem without purchasing any upgrade license.. what would be your impression when you ask the customer to buy an upgrade license then this will never solve his problem :)...
anyway my solution was about following very closely logs generated in the /opt/openv/netbackup/logs/bpdbm. in this folder the error logs were telling me exactly that at the cataloguing phase when the duplication process was trying to catalogue the mailbox of a specific user it fails and generates the database error 220 then the duplication faisd with error 191.
for that we deleted the messages of this user and then we were able to run the GRT backup policy again.
Would you like to reply?
Login or Register to post your comment.