Video Screencast Help

AIR Import Of BMR Data Unsuccessfull For Some Clients

Created: 06 Feb 2013 • Updated: 24 May 2013 | 11 comments
Mark_Solutions's picture
This issue has been solved. See solution.

This is already with Support (case 03591776) but wanted to put it out here as well to see if I can get it cracked.

I am pretty sure that this is an AIR issue more than a BMR issue but obviously there is a relationship between the two.

So the setup:

All a brand new installation

2 sites, each has 2 N5220 appliances (one Master / Media / BMR Master) one just a Media

All appliances on 2.5.1b with AIR EEB Bundle applied

All clients on 7.5.0.4 and set to collect BMR Data

One site is the main site and does most backups, no real backups done on site 2 as yet.

Backups and BMR collection work perfectly on the main site (just 8 Windows clients being backed up so far, mix of 2003 and 2008) and all show under BMR Clients section

AIR Replication works fine

AIR Import works fine in general but since starting to take BMR data some imports end with a Status of 1 - these relate to imports on the Master and Media Server at site 2.

Typical details of the import job here (edited names slightly that is all):

03/02/2013 20:10:07 - requesting resource @aaaaf
03/02/2013 20:10:07 - granted resource MediaID=@aaaaf;DiskVolume=PureDiskVolume;DiskPool=dp_disk_nbu02;Path=PureDiskVolume;StorageServer=nbu02;MediaServer=nbu02
03/02/2013 20:10:08 - begin Import
03/02/2013 20:10:08 - started process RUNCMD (15552)
03/02/2013 20:10:18 - Error bpimport(pid=15552) The replicated image has been imported successfully.. But failed to import bmr configuration of client client04 for backup id client04_1359921621.. Either BMR Master server is not configured on this setup or the service is down.. Please refer BMR administrator guide to know how to configure BMR Master server and import client BMR configuration manually.
03/02/2013 20:10:18 - import failed for backup id client04_1359921621 with status 1
03/02/2013 20:10:18 - Error bpimport(pid=15552) Imported 0 of 1 images successfully.      
03/02/2013 20:10:18 - Error bpimport(pid=15552) Imported 1 of 1 images partially successfully.     
03/02/2013 20:10:18 - Error bpimport(pid=15552) Status = the requested operation was partially successful.    
03/02/2013 20:10:18 - end Import; elapsed time: 00:00:10
03/02/2013 20:10:20 - Info bpdm(pid=31836) started           
03/02/2013 20:10:20 - started process bpdm (31836)
03/02/2013 20:10:20 - Info bpdm(pid=31836) reading backup image         
03/02/2013 20:10:21 - Info bpdm(pid=31836) origin master server nbu01, backup id client04_1359921621     
03/02/2013 20:10:23 - Info bpdm(pid=31836) EXITING with status 0        
03/02/2013 20:10:23 - Info nbu02(pid=31836) StorageServer=PureDisk:nbu02; Report=PDDO Stats for (nbu02): read: 3940 KB, CR received: 3939 KB, CR received over FC: 0 KB, dedup: 0.0%
the requested operation was partially successful(1)
The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details.

So out of the 8 clients being backed up and AIR replicated I have 3 on the remote site showing under BMR Clients.

Following the import the automatic restore job runs (for /C/Program Files/Veritas/NetBackup/BareMetal/client/data/bundle.dat) runs successfully.

So BMR is configure and working as it is all visible on the remote site and three clients have worked fine - so i need to track down why the others do not.

Any ideas?

Thanks

Comments 11 CommentsJump to latest comment

Mark_Solutions's picture

Ok - an update - I have been running all vxlogviews for all BMR processes and have the following outputs if they help (again slightly edited for client names) - these are from the Master Server doing the imports.

The issue does seem to be here - but what does it actually mean? 

nbu01:/ # vxlogview -o 119 -t 20:00:00
02/06/13 01:00:52.669 [winStreamHeader.cpp:removeWinStreamHeader()] Failure read
ing to end of file while processing file /usr/openv/netbackup/baremetal/server/d
ata/client05.bundle.dat.
02/06/13 01:00:52.670 [bmrd.cpp:ImportUpdateClientCurrentConfiguration()] Unable
to remove Windows Stream Header from bundle file /usr/openv/netbackup/baremetal
/server/data/client05.bundle.dat.
02/06/13 09:06:18.486 [Error] V-119-6 Server network failure
02/06/13 14:50:44.009 [Error] V-119-6 Server network failure
nbu01:/ # vxlogview -o 123 -t 20:00:00
02/06/13 15:45:07.224 [Error] V-123-48 Invalid table name specified.
02/06/13 15:45:07.251 [Error] V-123-49 Operation failed.
nbu01:/ # vxlogview -o 128 -t 20:00:00
02/06/13 01:00:47.450 [fileAttribsCommon.cpp:FileExistsCommon()] Could not do st
at on file /usr/openv/netbackup/baremetal/server/data/client05.bundle.dat
02/06/13 01:00:48.157 [fileAttribsCommon.cpp:FileExistsCommon()] Could not do st
at on file /usr/openv/netbackup/baremetal/server/data/rename.client05
02/06/13 09:06:18.470 [sendrecv.cpp:RecvPacket()] bad header, rc=0
02/06/13 14:50:43.994 [sendrecv.cpp:RecvPacket()] bad header, rc=0
 
nbu01:/usr/openv/netbackup/baremetal/server/data # ls -l
total 14732
-r--r--r-- 1 root bin 3224 Sep 17 16:00 BMRDB.load.xml
-r--r--r-- 1 root bin 720756 Sep 17 16:00 PnPDB_2003.xml
-r--r--r-- 1 root bin 524577 Sep 17 16:00 PnpDB_2000.xml
-r--r--r-- 1 root bin 526143 Sep 17 16:00 PnpDB_2000SP1.xml
-r--r--r-- 1 root bin 526630 Sep 17 16:00 PnpDB_2000SP2.xml
-r--r--r-- 1 root bin 526665 Sep 17 16:00 PnpDB_2000SP3.xml
-r--r--r-- 1 root bin 530528 Sep 17 16:00 PnpDB_2000SP4.xml
-r--r--r-- 1 root bin 770990 Sep 17 16:00 PnpDB_2003SP1.xml
-r--r--r-- 1 root bin 770996 Sep 17 16:00 PnpDB_2003SP2.xml
-r--r--r-- 1 root bin 1442188 Sep 17 16:00 PnpDB_WinPE_2003SP1.xml
-r--r--r-- 1 root bin 541349 Sep 17 16:00 PnpDB_WinPE_2003SP1_X64.xml
-r--r--r-- 1 root bin 1650573 Sep 17 16:00 PnpDB_WinPE_2008.xml
-r--r--r-- 1 root bin 382651 Sep 17 16:00 PnpDB_WinPE_2008_X64.xml
-r--r--r-- 1 root bin 730137 Sep 17 16:00 PnpDB_XP.xml
-r--r--r-- 1 root bin 741026 Sep 17 16:00 PnpDB_XPSP1.xml
-r--r--r-- 1 root bin 745725 Sep 17 16:00 PnpDB_XPSP2.xml
-r--r--r-- 1 root bin 28587 Sep 17 16:00 RuleDB_WinPE_Compatibility.xml
-r--r--r-- 1 root bin 4284 Sep 17 15:06 banner.img
-r--r--r-- 1 root bin 16 Sep 17 15:06 boot.msg
-r-xr-xr-x 1 root bin 44408 Sep 17 15:06 discoverDevices.linuxR_x86
-r-xr-xr-x 1 root bin 49548 Sep 17 15:06 discoverDevices.linuxR_x86_32
-r-xr-xr-x 1 root bin 44496 Sep 17 15:06 discoverDevices.linuxS_x86
-r-xr-xr-x 1 root bin 41236 Sep 17 15:06 discoverDevices.linuxS_x86_32
-r--r--r-- 1 root bin 1474560 Sep 17 16:00 dos.uncooked
-rwx------ 1 root root 1811152 Feb 5 20:41 client04.bundle.dat.tmp
-rwx------ 1 root root 302816 Feb 6 01:00 client05.bundle.dat.tmp

-r--r--r-- 1 root bin 3985 Sep 17 16:00 unattend.txt
-r--r--r-- 1 root bin 9554 Sep 17 16:00 unattend.xml

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Marianne's picture

Size of client05.bundle.dat.tmp looks way smaller than client04 file...

Maybe something wrong with network transfer or import...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

Thanks Marianne - lots of logging to setup now - can see this not being a quick fix!

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Marianne's picture

Have you tried to compare size of 'bundle.dat' with source master server?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

Good idea ... now where are they stored ... thanks

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mandar_khanolkar's picture

bundle.dat is inside client/data folder. Have you backed it up via your BMR enabled backup policy on primary?

Thanks.

Mandar

Mark_Solutions's picture

Backed up fine on primary site and populated the BMR Clients section - no errors on the BMR Jobs

Have looked on client04 and client05 in the primary site and there bundle.dat files are that size - so sizes are correct.

Still getting backups do the same thing today and still only 3 out of 8 clients showing on the remote site - and those three all uodated after last nights replications.

Awaiting some testing Support are doing ...

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Mark_Solutions's picture

Just an update ....

Increased logging for 119, 128 and 226 to 6/6

Also setup a 64 bit Windows SRT yesterday on the main site while continuing with the site setups (have made 32 bit as well today and same on secondary site)

So obviously, as always seems to happen when you ramp the logging levels up, it all worked perfectly last night!!

All imports worked and all clients are populated on both sites

As Bob would say - "Veritas - ain't it the Truth!"

Logs are off for analysis (and another case created this morning for some reason - so now 3630993) so will update you all when and if any reason is establish for the behavoiur I have seen.

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Jaime_Vazquez's picture

The issue in question can occur on any Unix/Linux based NetBackup Master but only  for Windows based BMR registration clients.  The client configuration file being used is called "bundle.dat".  Consider it just another tar file. For registration at AIR sites, the process will perform an "alternate client" restore (i.e. BMR will do a restore of this file from the newly duplicated and imported backup image itself. Because it came from a Windows backup image, it has some Windows based "baggage" which in this case is a Windows file header. The import process is supposed to strip off the header field before importing into the BMRDB.  The routine that is supposed to strip the header somehow failed to do so.  The resulting file can not be imported as a consequence.

To "see" what this means, use the command "file $client.bundle.dat.tmp" which will not show it as a tar file, but as some other file format.( if I remember correctly, a VAX executable). Of interest, the original reporting customer also had the  issue disappear when they set the debug level to a higher value.

This is a known issue on appliance 2.5.1 and NetBackup 7.5.0.4 .  Contact Symantec support and reference ET3081314 for an Engineering fix binary.  .The fix is included in the NetBackup 7.5.0.5 release.

 

 

SOLUTION
Marianne's picture

Mr BMR himself!

We are honoured to have you visit our community, Jaime!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

I cant believe I left this thread open and not updated - must have been busy!!!!

This was down to a bug on the appliance and was fixed by applying:

SYMC_NBAPP_EEB_ET3081314-2.5.1.0-1.x86_64.rpm

Which is ET3081314 exactly as Jaime has said.

Thanks for nudging the thread and providing the definitive explanation - close now to keep it out of the "can you solve these" that we all love so much!

Thanks Jaime - your points!

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.