Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Job to duplicate B2D sets to removable drive scatters data all over the place and kills Agent

Updated: 23 May 2010 | 26 comments
xilog's picture
0 0 Votes
Login to vote

Hello.

I hope you good people will be able to help me, I'm at my wits end.

Apologies for a lengthy post, the devil is probably in the detail with this problem

I'm attempting to create a backup server to enable a two-stage backup, first stage being a nightly backup from servers to the local drive array and then duplicating that backup to a removable drive.  Then at the end of the week the removable media is swapped and the local backup deleted, leaving the removable device as the "live" backup and the local array clean for another week's backups.  Full backup runs on Friday night with diffs every Sat-Thu night.

I have created what I think are suitable devices, media sets and jobs and indeed everything works fine until the duplicate jobs start.

There are three devices, a B2D folder called "Servers" for daily backups, a B2D folder called "backupsvr" which contains a backup of the backup server immediately after Backup Exec installation and a removable B2D device called "Disk_1" (set up as drive F:) which is the destination for duplication.  All these use the default settings of 1GB files and 100 backup sets.

There are just two media sets, "Daily Backups" and "Daily Duplicates".  Daily Backups has an OPP of 2 weeks and indefinite append, Daily Duplicates has an OPP of just 1 day since its disks will be ejected at the end of the week and not reinserted for at least 5 weeks and also has infinite append.

There are two main backup jobs, Friday Full and Daily Diff, both targeted at the Servers device and putting its media into the Daily Backups pool.  Each of those has an associated Duplicate task targeted at Disk_1, the idea being that after the main backup is complete, it's then duplicated on the removable device.

To my way of thinking, thios should all work, yes?

Well what actually happens is this:

The "Full Backup" runs properly, backing up to the Servers B2D device and putting the media in the Daily Backups pool.
The "Duplicate full backup" task starts, initially putting files onto Disk_1 as expected.  When it's put 20GB of data onto the disk (it's a 1TB drive) it goes very, very slowly and then kills the BE agent, finally it gets round to verifying this 20GB and when it starts writing again it starts putting the B2D files in the Servers device and the backupsvr device (seemingly at randon) and putting the media in any pool it seems to feel like; it's put some in Daily Backups, some in Daily Duplicates, some in Imported media.  Some even ends up in Scratch Media.

What the blazes can be going on?  It's driving me nuts trying to figure out what's gone wrong.

Any advice you can offer?

Thanks,

Kevin.

Comments

Masti24by7's picture
06
Jul
2009
0 Votes 0
Login to vote

Job to duplicate B2D sets to removable drive scatters data all o

Hello Xilog,

I guess you have selected the all devices in the job properties i would suggest you to target tat job to the B2D you want it to write.

you can change that settings by right clicking the job | Properties | device and media | Device(dropdown)

Select the Specific B2D for the job

xilog's picture
06
Jul
2009
0 Votes 0
Login to vote

Hi Masti, I wish it was that

Hi Masti,

I wish it was that simple; the job was explicitly targeted at Disk_1 already.

Deatheye's picture
06
Jul
2009
0 Votes 0
Login to vote

interesting. We got the same

interesting. We got the same problem with scattering data all over the place. Tried it out with two different medai servers... also definied the target device...
Looking forward to an answer to this ^^'

RahulG's picture
06
Jul
2009
0 Votes 0
Login to vote

hi

are there any error handling rules hwich you have configured ?? if there is not then it might be some corruption int he databse causing this issue .
Let me know if this happens only on duplicate backup jobs or it happens on all the backup job

If this response answers your concern, please mark it as a "solution"

Deatheye's picture
06
Jul
2009
0 Votes 0
Login to vote

Could you check what you

Could you check what you configured as the source device for the duplication job? I just noticed that my setting there was wrong. Still trying to find out how to repeat the duplication job for verification.

Ben L.'s picture
06
Jul
2009
0 Votes 0
Login to vote

I have a couple questions on

I have a couple questions on what you are doing.

1. "Daily Duplicates has an OPP of just 1 day"  - So with this configuration the Tuesday duplicate job could overwrite the Monday duplicate job.  Is this how you wanted it configured?  If you want to keep all the duplicates for the week you will need to increase this to atleast 5 days to not have any problems with previous duplicates getting overwritten.

2. "When it's put 20GB of data onto the disk (it's a 1TB drive) it goes very, very slowly and then kills the BE agent" - Explain kills the BE Agent.  This could be the root of all your problems. Please explain this with as much detail as you can provide.

If this response answers your concern, please mark it as a "solution"

xilog's picture
06
Jul
2009
0 Votes 0
Login to vote

Hi folks, I'll try to be as

Hi folks,

I'll try to be as detailed as possible:

  1. I chose 1 day just so that if the initial backup takes a very long time then the later part of the backup wouldn't overwrite the first part.  The duplicates for Sat-Thu are all "Append, fail if no appendable media" type. So there's no chance that there will be an overwrite.
  2. OK, 20GB gets written, and then the icon bottom right that shows the status of the media server services changes from the green "play" arrow to the red stop square, and in services.msc the agent is shown as stopped and there are error log entries showing that beagent caused an exception (0xc0000005 I think from memory; I'm at home atm) and terminated.  At this point there is still a huge amount of activity on the local RAID but nothing is apparently being written to the removable drive.  After about 45mins, the job switches to "verify" status briefly and then carries on backing up but to a wrong target device.
  3. No error handling rules are set up yet.

Since I first started this thread I've experimented by creating new B2D and removable B2D devices, media pools and jobs and tried a quick test and it seems to be working.  It's possible there was a problem with the database I guess or that I mismanaged the initial setup (which would be annoying but if so at least I know it's not a serious problem!)  I'll check it after tonight's jobs and see what happens.

Ben L.'s picture
06
Jul
2009
0 Votes 0
Login to vote

1. For a test, change the OPP

1. For a test, change the OPP of your duplicate media set to 5 days. Then change the duplicate job to "Append to media, Overwrite if no appendable media is available." 

2. beagent is not a part of Backup Exec 11.x or higher.  This is an older agent in 10.x and lower. You may have some pieces of an old install still running on the server that need to be removed.

If this response answers your concern, please mark it as a "solution"

xilog's picture
07
Jul
2009
0 Votes 0
Login to vote

OK Ben, OPP to 5 days, should

OK Ben, OPP to 5 days, should I leave append period as infinite?

Also, beagent was a slip of the fingers (I was writing from memory); it was beremote.exe that was faulting, with c0000005 error.

xilog's picture
08
Jul
2009
0 Votes 0
Login to vote

It's an Agent or dll problem

Came in this morning and saw that there was a pattern to the scattering of data across devices.  The full backup worked okay as before, then the duplicate started.  It initially used its targeted device Disk_1 for the first 10GB chunk, and then part way through the second 10GB chunk the following errors appear in the Windows event log:

07:23:44, Application Error: Faulting application beremote.exe, version 11.0.7170.0, faulting module EDBProv.dll, version 11.0.7170.0, fault address 0x00014d31

07:23:48, DrWatson: The application, C:\Program Files\Symantec\Backup Exec\beremote.exe, generated an application error.  The error occurred on 07/08/2009 @ 07:23:45.287. The exception generated was c0000005 at address 0C114D31 (edbprov!InitEdbProv)

After that, the duplicate job continues but puts data sequentially into the Disk_1, Backupsvr and Server B2D locations but is putting it into the correct media set, Daily Duplicates.  The job is also running very, very slowly at this point, presumably because the agent is not working.

It appears that the error is occurring as it's duplicating the backup of the Exchange server as the following errors also appear in the BE job log at the same time:

V-79-57344-65069 - WARNING: "\\{removed by poster}\Microsoft Information Store\First Storage Group\Staff" is a corrupt file.
This file cannot verify.
V-79-57344-65069 - WARNING: "\\{removed by poster}\Microsoft Information Store\First Storage Group\Students" is a corrupt file.
This file cannot verify.
V-79-57344-65072 - The connection to target system has been lost. Backup set canceled.

The original backup of these information stores was successful.

Does this help pinpoint the problem any better?

Thanks,

Kevin.

xilog's picture
08
Jul
2009
0 Votes 0
Login to vote

It might NOT be an agent problem. Now what?

So this time, having run liveupdate to solve the agent problem, the agent didn't fault.  The job still behaves in the same way though; fails at the same place. Next run will exclude the Exchange server to see if it's duplicating the Exchange data that's causing the problem as it starte misbehaving about then.

This is driving me insane; everything is set up correctly but BE just seems to want to scatter my data all over the place.

xilog's picture
09
Jul
2009
0 Votes 0
Login to vote

It's a bug. The culprit is GRT.

Searching the forums and knowledgebase yielded a few clues where GRT was failing duplicate to tape backups so I wondered if it might be the culprit here.  It is.  Last night a backup ran just as before but with just the GRT option disabled.  The duplicate job worked perfectly.

How does one report a bug?  Is there a bug report tracker/system here?

Kevin.

Ben L.'s picture
10
Jul
2009
0 Votes 0
Login to vote

How does one report a bug?

How does one report a bug? Is there a bug report tracker/system here?

Best way would be to open a case with support to have them look at it.  The issue you are facing may already be resolved with a later version of the product.
http://www.symantec.com/business/support/contact_t...

If this response answers your concern, please mark it as a "solution"

xilog's picture
10
Jul
2009
0 Votes 0
Login to vote

Thanks for he link Ben, I'll

Thanks for he link Ben, I'll give them a shout.

nn@digital-healthcare.com's picture
10
Jul
2009
0 Votes 0
Login to vote

Same problem here

I used to have all the media set to 64Gb and only started having the problem when I reduced it to 16Gb (so that I could pre-allocate and avoid fragmentation without wasting too much space). I will try this weekend with my offsite media set to 64Gb and see if it solves the problem. Maybe the problem comes when the GRT backup doesn't fit into a single file in the duplicate set? I am on BE12 with all service packs and hotfixes. No errors from the agent. I do have a GRT backup of Exchange 2007.

xilog's picture
10
Jul
2009
0 Votes 0
Login to vote

I'd be interested to hear

I'd be interested to hear your results.  Initially my disk sets were set to 100GB, more than twice the size required for the Exchange backup but the duplicate was failing at only 20GB into the backup (the size of my first information store)

nn@digital-healthcare.com's picture
13
Jul
2009
0 Votes 0
Login to vote

Still going...

The duplicate job has nearly finished and all the media has been allocated from the correct B2D device so far (680Gb backup with with 30Gb Exchange GRT and SQL but no Sharepoint). One thing I find strange is that it takes over half an hour to allocate each new 64Gb file on the external USB RAID array (these are new files as I had to wipe everything when starting with the new media size). Maybe it will be faster when it can overwrite existing ones.

Ken Putnam's picture
13
Jul
2009
0 Votes 0
Login to vote

Maybe it will be faster when

Maybe it will be faster when it can overwrite existing ones.\

Should definitely speed up when you no longer need to create/format new BKF files

If this response answers your concern, please mark it as a "solution"

Deatheye's picture
21
Jul
2009
0 Votes 0
Login to vote

Hmmm interesting. Could

Hmmm interesting. Could please inform us further about this problem if you know anything new xilog?

We have a case open with the spreading data everywhere problem with symantec support. Till now we aren't further anywhere there. The job the symantec supporter created worked fine for severall tries. But never any job we created on 3 different backup exec server.
I try to check if I can confirm this too with GRT.

EDIT: Where excatly should I deactivate GRT? I found several places to activate / deactivate GRT inside the job.

xilog's picture
21
Jul
2009
0 Votes 0
Login to vote

Deatheye, I've tried a couple

Deatheye, I've tried a couple of tests and it's definitely GRT that's causing the problem. If it's on, the replicate fails and sprays data all over the shop.  If it's off the replicate works.  Reproducible 100% here.

What seems to happen is that the duplicate job reaches a point in the duplication of the information store, borks and then seems to get confused over which device pool it's using, changing from the defined device pool for the job to the "All Devices (SERVER)" pool.

GRT is deactivated by unchecking "Enable the restore of individual mail messages and folders from information store backups" on the Exchange options tab of the backup job that the duplicate job is linked to.

Good luck, and if you get a solution that works without disabling GRT, please let us know :)

Deatheye's picture
22
Jul
2009
0 Votes 0
Login to vote

hmm. that job which has this

hmm. that job which has this problems doesn't even backup exchange.. well I try an ddeactivate every GRT Option I find. There are other places where you can activate GRT, like Sharepoint. For you only the Exchange GRT Option caused this problem?

I just let a copy job run inside a testenviroment. The data was copied to places where it souldn't, now I'll try and see what happens with GRT deactivated.

EDIT: deactivated every GRT Option I found, and copy job looks fine. Did you trace anything with sgmon?
Does this only happen if you actually backup something that has a GRT setting?
It's still kinda wierd cause the copy jobs we have this problem with (actuall all media servers we got have this problem), don't even backup an exchange store. At least not all of them. And one server has an backup job that only makes a backup onf an information store, duplicate job there worked fine when I tested it once. But on the same media server an other duplication job has this missbevhaviour and doesn't backup the information store.
So I kinda wonder if this only happens when GRT is activated, you backup objects which support GRT and objects that don't, and you duplicate the backup.
Some jobs we had even worked fine with GRT activated. But I think they don't backup any data where GRT is used.

So right now it looks like there is no problem with GRT activated, and only GRT data, GRT activated and only non GRT Data, but if a job contains GRT data and non GRT data and you duplicate that it happens.

I'd like to analyse this further and make some more test jobs to be sure the last job not only did work out by accident.
Is there anything else besides Exchange, Sharepoint and active Directory that got a GRT Option?

I just noticed that this would mean about 36 or more different backup jobs with a copy job to define which GRT Optiojn excatly causes this, and if the assumption is really correct. O_o
I think I just make again 2 jobs one with GRT one without to be sure that the behavior really isn't by accident and is repeatable.
This really did cost a lot of time allready (severell days... didn't rellay count them)... and without the hint from xilog we propably still would wander blind together with the support...

xilog's picture
22
Jul
2009
0 Votes 0
Login to vote

No, I didn't use sgmon yet,

No, I didn't use sgmon yet, that's usually a last resort :)

I'd like to hear from a few others using GRT and B2D duplication to see if they suffer the same issues.

I've not yet tried an information store only backup so that there's only a GRT-using element to the backup and given that I can restore to a restore group in exchange 2003 anyway, I think I'm just going to settle for not using GRT.  It's a nice feature but far from critical for us.

You're right, there are a large number of combinations to get this right, for me there's only Exchange to worry about but if you're using Exchange, sharepoint, AD and SQL then that's a hell of a lot of potential failure points in an enterprise's strategy.

Deatheye's picture
24
Jul
2009
0 Votes 0
Login to vote

We once had an Information

We once had an Information Store only backup with duplication job to B2D. Which worked fine but since other had the error, we also did stop that one, in case it fucks up too.
We got Exchange, Sharepoint, SQL and AD ... xD

Well right now it's going to the internal symantec bug team. Waiting for a responce from there.

Deatheye's picture
24
Jul
2009
0 Votes 0
Login to vote

Nice I'm right now looking

Nice I'm right now looking into an other error:
http://seer.entsupport.symantec.com/docs/328487.htm
Why do you think we installed that update...? Yes, excatly cause of the duplicate job. I really need vacation...

EDIT: And just found this:
http://seer.entsupport.symantec.com/docs/305109.htm

specially nice is the note at the bottom... not inteded to be fixed...
Seriously... we got envirements where we solved this creating a second backup job that does backup to an USB-Device.
That works for small envirements but allready makes it more complex then needed.
Our internal backup takes over 30h to finish. There is just no way to add another backup job cause of time issues, not even talking about the complexity added in our enviroment.

xilog's picture
24
Jul
2009
0 Votes 0
Login to vote

Jeez, I wish I'd found that

Jeez, I wish I'd found that article before.  And they're not going to fis what is a quite serious bug?  Thanks Symantec, thanks a bunch.  It's not as if your software is cheap, y'know?  If it was shareware, then hey, no problem but having spent over £4k on software for backups, I kinda expect it to work, and if there's a serious bug, that it be fixed.

Harumph.  That's put me in a sour mood for the weekend.

Deatheye's picture
26
Jul
2009
0 Votes 0
Login to vote

Just got the confirmation

Just got the confirmation that this aplies to 12.5 as well. They update the articel since it's only listening 12.0 as affected.
So the only solution at the moment is to seperate GRT-Date from other data...
I'l try to find out what excatly this means. Can I just put all the GRT-data into one job? What about system state where active directory data is stored? I can't select AD-data alone...