Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

BE2010 Slow Windows 2008 System State Backup

Created: 17 Mar 2010 | 44 comments
Tom Mucha's picture

Has anyone tried BE2010 with a 2008 server system state backup?  I'm seeing the same problem I saw with 12.5 that the system state slows to a crawl during backup.  This was supposedly fixed in 12.5sp2, but apparantly it's still an issue in 12.5 - http://seer.entsupport.symantec.com/docs/336283.htm, and I'm seeing it in 2010

Here are the results of backing up a 2008R2 Enterprise DC with BE2010 installed (latest live updates) attached to a LTO3 drive on an HP DL380G5.

c: drive backup:

Backed up 11514 files in 2660 directories.
Processed 13,728,690,754 bytes in  3 minutes and  31 seconds.
Throughput rate: 3723 MB/min
Compression Type: Hardware

System State backup:

Backed up 18 System State components
Processed 6,972,047,332 bytes in  24 minutes and  2 seconds.
Throughput rate: 277 MB/min
Compression Type: Hardware

When I backup a 2003 server I have no performance issues.  Hopefully I'm not the only one seeing issues!

Tom

Comments 44 CommentsJump to latest comment

Chuckster's picture

Hi Tom,

I'm having the same thing here on my Server 2008 R2 box.  Did you ever get any kind of resolution to this?

Cheers,

Andy

GDaddy's picture

win2008 x64 and be 2010

I'm also having that issue.  I just noticed it today but its been going on for a while. 

Tom Mucha's picture

I never found a fix for it, sorry guys.  I had a case open with support and we started down the road of basic troubleshooting, and I just wanted them to try it in their lab to see it themselves.  I was in no mood to waste my time so I just ended up closing the case. 

(This is sarcasm) I guess this is a new feature that might get fixed in the next release, just like the good ol' daylight savings time issue since Backup Exec 9.0 that would get fixed in 9.1 that we still see in BE 12.5. 
http://seer.entsupport.symantec.com/docs/336283.htm

scudulike's picture

I have exactly the same issue - BE 2010 running on SBS2008 - the job starts and runs fine until it gets to System State - it then takes about an hour to backup system state, and as soon as it's done it speeds up again.

Any sign of a fix for this?

ITTZ's picture

I have the same problem on three Win 2008 R2 Servers. The backup of the System State is very slow - all other things work fine.

Six months (since opening this discussion) without solving this problem is unacceptable.

I hope Symantec is reading this discussion - the case at Symantec is still open since weeks.

Tom Mucha's picture

Nothing guys - same issues with BE2010 R2.  If you open the link to my original post up top, they still say the issue is being looking into.

Paul Gazo's picture

Same issue with my clients.  Multiple customers, multiple servers, multiple destination devices.  Server 2008 (I don't have any R2 boxes yet) are always pathetic at System State.  I have a 600G nightly array of four servers being done that takes 7 hours and would be done in half that if it weren't for the crawling System State portion.

Every few months I do a quick search to see if there's been a whitepaper or patch or anything and continue to be let down.

BE is a great product and I have no intention of going to an alternate, but I don't see that consumer needs are actually part of the development process.  Now we've got an awesome "Typical Install" in 2010 R2 to help us get to our Wizards that will help us remember our yearly support renewals.  Thank goodness.  It'd be a shame if B2D to USB/eSATA external actually behaved itself and overwrote GRT media like it's told, or if System State wasn't so drag-butt slow.  I admit I haven't seen the DST bug in a while, but since that was around since 7.0 who knows.

WesZ's picture

Hi Technicians,

I'm also experiencing this problem with multiple servers. I'm using Backup Exec 2010 R2 right now and we have the following Windows versions in our environment:

- Windows 2003 R2
- Windows 2008 SP2
- Windows 2008 R2

All of the servers have a very slow backup of the System State. It's beginning to be a bottleneck in our Backup Window.

Byte count : 9,741,829,623 bytes
Rate : 121.00 MB/Min

We also experiencing  a slow backup for the C:\ drive on some servers, but the D:\ drive is going fine (about 1,5GB/Min).

I hope Symantec has a proper solution for this problem!

gr.

Wesley

BE-Vita's picture

Is there any solution for this issue? I also have another problem with the system state:

I just wanted to create a new backup job for one of my domain controllers in BE 2010 R2 and can't even choose the system state. I am getting an error message, that the AD domain services have to run, but of course they are running.

Any hints?

Mahesh Roja's picture

Refer this link

http://seer.entsupport.symantec.com/docs/336283.htm

I  think still symantec is working on this issue.

If this Info helps to resolve the issue please Mark as Solution

Thanks

CraigV's picture

...not sure if anybody has come right here, but I am going to be following the following TN, and wondered if anybody had come right? Got the same issue with BE 2010 R2 with all patches installed...

 

http://www.symantec.com/business/support/index?page=content&id=TECH66758

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

sbudhdeo's picture

Have the same problem ever since BE 12.  AD SystemState backups are at a crawl.  Hope a fix is released soon

Steen Kirkby's picture

Suddenly started yesterday with a domain controller that was updated to Windows 2008 R2 Service Pack 1. Backup Exec + remote agent was also updated with newest updates as of yesterday.

Other servers were updated with no problems at all (other domain controllers as well). Not sure if this is related to the installation of Win2008 R2 SP1 or installation of the Backup Exec updates. The Windows service pack failed on first try (after running almost 3 hours) but installed on second try (this time only 20-30 minutes) - maybe somethings is wrong on that server - will try to reinstall the backup exec agent on it and see if it makes a difference.

Current system state backup time:

Backed up 18 System State components
Processed 12.066.756.471 bytes in  9 hours,  23 minutes, and  1 second.
Throughput rate: 20.4 MB/min
Compression Type: Software
Software compression ratio: 1,7:1

It used to be like this (last week):

Backed up 18 System State components
Processed 9.251.657.723 bytes in  1 hour,  1 minute, and  13 seconds.
Throughput rate: 144 MB/min
Compression Type: Software
Software compression ratio: 1,7:1

It actually look like the backup job is hanging for 8 straight hours when it gets to the System State (no change in bytes counted for 8 hours) and then it continues with regular speed. All other parts of the job runs at the same speed as usual.

Steen Kirkby's picture

I just noticed this event (event id 8219 in Application Log). This event never occured before the problem started and is now logged every time I start a system state backup:

Ran out of time while expanding file specification \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4\Windows\softwaredistribution\*.*. This was being done for the WUA subscriber.

Operation:

OnPostSnapshot event

PostSnapshot Event

Context:

Execution Context: Shadow Copy Optimization Writer

Execution Context: Writer

Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

Writer Name: Shadow Copy Optimization Writer

Writer Instance ID: {de874263-3585-4a3f-9b77-bdbbdb732903}

Steen Kirkby's picture

Tried reinstalling Backup Exec Remote Agent.

Tried re-registering VSS Components

(as described here: http://www.symantec.com/business/support/index?page=content&id=TECH70486)

Still no luck

JJohnson1's picture

Windows Server 2008 R2 with BE 2010 R2, good transfer speeds before adding Service Pack 1. Used to run 1.5 hours on BU now nearly 8 hours.

Error 8220 in application log:

Log Name:      Application
Source:        VSS
Date:          3/4/2011 11:45:27 PM
Event ID:      8220
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      Server
Description:
Ran out of time while deleting files.

Operation:
   OnPostSnapshot event
   PostSnapshot Event

Context:
   Execution Context: Shadow Copy Optimization Writer
   Execution Context: Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {8bd85a38-577a-4423-8c4a-3a20f71cf15b}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

 

Ian McGuinness's picture

I think i have the same issue here now..  Only started after installing Win2008R2 SP1.

My backup of 8 Win2008R2 VMs (including 2 DCs) has gone from 7 hours to 35 hours with only a small change in total data.

rolapro's picture

I have a brand new Dell Powervault T310 running Win 2008 R2 SP1 with an LTO 3 tape drive and BE 2010 SP1 plus hotfix

I am backing up 32 Gb of data, including System State with a throughput of 1.4Gb per minute and it's taking over 10 hours instead of 20 mins.

Last night's backup for some reason only ran at 385Mb per minute

rendersr's picture

When having multiple w2k8 servers, my backupwindow was filling up solely with system backups.

It's rather normal that this is taking a lot of time.

The system state is hold in an unusual large number of small files which is keeping the backup to perform optimal.

Therefore i opted to:

  • run system backups only in weekends (longer timeframe);
  • data backups in the week.

I know this is a workaround, but in some environments this is the only thing doable..

Hopefully there is a solution soon.

Ruben

 

Ian McGuinness's picture

Just as a test last night i ran the backup without the system state and the backup droped from 35 hours to just over 3 hours.

I hope they fix this soon.

Ian

JJohnson1's picture

I removed the system state from the items to backup and backup time went from 8+ hours to less than 1.5 hours, with only 2 GB less data backed up. Of course I don't have a system state backup now. this needs to be fixed.

CraigV's picture

Apparently BE 2010 R3 will be addressing this.

In the meantime, you can consider doing a normal Windows backup of your System State, and then backing that up through BE...

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Ian McGuinness's picture

Hi Everyone,

I understand that the issue is (hopefuly) due to be fixed in R3(?) but i have been reading everything i can about the problem and looking at all of the options about possible work arrounds.  I tried this process (http://www.symantec.com/business/support/index?page=content&id=TECH70486) for "How to Re-register Volume Shadow Copy Service (VSS) Components on Windows 2008" on all of my servers yesterday and ran a full backup overnight and it compleated in about 8.5 hours.  Not as good as it should be (4 to 5 hours) but better than the 35 hours it was taking before.

I look forward to better performance after the next release but atleast now i can backup.

Ian

yak's picture

Same problem as everyone here.  Updated my R2 member servers (non DCs) with SP1 and now my backup times have doubled or tripled.  At least I now know the problem lies with the system state.  I guess I'll separate that backup or do a windows sys state backup and backup that.  Jeez we need a fix now!

 

Before:
Backed up 14 System State components
Processed 8,967,868,975 bytes in  34 minutes and  6 seconds.
Throughput rate: 251 MB/min
Compression Type: None
 
After:
Backed up 14 System State components
Processed 10,019,332,101 bytes in  4 hours,  35 minutes, and  10 seconds.
Throughput rate: 34.7 MB/min
Compression Type: None
 

 

rendersr's picture

@CraigV 

Do you have an estimated release date for the R3?

Ruben

 

Ken Putnam's picture

Last I heard was "Early April"

If this response answers your concern, please mark it as a "solution"

Ian McGuinness's picture

Yeah, I heard it was early April too.

Seriously tho, try the fix i listed 4 posts above..  it actualy worked for me and i'm now backing up in a normal window of time.
All it takes is to run the batch file on each server and a restart.

Ian

spro's picture

Hi Everyone,

same here, since I update my Win 2008 R2 DC Server to SP1, the System State Backup takes now nearly 8 hours (before arround 1 hour)...

I have tested all solutions from other comments here, but no solution worked for me e.g.

- Re-register Volume Shadow Copy Service (TECH70486)

- Remove Backup Files from SP1 with this command DISM.exe /online /Cleanup-Image /spsuperseded

- disabling anti-virus scanning

- Confirm the issue here: https://www-secure.symantec.com/connect/issue/perf...

Now I have now idea and need help to fix this problem as soon as possible...

Yes, I now this (http://www.symantec.com/business/support/index?pag...), but I won´t wait till R3 is released...

CraigV's picture

...a work-around will be to use the Backup software that comes with Windows Server 2008 R2 to do a backup to disk, and then back that up with BE.

I currently have the same sort of issue with ARCserve 15 where it takes forever to back up the System State of the 2008 servers in our data center...I look forward to a fix for BE for my sites though =/

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

spro's picture

I have on my server only one partition (C:) and a RDX Backup-Device...

so, both of them I can not use to backup the systemstate...

Do you have any sugesstions for me?

Colin@aspin's picture

I am having the same issue with this System State - at first I could not figure out the problem because it suddenly happened for no apparent reason.

I have two servers running Windows 2008 R2 and  Backup Exec 2010 both had first revision with SP1. The File Server had been running happily for a year, suddenly a couple of weeks back it went on strike.

I removed all the hotfixes and still had no joy. I have since upgraded the File Server to R2 - the performance is not still not great but better than it was. 

My SQL Server is still running the 1st revision and so far is fine - like so many things in this industry - if it works don't mess about with it!

pkh's picture

This is a known problem  Check the Known Issues section.

Steen Kirkby's picture

Hmm - my problem disappeared more or less by itself - unsure if it was a couple of reboots or a new Windows Update Hotfix that did the job. At some point I also did a liveupdate on the BE Server (again) + push of agent updates from the BE server although it didn't have any immediate effect (despite reboots after). Confused...

ZeRoC00L's picture

Steen,

Do you have BE2010R3 installed ?

I have seen a technote that the slow system state backup issues will be resolved in R3.

If this response answers your concern, please mark it as a "solution"

AstonT's picture

I've just installed a new 2008 R2 server and patched it (inc SP1).  I then installed Backupexec 2010 R2 and ran Live update.  A week later they released R3 and I installed that on top of R2.  Backup times have not improved at all.  Data backups are very quick, but when System State starts to backup - it drags on and on.  Symantec really need to get this resolved as its causing alot of trouble for a lot of techs!

AstonT's picture

A System State backup of a Windows 2008 R2 server (across the LAN) to Tape runs at 100-170Mb/min.  The System State is 10.5Gb in size, so takes around 1h 10m to backup.  With 20 servers - the backup window for System State backups alone is around 12 hours.

ZeRoC00L's picture

After updating Backup Exec, did you update the remote agent ?

If this response answers your concern, please mark it as a "solution"

AstonT's picture

He Zero - yeah - I updated all remote agents.  It's better than it was, but still very slow...

yak's picture

I installed R3 and the remote agents and now my system state backups on my 2008 R2 servers are back to normal - even slightly faster than they were pre- Server 2008 R2.

Now I can stop my convoluted - native windows sys-state backup to a remote share which is then backed up by backup exec.

ITSupp's picture

Just in case those of you had the problem with slow system state backup then upgraded to R3 and fix the speed problem but appear another problem with stalled system state backup, see this forum:

http://aka-community.symantec.com/connect/fr/forums/very-slow-performance-backup-systemstate-w2k8r2sp1

Riyaj S's picture

This issue is now fixed in,

BE2010 R3 : http://www.symantec.com/docs/TECH76051

BEWS 12.5 SP2 : http://www.symantec.com/docs/TECH66758

You may also try applying the performance patches mentioned in Technote : http://www.symantec.com/docs/TECH76051

Hope this helps.

 

Try out the new Backup Exec Support Tool that diagnoses many common issues found in Backup Exec and Symantec System Recovery environments. 
 
http://www.symantec.com/docs/TECH126539