Video Screencast Help
Search Video Help Close Back
to help
Not able to make it to Vision this year? Get a sampling in the Best of Vision on Demand group.

Performance issues maybe observed when backing up the System State of a Windows 2008 Server

Updated: 09 May 2011 | 40 comments
DanielBickford's picture
61
Votes
Login to vote
Me Too!I have the same issue
Product(s): Backup Exec - 12.x, Backup Exec - 2010, Backup Exec
Severity: Non-data threatening / major functionality
Status: Public fix available
Tech Note: More Information
Status Update: This issue is addressed in Backup Exec 2010 R3
 
Slow backup throughput is observed while backing up the System State of a Windows 2008 Operating System.
 
 
Additional information:
 
- This issue has been seen on 32bit and 64bit Window 2008 servers
 
- This issue has been seen on local and remote System State backups of Windows 2008 servers.
 
- This issue has been seen on Domain Controllers and Non Domain Controller servers.
- This issue has been seen with Backup Exec 12.5, 2010 and 2010 R2.

Comments

Jyrki's picture
19
Feb
2012
0 Votes 0
Login to vote

Same slow performance also with BE 2012

Hi,

new version with same problemsangry

For example one W2008 fully patched server, system state de-dup backup extremely slow, under 30 MB/Min. Whole server differential backup takes more than 7 h.

Tape backup same server more than 1,300 MB/Min. Full backup takes less than 30 min.

-Jyrki

navar holmes's picture
06
Jan
2012
0 Votes 0
Login to vote
navar holmes's picture
06
Jan
2012
0 Votes 0
Login to vote

Same Slow Problem

Here is what i posted on this same type of issue.

https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2

 

I have the same slow verify problem on multiple jobs/servers with the verify part.

The Big Question?  How to speed up verifies?

BU servers are 2008 R2 with BE 2010.

BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5.  Lots of horsepower.

Started migrating my user folders to a new 2008 R2 server.

B2D file size at first was 100GB.

Fulls on Sat and Diff on other days.

1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.

2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.

3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.

4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.

All other full backups run able the same as the 4th full backup.

Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.

1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.

2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.

3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.

4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.

 

I thought about Defraging but lets face it.  You can't defrag a 10TB volume.

The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.

 

I have another file server that is server 2003 with SQL 2005 also installed.

When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.

And the Dup2Tape takes about 41min at 4,243MB.

 

So the larger that data being backed up the slower the job.  I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.

Here is the something else.

As i said that i have been migrating user folder to a new server.

I still need to backup the old server (2003).  The backup jobs have not changed. Still slow.

full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr

navar holmes's picture
06
Jan
2012
0 Votes 0
Login to vote

Having same issues

Here is what i posted in https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2 which talks about the same issues here.

 

I have the same slow verify problem on multiple jobs/servers with the verify part.

The Big Question?  How to speed up verifies?

BU servers are 2008 R2 with BE 2010.

BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5.  Lots of horsepower.

Started migrating my user folders to a new 2008 R2 server.

B2D file size at first was 100GB.

Fulls on Sat and Diff on other days.

1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.

2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.

3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.

4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.

All other full backups run able the same as the 4th full backup.

Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.

1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.

2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.

3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.

4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.

 

I thought about Defraging but lets face it.  You can't defrag a 10TB volume.

The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.

 

I have another file server that is server 2003 with SQL 2005 also installed.

When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.

And the Dup2Tape takes about 41min at 4,243MB.

 

So the larger that data being backed up the slower the job.  I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.

Here is the something else.

As i said that i have been migrating user folder to a new server.

I still need to backup the old server (2003).  The backup jobs have not changed. Still slow.

full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr

RDesai's picture
04
Jan
2012
0 Votes 0
Login to vote

I have the same problem but with Netbackup (and not Backup Exec)

Most backups of all clients runs at minimum of 5 MB/s (or 300 MB/min) to as high as 32 MB/s (1.92 GB/min) but "Shadow Copy Components" for a client with Windows 2008 R2 server takes for ever. It starts at @300KB/s (18MB/min) to @1000KB/s (60MB/min). The total backup for "Shadow Copy Components" is ranges to about 17 to 18GB and it takes around 4 hours to complete. This client is having SSDs (Solid State Drives) and I was expecting fast performance for everrthing including but I am looking for a solution. I saw in the current thread that it has been solved for Backup Exec but I have not seen any updates for NetBackup 7.1. Help!!! 

rrsymantec's picture
03
Jan
2012
0 Votes 0
Login to vote

This problem does not seem to

This problem does not seem to be resolved!

jzondervan's picture
02
Nov
2011
1 Vote +1
Login to vote

Same issue here

We have a 2008R2 server (64 bits) experiencing the same problem, using BE 2010R3. It seems like all jobs using VSS are extremely slow.

Performance drops as low as 200mb/minute when performing backup of system state, volume shadow copies or the Exchange 2003 store on both 2008, 2008R2 and 2003 servers.

We are performing a backup-2-disk backup and we expected a much higher performance then the old tape backup with BE12.5, wich was actually faster then it is now.

Please resolve this issue ASAP...

 

eg, a 2003 exchange server running on Windows 2003 Server:

Backed up 2 Exchange Server stores
Backed up 8 Exchange Server logs
Processed 27,321,208,308 bytes in  2 hours,  23 minutes, and  11 seconds.
Throughput rate: 182 MB/min

Same server:

Backed up 14 System State components
Processed 1,166,787,157 bytes in  3 minutes and  8 seconds.
Throughput rate: 355 MB/min
Compression Type: None

Also on this server:

Backed up 138864 files in 9961 directories.
Processed 101231775511 bytes in  42 minutes and  13 seconds.
Throughput rate: 2287 MB/min
Compression Type: None

 

We experience the same problems when performing a backup of a 2008R2 server:

Backed up 14 System State components
Processed 12,508,917,789 bytes in  22 minutes and  1 second.
Throughput rate: 542 MB/min
Compression Type: None

While the file backup (again) is much faster:

Backed up 1110 files in 767 directories.
2 items were skipped.
Processed 614944418 bytes in  28 seconds.
Throughput rate: 1257 MB/min
Compression Type: None

 

Applied patches to BE2010R3:

SP1
Hotfix 167567, 164659, 170962

 

Please help us out!! 

Jasper

TDOC12's picture
14
Dec
2011
0 Votes 0
Login to vote

Possible solution: I also

Possible solution:

I also have been dealing with slow backups on 2008 R2 system state.  Our system state was running at 100mb per minute.  After the R3 update it went to 200mb per minute.  Still slow.  On average our system state was taking between 45min to an hour.  Disabling McAfee antivirus brought the speed to 10 to 20 minutes.  There is a checkbox in the McAfee On Access settings that says "opened by backup".  Unchecking this allowed us to get about 1500mb per minute on the system state.  I think this is going to be an average time for the way server 2008 system state is so i am happy with those results.  If you run the windows backup that comes with server 2008 just on the system state, you will not get much better.  Hope this helps.

SJenkins's picture
07
Sep
2011
0 Votes 0
Login to vote

SBS2011

I am running SBS2011 with Exchange 2010 with BE2010R3 and have had the same issues on speed. We ran several weeks at 1.5GB then overnight dropped to 200-400MB. I've run LiveUpdate and all software registers up-to-date. Am now at a loss for what to try next. We are only backing up about 120GB of data, but have to kill the job some mornings because it hasn't completed.

kathryns's picture
13
Jul
2011
0 Votes 0
Login to vote

My issue is with Windows 7 workstation system state backups.

I've updated to R3 as of this past Sunday (10 Jul 2011). No change. Still getting the workstations at an Extremely Slow pace for the System State.

We just got Backup Exec, and I'm thinking we should go back to Arcserver. I'm running the same backups on both systems, with a time offset of 4 hours. The Arcserver jobs finish faster, and they go to tape directly instead of disk. I expect my daily incrementals to finish over night. BE doesn't. When it gets to a Windows 7 system state, instead of even just an hour, it's taking 6 to 8 hours for just 1 workstation.

I'm not impressed.

 

Kathryn

sergelab's picture
13
Jul
2011
0 Votes 0
Login to vote

Have encountered the same with system states plus other issues

Found that the userid used by Media server to backup the Windows 7 or Server 2008 must have Administrator priviledge and be allowed to Logon as a batch job on every machine you backup.

Then this problem should run away.

The suggested fixes in the other reply thread applies only to Server 2008 prior to SP1. And does not help.

The feature that seems to help most in server 2008 R2 but might apply also to Windows 7 boxes is the disabling of the TCP Offload feature.

For Windows Server 2008 R2 the command is "netsh int tcp set global chimney=disabled"

You can verify state of feature with command "netsh int tcp show global"

I have also discovered performance issue if your backup sessions traverse a Firewall module such as Cisco 65xx

1st all in same vlan backups go up to 5.5GB/min with or without TCP offload.

But if you have the Backup server in higher security level and agent server (2008 R2) unit in second vlan lower security level with TCP offload it crawls to less than 100MB/min,  then if you disable TCP offload it performs almost half speed 1.5 to 2.2 GB/min. about same rate as a Windows file transfer.

DanielBickford's picture
13
Jul
2011
0 Votes 0
Login to vote

Have you tried...

kathryns,

Have you tried the suggestion noted earlier in the thread:

Additionally some further performance testing found an increase in performance after the following Microsoft patches were applied to 2008 and 2008 R2 servers. The following Microsoft Patches are included in Windows 2008 R2 SP1. 

A hotfix is available that improves TCP loopback latency and UDP latency in Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7

http://support.microsoft.com/kb/979612

 You encounter a decrease in I/O performance under a heavy disk I/O load on a Windows Server 2008 R2-based or Windows 7-based computer

http://support.microsoft.com/kb/982383

 

We have made significant changes in 2010R3 to address this issue, so I am surprised you had no change in performance.

I am checking with engineering now to see if we need any more environmental information from users expoeriencing the problem.

ptodd's picture
07
Jul
2011
0 Votes 0
Login to vote

Slow backup of Windows 2008 R2 System State

I upgraded our Backup Exec to 2010 R3 today.  The system state backups improved from 150MB/minute to 380MB/minute.  This is still taking too long to back up.  The C: itself backups up over 2GB/minute.  Where do I go now?

Binu Alex's picture
11
May
2011
0 Votes 0
Login to vote

update

 

I got an email from Symantec ADV Technical Support Engineer

"I have taken assignment of your escalated case at this time, and have reviewed the previous case notes. This appears to be related to a known issue currently under investigation by our developers. I will update you shortly with further information."

 

Steve Y's picture
11
May
2011
1 Vote +1
Login to vote

Binu It sounds like your

Binu It sounds like your issue is more on dedupe side (jobs queued).

Can you review the following technote as some changes were made in Backup Exec 2010 R3 Reclaiming space in a deduplication folder

 

OST Media reclamation in Backup Exec 2010 R3 (and 2010 R2 Hotfix 154003) cause jobs to go in to the queued state for an extended period

http://www.symantec.com/business/support/index?page=content&id=TECH159843 

Binu Alex's picture
16
May
2011
0 Votes 0
Login to vote

My issue is fixed with this Tech Article.

My issue is fixed with this Tech Article.

 

I think I spoke too soon..  it ran fine for 1 day now its back to samething.. Loading Media / Queued!!!  :-(

Maynard01's picture
11
May
2011
0 Votes 0
Login to vote

My case has not been escalated yet

I'll be anxious to hear what the outcome of your case is. Please share when you get further into it.

Binu Alex's picture
06
May
2011
0 Votes 0
Login to vote

any luck?

anyone had any luck with this?  my jobs are going in a queued mode and sometimes it can stay there at 45 minutes to 6 hours!!!!!!  and throughput is awful

Steve Y's picture
06
May
2011
0 Votes 0
Login to vote

@Binu Alex   Are you using

@Binu Alex   Are you using Dedupe option?

Binu Alex's picture
09
May
2011
0 Votes 0
Login to vote

yes

yes

Binu Alex's picture
04
May
2011
0 Votes 0
Login to vote

having the same issue

having the same issue

Steve Y's picture
04
May
2011
1 Vote +1
Login to vote

This issue is now resovled

This issue is now resovled with Backup Exec 2010 R3.  Released (May 2nd, 2011).

 

Additionally some further performance testing found an increase in performance after the following Microsoft patches were applied to 2008 and 2008 R2 servers.  The following Microsoft Patches are included in Windows 2008 R2 SP1.

 

A hotfix is available that improves TCP loopback latency and UDP latency in Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7

http://support.microsoft.com/kb/979612

 

 

You encounter a decrease in I/O performance under a heavy disk I/O load on a Windows Server 2008 R2-based or Windows 7-based computer

http://support.microsoft.com/kb/982383

Maynard01's picture
06
May
2011
0 Votes 0
Login to vote

Update

Our Backup server is 2008 SP2 64-bit with Backup Exec 2010 R3. I have verified that the I/O hotfix has helped. However, the performance is not at a point that I would consider it a fixed.

I was told by a Deduplication Engineer that there is a known bug but there is nothing else they can do at this time.

LMDB's picture
07
May
2011
1 Vote +1
Login to vote

This issue isn't fixed for me

This issue isn't fixed for me on R3 (2008 SP2 64bit server). Still the system state verify is cripplingly slow at around 30mb/s making it useless as our backups never complete. If it would verify the system state at the correct speed then our backup should only take a few hours.

Very annoying and Symantec should not be reporting that this issue is fixed on R3 when it clearly isn't!

Daniel Bickford's picture
09
May
2011
0 Votes 0
Login to vote

Sorry you are still seeing the issue

First, I'm sorry your issue still isn't resolved.  I just want to be clear that when we say it is fixed, we do so only after extensive testing and validation in several different environments.

In your case, there are a couple of possibilities:

1. Did you push out new R3 remote agents to the boxes being protected?

2. Did you apply the MS patch recommended by SteveY?

If you did both of those, you may be experiencing something we haven't seen in house.  it may be necessary for you to open a support case and let us dig deeper to find the source of the problem.

sergelab's picture
09
Jun
2011
0 Votes 0
Login to vote

still slow

the microsoft patches wont apply on 2008 R2 SP1 any suggestions?

Maynard01's picture
09
May
2011
0 Votes 0
Login to vote

Yes to both

We have pushed out the new R3 client to all of our remote agents. We have also applied to the two recommended hot fixes to both the Media server and the remote agent machines.

For us the patches helped but did not resolve the problem completely. We have had to bring up another Media server to offload some of the jobs to so that our backups will complete within our window.

I would like to add that Exchange 2010 with GRT enabled shows the same symptoms of the Windows Server 2008 System State issue.

DanielBickford's picture
13
May
2011
0 Votes 0
Login to vote

Are you using dedupe?

Just want to see if you are having the same issue as others on this thread.

MKayler's picture
17
Mar
2011
0 Votes 0
Login to vote

Gathering Further information on Slow System State Backup issue:

Hello,

     We are trying to gather further information in troubleshooting this slow System State backups. Below are a steps below for the data that is required.

1. Run a Vxgather for the Server that is experiencing the slow System State backup.

  • Navigate to C:\Program Files\Backup Exec then run the Vxgather.exe
  • Save the .cab file.

2. Size of the System State/Shadow Copy backup in a saved Job log prior to step 3.

3. Re register the VSS Writers by following technote below.

4. Gather another Vxgather and save the file.

5. Run another System State/Shadow Copy backup and save Job log to compare with first.

 

Note: If you would like to supply us with the above data please post here that you have data and we will contact you to gather data for review.

sergelab's picture
09
Jun
2011
1 Vote +1
Login to vote

VSS Re-register ?

The problem only disappeared when I gave the Backup Operator userid Local Administrator group membership.

 

And most VSS errors disappeared. So did the slow to nill performance on that part of the backup.

 

PS   The performance of backups are still below acceptabale for any part of my backups to tape ( 150 MB/min ) when I have seen up to 4GiGs/min on some rare instances.  With 2010 before R2 and older tape devices I would get up to 2Gig/min.

 

Still looking for the item that is slowing it all down.

 

PS 2  Backup Exec 2010 R3 does not make a diff.  still slow.. ( Why and Help?)

 

DanielBickford's picture
17
Mar
2011
0 Votes 0
Login to vote

Next release of Backup Exec is currently in Beta

See the following technote if you would like to sign up for Beta:

http://www.support.com/docs/tech43417

 

And we will be posting more information on this issue here shortly.  We hear you, and we are working on what we can do in the short term (if anything).

spro's picture
16
Mar
2011
0 Votes 0
Login to vote

Confirm this, after

I installing the SP1 on Windows 2008 R2 Server!

No solution worked... job rate to RDX 52MB/min

@Symantec Support: Please fix this problem!

JeWa's picture
09
Mar
2011
0 Votes 0
Login to vote

Same issue but solved

After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden

Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051

Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486

After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)

Brian Longworth's picture
08
Mar
2011
0 Votes 0
Login to vote

It's worse when verifying a deduplication job

Here's an idea of how bad.

Backup:

  • Data:   3386 MB/min
  • System State:   456 MB/min

Verify:

  • Data:  2061 MB/min
  • System State:    22.6 MB/min

It takes 2 hours and 1 minute to backup this server and 9 hours and 51 minutes to verify, of which 7 hours and 21 minutes are the System State.

Before I started using deduplication, this server took a combined total of 4.5 hours to process and now it's just under 12 hours.

These statistics are representative of all my DCs - virtual and physical; all of which are Win2008 R2 x64. All are at current Windows and BE agent patch levels and it doesn't matter if the processing is done on the client or on the media server. All backups are to a locally attached disk array.

This is a mess and I believe that it should be researched and a hotfix issued rater than waiting until the next major release.

Fix it please!

moonshdw8's picture
08
Mar
2011
0 Votes 0
Login to vote

When?

Acknowledging that this is a known issue is the first step, but when is the anticipate release date for the fix, or do you have a temporary work around.

10MB / min pretty much makes the system unusable. 

David G.'s picture
08
Mar
2011
0 Votes 0
Login to vote

I have the same issue after

I have the same issue after installing the SP1 of Windows 2008 R2...

JeWa's picture
09
Mar
2011
1 Vote +1
Login to vote

Same issue but resolved

After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden

Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051

Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486

After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)

cov's picture
01
Mar
2011
3 Votes +3
Login to vote

When is the next full release

When is the next full release of Backup Exec  expected to be released?

UrbanLegendPA's picture
18
Feb
2011
0 Votes 0
Login to vote

Possible workaround

My issue is also on 64bit Windows 2008 R2 server using BE 2010 R2. It only seemed to occur when backing up the media server and all resources being backed-up are local. I changed the backup selections to not backup the local resources but instead selected the machine using the domain. This resulted in the local machine being backed-up as if it were a remote computer. Still slow, but tolerable.

Michael Pultz-Hansen's picture
02
Feb
2011
1 Vote +1
Login to vote

My issue is on 64bit Windows

My issue is on 64bit Windows 2008 R2 servers using BE 2010 R2.

 

/Michael

MCP/MCTS/MCSA - www.compugame.dk