Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Performance issues maybe observed when backing up the System State of a Windows 2008 Server

Created: 26 Jan 2011 • Updated: 09 May 2011 | 41 comments
DanielBickford's picture
74
Votes
Login to vote
Me Too!I have the same issue
Product(s): Backup Exec - 12.x, Backup Exec, Backup Exec - 2010
Severity: Non-data threatening / major functionality
Status: Public fix available
Tech Note: More Information
Status Update: This issue is addressed in Backup Exec 2010 R3
 
Slow backup throughput is observed while backing up the System State of a Windows 2008 Operating System.
 
 
Additional information:
 
- This issue has been seen on 32bit and 64bit Window 2008 servers
 
- This issue has been seen on local and remote System State backups of Windows 2008 servers.
 
- This issue has been seen on Domain Controllers and Non Domain Controller servers.
- This issue has been seen with Backup Exec 12.5, 2010 and 2010 R2.

Comments 41 CommentsJump to latest comment

eduncan22's picture

BE 12.5 SP4 does not resolve this issue.

Remote Agent Client version [installed on Windows 2008 R2 SP1] is 12.5 Rev 2213.175

Throughput on  the System State backup is  375 MB/min

Processed 9,915,131,926 bytes in  25 minutes and  13 seconds.
 

0
Login to vote
Jyrki's picture

Hi,

new version with same problemsangry

For example one W2008 fully patched server, system state de-dup backup extremely slow, under 30 MB/Min. Whole server differential backup takes more than 7 h.

Tape backup same server more than 1,300 MB/Min. Full backup takes less than 30 min.

-Jyrki

0
Login to vote
navar holmes's picture

Here is what i posted on this same type of issue.

https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2

I have the same slow verify problem on multiple jobs/servers with the verify part.

The Big Question?  How to speed up verifies?

BU servers are 2008 R2 with BE 2010.

BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5.  Lots of horsepower.

Started migrating my user folders to a new 2008 R2 server.

B2D file size at first was 100GB.

Fulls on Sat and Diff on other days.

1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.

2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.

3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.

4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.

All other full backups run able the same as the 4th full backup.

Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.

1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.

2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.

3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.

4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.

I thought about Defraging but lets face it.  You can't defrag a 10TB volume.

The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.

I have another file server that is server 2003 with SQL 2005 also installed.

When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.

And the Dup2Tape takes about 41min at 4,243MB.

So the larger that data being backed up the slower the job.  I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.

Here is the something else.

As i said that i have been migrating user folder to a new server.

I still need to backup the old server (2003).  The backup jobs have not changed. Still slow.

full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr

0
Login to vote
navar holmes's picture

Here is what i posted in https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2 which talks about the same issues here.

I have the same slow verify problem on multiple jobs/servers with the verify part.

The Big Question?  How to speed up verifies?

BU servers are 2008 R2 with BE 2010.

BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5.  Lots of horsepower.

Started migrating my user folders to a new 2008 R2 server.

B2D file size at first was 100GB.

Fulls on Sat and Diff on other days.

1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.

2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.

3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.

4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.

All other full backups run able the same as the 4th full backup.

Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.

1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.

2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.

3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.

4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.

I thought about Defraging but lets face it.  You can't defrag a 10TB volume.

The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.

I have another file server that is server 2003 with SQL 2005 also installed.

When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.

And the Dup2Tape takes about 41min at 4,243MB.

So the larger that data being backed up the slower the job.  I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.

Here is the something else.

As i said that i have been migrating user folder to a new server.

I still need to backup the old server (2003).  The backup jobs have not changed. Still slow.

full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr

0
Login to vote
RDesai's picture

Most backups of all clients runs at minimum of 5 MB/s (or 300 MB/min) to as high as 32 MB/s (1.92 GB/min) but "Shadow Copy Components" for a client with Windows 2008 R2 server takes for ever. It starts at @300KB/s (18MB/min) to @1000KB/s (60MB/min). The total backup for "Shadow Copy Components" is ranges to about 17 to 18GB and it takes around 4 hours to complete. This client is having SSDs (Solid State Drives) and I was expecting fast performance for everrthing including but I am looking for a solution. I saw in the current thread that it has been solved for Backup Exec but I have not seen any updates for NetBackup 7.1. Help!!! 

0
Login to vote
jzondervan's picture

We have a 2008R2 server (64 bits) experiencing the same problem, using BE 2010R3. It seems like all jobs using VSS are extremely slow.

Performance drops as low as 200mb/minute when performing backup of system state, volume shadow copies or the Exchange 2003 store on both 2008, 2008R2 and 2003 servers.

We are performing a backup-2-disk backup and we expected a much higher performance then the old tape backup with BE12.5, wich was actually faster then it is now.

Please resolve this issue ASAP...

eg, a 2003 exchange server running on Windows 2003 Server:

Backed up 2 Exchange Server stores
Backed up 8 Exchange Server logs
Processed 27,321,208,308 bytes in  2 hours,  23 minutes, and  11 seconds.
Throughput rate: 182 MB/min

Same server:

Backed up 14 System State components
Processed 1,166,787,157 bytes in  3 minutes and  8 seconds.
Throughput rate: 355 MB/min
Compression Type: None

Also on this server:

Backed up 138864 files in 9961 directories.
Processed 101231775511 bytes in  42 minutes and  13 seconds.
Throughput rate: 2287 MB/min
Compression Type: None

We experience the same problems when performing a backup of a 2008R2 server:

Backed up 14 System State components
Processed 12,508,917,789 bytes in  22 minutes and  1 second.
Throughput rate: 542 MB/min
Compression Type: None

While the file backup (again) is much faster:

Backed up 1110 files in 767 directories.
2 items were skipped.
Processed 614944418 bytes in  28 seconds.
Throughput rate: 1257 MB/min
Compression Type: None

Applied patches to BE2010R3:

SP1
Hotfix 167567, 164659, 170962

Please help us out!! 

Jasper

+1
Login to vote
TDOC12's picture

Possible solution:

I also have been dealing with slow backups on 2008 R2 system state.  Our system state was running at 100mb per minute.  After the R3 update it went to 200mb per minute.  Still slow.  On average our system state was taking between 45min to an hour.  Disabling McAfee antivirus brought the speed to 10 to 20 minutes.  There is a checkbox in the McAfee On Access settings that says "opened by backup".  Unchecking this allowed us to get about 1500mb per minute on the system state.  I think this is going to be an average time for the way server 2008 system state is so i am happy with those results.  If you run the windows backup that comes with server 2008 just on the system state, you will not get much better.  Hope this helps.

0
Login to vote
SJenkins's picture

I am running SBS2011 with Exchange 2010 with BE2010R3 and have had the same issues on speed. We ran several weeks at 1.5GB then overnight dropped to 200-400MB. I've run LiveUpdate and all software registers up-to-date. Am now at a loss for what to try next. We are only backing up about 120GB of data, but have to kill the job some mornings because it hasn't completed.

0
Login to vote
kathryns's picture

I've updated to R3 as of this past Sunday (10 Jul 2011). No change. Still getting the workstations at an Extremely Slow pace for the System State.

We just got Backup Exec, and I'm thinking we should go back to Arcserver. I'm running the same backups on both systems, with a time offset of 4 hours. The Arcserver jobs finish faster, and they go to tape directly instead of disk. I expect my daily incrementals to finish over night. BE doesn't. When it gets to a Windows 7 system state, instead of even just an hour, it's taking 6 to 8 hours for just 1 workstation.

I'm not impressed.

Kathryn

0
Login to vote
sergelab's picture

Found that the userid used by Media server to backup the Windows 7 or Server 2008 must have Administrator priviledge and be allowed to Logon as a batch job on every machine you backup.

Then this problem should run away.

The suggested fixes in the other reply thread applies only to Server 2008 prior to SP1. And does not help.

The feature that seems to help most in server 2008 R2 but might apply also to Windows 7 boxes is the disabling of the TCP Offload feature.

For Windows Server 2008 R2 the command is "netsh int tcp set global chimney=disabled"

You can verify state of feature with command "netsh int tcp show global"

I have also discovered performance issue if your backup sessions traverse a Firewall module such as Cisco 65xx

1st all in same vlan backups go up to 5.5GB/min with or without TCP offload.

But if you have the Backup server in higher security level and agent server (2008 R2) unit in second vlan lower security level with TCP offload it crawls to less than 100MB/min,  then if you disable TCP offload it performs almost half speed 1.5 to 2.2 GB/min. about same rate as a Windows file transfer.

0
Login to vote
DanielBickford's picture

kathryns,

Have you tried the suggestion noted earlier in the thread:

Additionally some further performance testing found an increase in performance after the following Microsoft patches were applied to 2008 and 2008 R2 servers. The following Microsoft Patches are included in Windows 2008 R2 SP1. 

A hotfix is available that improves TCP loopback latency and UDP latency in Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7

http://support.microsoft.com/kb/979612

 You encounter a decrease in I/O performance under a heavy disk I/O load on a Windows Server 2008 R2-based or Windows 7-based computer

http://support.microsoft.com/kb/982383

We have made significant changes in 2010R3 to address this issue, so I am surprised you had no change in performance.

I am checking with engineering now to see if we need any more environmental information from users expoeriencing the problem.

0
Login to vote
ptodd's picture

I upgraded our Backup Exec to 2010 R3 today.  The system state backups improved from 150MB/minute to 380MB/minute.  This is still taking too long to back up.  The C: itself backups up over 2GB/minute.  Where do I go now?

0
Login to vote
Binu Alex's picture

I got an email from Symantec ADV Technical Support Engineer

"I have taken assignment of your escalated case at this time, and have reviewed the previous case notes. This appears to be related to a known issue currently under investigation by our developers. I will update you shortly with further information."

0
Login to vote
Steve Y's picture

Binu It sounds like your issue is more on dedupe side (jobs queued).

Can you review the following technote as some changes were made in Backup Exec 2010 R3 Reclaiming space in a deduplication folder

OST Media reclamation in Backup Exec 2010 R3 (and 2010 R2 Hotfix 154003) cause jobs to go in to the queued state for an extended period

http://www.symantec.com/business/support/index?page=content&id=TECH159843 

+1
Login to vote
Binu Alex's picture

My issue is fixed with this Tech Article.

I think I spoke too soon..  it ran fine for 1 day now its back to samething.. Loading Media / Queued!!!  :-(

0
Login to vote
Maynard01's picture

I'll be anxious to hear what the outcome of your case is. Please share when you get further into it.

0
Login to vote
Binu Alex's picture

anyone had any luck with this?  my jobs are going in a queued mode and sometimes it can stay there at 45 minutes to 6 hours!!!!!!  and throughput is awful

0
Login to vote
Steve Y's picture

This issue is now resovled with Backup Exec 2010 R3.  Released (May 2nd, 2011).

Additionally some further performance testing found an increase in performance after the following Microsoft patches were applied to 2008 and 2008 R2 servers.  The following Microsoft Patches are included in Windows 2008 R2 SP1.

A hotfix is available that improves TCP loopback latency and UDP latency in Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7

http://support.microsoft.com/kb/979612

You encounter a decrease in I/O performance under a heavy disk I/O load on a Windows Server 2008 R2-based or Windows 7-based computer

http://support.microsoft.com/kb/982383

+1
Login to vote
Maynard01's picture

Our Backup server is 2008 SP2 64-bit with Backup Exec 2010 R3. I have verified that the I/O hotfix has helped. However, the performance is not at a point that I would consider it a fixed.

I was told by a Deduplication Engineer that there is a known bug but there is nothing else they can do at this time.

0
Login to vote
LMDB's picture

This issue isn't fixed for me on R3 (2008 SP2 64bit server). Still the system state verify is cripplingly slow at around 30mb/s making it useless as our backups never complete. If it would verify the system state at the correct speed then our backup should only take a few hours.

Very annoying and Symantec should not be reporting that this issue is fixed on R3 when it clearly isn't!

+1
Login to vote
Daniel Bickford's picture

First, I'm sorry your issue still isn't resolved.  I just want to be clear that when we say it is fixed, we do so only after extensive testing and validation in several different environments.

In your case, there are a couple of possibilities:

1. Did you push out new R3 remote agents to the boxes being protected?

2. Did you apply the MS patch recommended by SteveY?

If you did both of those, you may be experiencing something we haven't seen in house.  it may be necessary for you to open a support case and let us dig deeper to find the source of the problem.

0
Login to vote
sergelab's picture

the microsoft patches wont apply on 2008 R2 SP1 any suggestions?

0
Login to vote
Maynard01's picture

We have pushed out the new R3 client to all of our remote agents. We have also applied to the two recommended hot fixes to both the Media server and the remote agent machines.

For us the patches helped but did not resolve the problem completely. We have had to bring up another Media server to offload some of the jobs to so that our backups will complete within our window.

I would like to add that Exchange 2010 with GRT enabled shows the same symptoms of the Windows Server 2008 System State issue.

0
Login to vote
DanielBickford's picture

Just want to see if you are having the same issue as others on this thread.

0
Login to vote
MKayler's picture

Hello,

     We are trying to gather further information in troubleshooting this slow System State backups. Below are a steps below for the data that is required.

1. Run a Vxgather for the Server that is experiencing the slow System State backup.

  • Navigate to C:\Program Files\Backup Exec then run the Vxgather.exe
  • Save the .cab file.

2. Size of the System State/Shadow Copy backup in a saved Job log prior to step 3.

3. Re register the VSS Writers by following technote below.

4. Gather another Vxgather and save the file.

5. Run another System State/Shadow Copy backup and save Job log to compare with first.

Note: If you would like to supply us with the above data please post here that you have data and we will contact you to gather data for review.

0
Login to vote
sergelab's picture

The problem only disappeared when I gave the Backup Operator userid Local Administrator group membership.

And most VSS errors disappeared. So did the slow to nill performance on that part of the backup.

PS   The performance of backups are still below acceptabale for any part of my backups to tape ( 150 MB/min ) when I have seen up to 4GiGs/min on some rare instances.  With 2010 before R2 and older tape devices I would get up to 2Gig/min.

Still looking for the item that is slowing it all down.

PS 2  Backup Exec 2010 R3 does not make a diff.  still slow.. ( Why and Help?)

+1
Login to vote
DanielBickford's picture

See the following technote if you would like to sign up for Beta:

http://www.support.com/docs/tech43417

And we will be posting more information on this issue here shortly.  We hear you, and we are working on what we can do in the short term (if anything).

0
Login to vote
spro's picture

I installing the SP1 on Windows 2008 R2 Server!

No solution worked... job rate to RDX 52MB/min

@Symantec Support: Please fix this problem!

0
Login to vote
JeWa's picture

After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden

Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051

Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486

After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)

0
Login to vote
Brian Longworth's picture

Here's an idea of how bad.

Backup:

  • Data:   3386 MB/min
  • System State:   456 MB/min

Verify:

  • Data:  2061 MB/min
  • System State:    22.6 MB/min

It takes 2 hours and 1 minute to backup this server and 9 hours and 51 minutes to verify, of which 7 hours and 21 minutes are the System State.

Before I started using deduplication, this server took a combined total of 4.5 hours to process and now it's just under 12 hours.

These statistics are representative of all my DCs - virtual and physical; all of which are Win2008 R2 x64. All are at current Windows and BE agent patch levels and it doesn't matter if the processing is done on the client or on the media server. All backups are to a locally attached disk array.

This is a mess and I believe that it should be researched and a hotfix issued rater than waiting until the next major release.

Fix it please!

0
Login to vote
moonshdw8's picture

Acknowledging that this is a known issue is the first step, but when is the anticipate release date for the fix, or do you have a temporary work around.

10MB / min pretty much makes the system unusable. 

0
Login to vote
David G.'s picture

I have the same issue after installing the SP1 of Windows 2008 R2...

0
Login to vote
JeWa's picture

After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden

Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051

Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486

After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)

+1
Login to vote
cov's picture

When is the next full release of Backup Exec  expected to be released?

+3
Login to vote
UrbanLegendPA's picture

My issue is also on 64bit Windows 2008 R2 server using BE 2010 R2. It only seemed to occur when backing up the media server and all resources being backed-up are local. I changed the backup selections to not backup the local resources but instead selected the machine using the domain. This resulted in the local machine being backed-up as if it were a remote computer. Still slow, but tolerable.

0
Login to vote
Michael Pultz-Hansen's picture

My issue is on 64bit Windows 2008 R2 servers using BE 2010 R2.

/Michael

MCP/MCTS/MCSA - www.compugame.dk

+1
Login to vote