Performance issues maybe observed when backing up the System State of a Windows 2008 Server
Updated: 09 May 2011 | 40 comments
Status Update: This issue is addressed in Backup Exec 2010 R3
Slow backup throughput is observed while backing up the System State of a Windows 2008 Operating System.
Additional information:
- This issue has been seen on 32bit and 64bit Window 2008 servers
- This issue has been seen on local and remote System State backups of Windows 2008 servers.
- This issue has been seen on Domain Controllers and Non Domain Controller servers.
- This issue has been seen with Backup Exec 12.5, 2010 and 2010 R2.
Known Issue Filed Under:
Comments
Same slow performance also with BE 2012
Hi,
new version with same problems
For example one W2008 fully patched server, system state de-dup backup extremely slow, under 30 MB/Min. Whole server differential backup takes more than 7 h.
Tape backup same server more than 1,300 MB/Min. Full backup takes less than 30 min.
-Jyrki
VirusScan Enterprise Exclusions
I found that adding these exclusion did help some.
https://kc.mcafee.com/corporate/index?page=content&id=KB66909&actp=search&viewlocale=en_US&searchid=1318380358919
Same Slow Problem
Here is what i posted on this same type of issue.
https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2
I have the same slow verify problem on multiple jobs/servers with the verify part.
The Big Question? How to speed up verifies?
BU servers are 2008 R2 with BE 2010.
BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5. Lots of horsepower.
Started migrating my user folders to a new 2008 R2 server.
B2D file size at first was 100GB.
Fulls on Sat and Diff on other days.
1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.
2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.
3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.
4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.
All other full backups run able the same as the 4th full backup.
Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.
1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.
2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.
3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.
4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.
I thought about Defraging but lets face it. You can't defrag a 10TB volume.
The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.
I have another file server that is server 2003 with SQL 2005 also installed.
When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.
And the Dup2Tape takes about 41min at 4,243MB.
So the larger that data being backed up the slower the job. I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.
Here is the something else.
As i said that i have been migrating user folder to a new server.
I still need to backup the old server (2003). The backup jobs have not changed. Still slow.
full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr
Having same issues
Here is what i posted in https://www-secure.symantec.com/connect/forums/backup-disk-extremely-slow-verify-be-2010-r2 which talks about the same issues here.
I have the same slow verify problem on multiple jobs/servers with the verify part.
The Big Question? How to speed up verifies?
BU servers are 2008 R2 with BE 2010.
BU are Dell515 with internal (10) 2TB near-line SAS in a raid 5. Lots of horsepower.
Started migrating my user folders to a new 2008 R2 server.
B2D file size at first was 100GB.
Fulls on Sat and Diff on other days.
1st full backup: Byte Count-142GB, Job Rate-1,623MB, Elapse Time-1:57hr.
2nd full backup: Byte Count-155GB, Job Rate-1,450MB, Elapse Time-2:30hr.
3rd full backup: Byte Count-158GB, Job Rate-1,361MB, Elapse Time-2:42hr.
4th full backup: Byte Count-163GB, Job Rate-795MB, Elapse Time-13:03hr.
All other full backups run able the same as the 4th full backup.
Then after the B2D job has ran I duplicate that job to a Fiber Channel LTO-4 Dual Tape library.
1st Dup2Tape: Byte Count-142GB, Job Rate-3,310MB, Elapse Time-1:10hr.
2nd Dup2Tape: Byte Count-155GB, Job Rate-2,997MB, Elapse Time-1:17hr.
3rd Dup2Tape: Byte Count-158GB, Job Rate-2,831MB, Elapse Time-1:21hr.
4th Dup2Tape: Byte Count-163GB, Job Rate-849MB, Elapse Time-3:31hr.
I thought about Defraging but lets face it. You can't defrag a 10TB volume.
The pattern I see on my two BU server is any backup job that is over about 150GB has the slow verify proccess.
I have another file server that is server 2003 with SQL 2005 also installed.
When I do a full backup of it's data (not a SQL job) it always takes about 1hour for 94GB at 1,706MB.
And the Dup2Tape takes about 41min at 4,243MB.
So the larger that data being backed up the slower the job. I would expect this but going from 142GB for 1:57hr to 163GB for 13:03hr is not normal.
Here is the something else.
As i said that i have been migrating user folder to a new server.
I still need to backup the old server (2003). The backup jobs have not changed. Still slow.
full backup: Byte Count-369GB, Job Rate-815MB, Elapse Time-14:22hr
I have the same problem but with Netbackup (and not Backup Exec)
Most backups of all clients runs at minimum of 5 MB/s (or 300 MB/min) to as high as 32 MB/s (1.92 GB/min) but "Shadow Copy Components" for a client with Windows 2008 R2 server takes for ever. It starts at @300KB/s (18MB/min) to @1000KB/s (60MB/min). The total backup for "Shadow Copy Components" is ranges to about 17 to 18GB and it takes around 4 hours to complete. This client is having SSDs (Solid State Drives) and I was expecting fast performance for everrthing including but I am looking for a solution. I saw in the current thread that it has been solved for Backup Exec but I have not seen any updates for NetBackup 7.1. Help!!!
This problem does not seem to
This problem does not seem to be resolved!
Same issue here
We have a 2008R2 server (64 bits) experiencing the same problem, using BE 2010R3. It seems like all jobs using VSS are extremely slow.
Performance drops as low as 200mb/minute when performing backup of system state, volume shadow copies or the Exchange 2003 store on both 2008, 2008R2 and 2003 servers.
We are performing a backup-2-disk backup and we expected a much higher performance then the old tape backup with BE12.5, wich was actually faster then it is now.
Please resolve this issue ASAP...
eg, a 2003 exchange server running on Windows 2003 Server:
Backed up 2 Exchange Server stores
Backed up 8 Exchange Server logs
Processed 27,321,208,308 bytes in 2 hours, 23 minutes, and 11 seconds.
Throughput rate: 182 MB/min
Same server:
Also on this server:
We experience the same problems when performing a backup of a 2008R2 server:
While the file backup (again) is much faster:
Applied patches to BE2010R3:
SP1
Hotfix 167567, 164659, 170962
Please help us out!!
Jasper
Possible solution: I also
Possible solution:
I also have been dealing with slow backups on 2008 R2 system state. Our system state was running at 100mb per minute. After the R3 update it went to 200mb per minute. Still slow. On average our system state was taking between 45min to an hour. Disabling McAfee antivirus brought the speed to 10 to 20 minutes. There is a checkbox in the McAfee On Access settings that says "opened by backup". Unchecking this allowed us to get about 1500mb per minute on the system state. I think this is going to be an average time for the way server 2008 system state is so i am happy with those results. If you run the windows backup that comes with server 2008 just on the system state, you will not get much better. Hope this helps.
SBS2011
I am running SBS2011 with Exchange 2010 with BE2010R3 and have had the same issues on speed. We ran several weeks at 1.5GB then overnight dropped to 200-400MB. I've run LiveUpdate and all software registers up-to-date. Am now at a loss for what to try next. We are only backing up about 120GB of data, but have to kill the job some mornings because it hasn't completed.
My issue is with Windows 7 workstation system state backups.
I've updated to R3 as of this past Sunday (10 Jul 2011). No change. Still getting the workstations at an Extremely Slow pace for the System State.
We just got Backup Exec, and I'm thinking we should go back to Arcserver. I'm running the same backups on both systems, with a time offset of 4 hours. The Arcserver jobs finish faster, and they go to tape directly instead of disk. I expect my daily incrementals to finish over night. BE doesn't. When it gets to a Windows 7 system state, instead of even just an hour, it's taking 6 to 8 hours for just 1 workstation.
I'm not impressed.
Kathryn
Have encountered the same with system states plus other issues
Found that the userid used by Media server to backup the Windows 7 or Server 2008 must have Administrator priviledge and be allowed to Logon as a batch job on every machine you backup.
Then this problem should run away.
The suggested fixes in the other reply thread applies only to Server 2008 prior to SP1. And does not help.
The feature that seems to help most in server 2008 R2 but might apply also to Windows 7 boxes is the disabling of the TCP Offload feature.
For Windows Server 2008 R2 the command is "netsh int tcp set global chimney=disabled"
You can verify state of feature with command "netsh int tcp show global"
I have also discovered performance issue if your backup sessions traverse a Firewall module such as Cisco 65xx
1st all in same vlan backups go up to 5.5GB/min with or without TCP offload.
But if you have the Backup server in higher security level and agent server (2008 R2) unit in second vlan lower security level with TCP offload it crawls to less than 100MB/min, then if you disable TCP offload it performs almost half speed 1.5 to 2.2 GB/min. about same rate as a Windows file transfer.
Have you tried...
kathryns,
Have you tried the suggestion noted earlier in the thread:
We have made significant changes in 2010R3 to address this issue, so I am surprised you had no change in performance.
I am checking with engineering now to see if we need any more environmental information from users expoeriencing the problem.
Slow backup of Windows 2008 R2 System State
I upgraded our Backup Exec to 2010 R3 today. The system state backups improved from 150MB/minute to 380MB/minute. This is still taking too long to back up. The C: itself backups up over 2GB/minute. Where do I go now?
update
I got an email from Symantec ADV Technical Support Engineer
"I have taken assignment of your escalated case at this time, and have reviewed the previous case notes. This appears to be related to a known issue currently under investigation by our developers. I will update you shortly with further information."
Binu It sounds like your
Binu It sounds like your issue is more on dedupe side (jobs queued).
Can you review the following technote as some changes were made in Backup Exec 2010 R3 Reclaiming space in a deduplication folder
OST Media reclamation in Backup Exec 2010 R3 (and 2010 R2 Hotfix 154003) cause jobs to go in to the queued state for an extended period
http://www.symantec.com/business/support/index?page=content&id=TECH159843
My issue is fixed with this Tech Article.
My issue is fixed with this Tech Article.
I think I spoke too soon.. it ran fine for 1 day now its back to samething.. Loading Media / Queued!!! :-(
My case has not been escalated yet
I'll be anxious to hear what the outcome of your case is. Please share when you get further into it.
any luck?
anyone had any luck with this? my jobs are going in a queued mode and sometimes it can stay there at 45 minutes to 6 hours!!!!!! and throughput is awful
@Binu Alex Are you using
@Binu Alex Are you using Dedupe option?
yes
yes
having the same issue
having the same issue
This issue is now resovled
This issue is now resovled with Backup Exec 2010 R3. Released (May 2nd, 2011).
Additionally some further performance testing found an increase in performance after the following Microsoft patches were applied to 2008 and 2008 R2 servers. The following Microsoft Patches are included in Windows 2008 R2 SP1.
A hotfix is available that improves TCP loopback latency and UDP latency in Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7
http://support.microsoft.com/kb/979612
You encounter a decrease in I/O performance under a heavy disk I/O load on a Windows Server 2008 R2-based or Windows 7-based computer
http://support.microsoft.com/kb/982383
Update
Our Backup server is 2008 SP2 64-bit with Backup Exec 2010 R3. I have verified that the I/O hotfix has helped. However, the performance is not at a point that I would consider it a fixed.
I was told by a Deduplication Engineer that there is a known bug but there is nothing else they can do at this time.
This issue isn't fixed for me
This issue isn't fixed for me on R3 (2008 SP2 64bit server). Still the system state verify is cripplingly slow at around 30mb/s making it useless as our backups never complete. If it would verify the system state at the correct speed then our backup should only take a few hours.
Very annoying and Symantec should not be reporting that this issue is fixed on R3 when it clearly isn't!
Sorry you are still seeing the issue
First, I'm sorry your issue still isn't resolved. I just want to be clear that when we say it is fixed, we do so only after extensive testing and validation in several different environments.
In your case, there are a couple of possibilities:
1. Did you push out new R3 remote agents to the boxes being protected?
2. Did you apply the MS patch recommended by SteveY?
If you did both of those, you may be experiencing something we haven't seen in house. it may be necessary for you to open a support case and let us dig deeper to find the source of the problem.
still slow
the microsoft patches wont apply on 2008 R2 SP1 any suggestions?
Yes to both
We have pushed out the new R3 client to all of our remote agents. We have also applied to the two recommended hot fixes to both the Media server and the remote agent machines.
For us the patches helped but did not resolve the problem completely. We have had to bring up another Media server to offload some of the jobs to so that our backups will complete within our window.
I would like to add that Exchange 2010 with GRT enabled shows the same symptoms of the Windows Server 2008 System State issue.
Are you using dedupe?
Just want to see if you are having the same issue as others on this thread.
Gathering Further information on Slow System State Backup issue:
Hello,
We are trying to gather further information in troubleshooting this slow System State backups. Below are a steps below for the data that is required.
1. Run a Vxgather for the Server that is experiencing the slow System State backup.
2. Size of the System State/Shadow Copy backup in a saved Job log prior to step 3.
3. Re register the VSS Writers by following technote below.
4. Gather another Vxgather and save the file.
5. Run another System State/Shadow Copy backup and save Job log to compare with first.
Note: If you would like to supply us with the above data please post here that you have data and we will contact you to gather data for review.
VSS Re-register ?
The problem only disappeared when I gave the Backup Operator userid Local Administrator group membership.
And most VSS errors disappeared. So did the slow to nill performance on that part of the backup.
PS The performance of backups are still below acceptabale for any part of my backups to tape ( 150 MB/min ) when I have seen up to 4GiGs/min on some rare instances. With 2010 before R2 and older tape devices I would get up to 2Gig/min.
Still looking for the item that is slowing it all down.
PS 2 Backup Exec 2010 R3 does not make a diff. still slow.. ( Why and Help?)
Next release of Backup Exec is currently in Beta
See the following technote if you would like to sign up for Beta:
http://www.support.com/docs/tech43417
And we will be posting more information on this issue here shortly. We hear you, and we are working on what we can do in the short term (if anything).
Confirm this, after
I installing the SP1 on Windows 2008 R2 Server!
No solution worked... job rate to RDX 52MB/min
@Symantec Support: Please fix this problem!
Same issue but solved
After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden
Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051
Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486
After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)
It's worse when verifying a deduplication job
Here's an idea of how bad.
Backup:
Verify:
It takes 2 hours and 1 minute to backup this server and 9 hours and 51 minutes to verify, of which 7 hours and 21 minutes are the System State.
Before I started using deduplication, this server took a combined total of 4.5 hours to process and now it's just under 12 hours.
These statistics are representative of all my DCs - virtual and physical; all of which are Win2008 R2 x64. All are at current Windows and BE agent patch levels and it doesn't matter if the processing is done on the client or on the media server. All backups are to a locally attached disk array.
This is a mess and I believe that it should be researched and a hotfix issued rater than waiting until the next major release.
Fix it please!
When?
Acknowledging that this is a known issue is the first step, but when is the anticipate release date for the fix, or do you have a temporary work around.
10MB / min pretty much makes the system unusable.
I have the same issue after
I have the same issue after installing the SP1 of Windows 2008 R2...
Same issue but resolved
After cleaning up as Nibbler stated it went somewhat faster but not as before.(job rate to disk 86MB/min, job rate to tape 110Mb/min)
see: https://www-secure.symantec.com/connect/forums/system-state-backups-slow-all-sudden
Created a case by support but finally got the answer that it probably will be solved in the next release as mentioned here: http://www.symantec.com/business/support/index?page=content&id=TECH76051
Couldn't leave it alone and searched further on the internet and found this:
http://www.symantec.com/business/support/index?page=content&id=TECH70486
After reregistering the VSS as described in TECH70486 it went almost as fast as before.(now job rate to disk 428MB/min, job rate to tape I did not test yet, before W2K8 R2 SP1 job rate to tape ± 550MB/min)
When is the next full release
When is the next full release of Backup Exec expected to be released?
Possible workaround
My issue is also on 64bit Windows 2008 R2 server using BE 2010 R2. It only seemed to occur when backing up the media server and all resources being backed-up are local. I changed the backup selections to not backup the local resources but instead selected the machine using the domain. This resulted in the local machine being backed-up as if it were a remote computer. Still slow, but tolerable.
My issue is on 64bit Windows
My issue is on 64bit Windows 2008 R2 servers using BE 2010 R2.
/Michael
MCP/MCTS/MCSA - www.compugame.dk
Would you like to reply?
Login or Register to post your comment.