Backup Exec 2012 backup very slow (catalog problem?)
We've been having some problems with the speed of our backup. The setup is the following:
physical BE2012 server coonected to disk storage via fibre and connected to VmWare enviroment via Fibre as well. When backup up our file and print server to DeDupe location on our disk storage it takes a very long time (+19hr) for 900GB of data. It seems to hang on updating catalog.
History: Everything working fine and backupjob of all servers (1.4TB) runnning at speeds of between 4000MB/min to 6000MB/min. Then from one day to the other (no windows or BE updates done) the speed slowed down to between 600MB/min and 900MB/min.
Things allready tried: delete and recreate the backup job. update to SP1a. Thick all the boxes under the catalog tab and set to trucate after 4 weeks. run database maintenance at a time when no backup jobs are running. split up the job into different jobs (this alleviated the problem a bit)
After splitting the job up so that the exchange server, the SQL server and the file and print server had different jobs I noticed that the exchange and SQL were backing up at normal speeds (+4000MB/min) but the file and print server was still taking 18Hrs to backup. a bit more digging and a couple of days later I noticed that the data of file and print was backing up at speeds of +4000MB/min itself but it was hanging (going verrry slow) on updating the catalog at the end of the job while the size stayed unchanged.
My counterpart in our other location the other side of Australia is having the same problem for the exact same setup. (his problem started about 2 weeks before mine) He has been testing with going from full daily backups to incremental but the same problem stays.
We do not have a bottleneck problem with our disks, speed of our servers or any other hardware but it seems to come backup to the updating of the catalog for the file and print server. (my counterpart has also tried to uninstall and reinstall BE2012 but no sollution there either)
Does anyone have an idea how to clean, compact or do anything else with the catalog to get it back to normal speed? (catalog folder is currently 9GB with 16GB available)
Thanks in advance
Comments 26 Comments • Jump to latest comment
Hi,
Is BE 2012 loaded with SP1a and all other available patches?
Are you using SAN Transport mode for your backups as well? If not, check the TN below:
http://www.symantec.com/business/support/index?pag...
Thanks!
If you find this is a solution, please mark it as such.
As mentioned in the first post sp1a is installed. We have indeed loaded all other available patches.
We have it configured exactly as the link shows.
It has always been working fine up untill one day where it decided to go slow. it hangs on the "changing catalog" at the end of the job. which makes me think that there ight be a problem with the catalogs or the database (but I have not found any info yet on how to troubleshoot the db problems.)
Cheers.
The "updating catalog" phase is when BE enumerates the individual items for GRT restores. If you have Exchange and files on the same VM, then each mail item and each file/folder needs to be enumerated and this can take some time.
If you suspect a corrupt catalog, you can try creating a new catalog
1) stop all BE services
2) rename the catalog directory under the BE directory to catalog.old
3) create a new catalog directory
4) start all BE services
5) try your job again.
still the same problem, I've renamed the folder, restarted the server, ran the backup again. It backed up all data to dedupe at a rate of 6000MB/min and once the updating catalogs starts nothing happens anymore and you see the Job rate going down steadily. files in the catalog folder are populating very very slowly.
Any other idea's would be welcome.
I'll give that a try and update when done.
thanks.
I have the same problem. Suddenly, updating catalogs is very slow. I think it happened after a microsoft update. Do you know if there is added microsoft updates to your backup exec 2012 server in January?
Sorry I forgot to read, that you havent applyed any new windows updates.
I have tried to rename the catalog folder, with no luck. Perhaps you are in luck...... Looking forward for your result.
I've noticed something else, When I go into the Job history, the start time and end time are showing in US date format instead of AU date format. (MM/DD/YYYY) all the rest in Backup exec and windows is showing AU date format. Has anyone has an idea if that might cause an issue? (grasping straws here).
cheers.
My counterpart in the other location has tried to format the Dedupe storage and backup exec hard drive as well with no positive outcome. the problem stays the same.
cheers
I have exactly the same issue,
VMware backups via fibre channel to a media server with local dedupe store (9TB). Usual performance is 4000-6000MB/s with catalog updates taking anywhere between 1-20 mins at the end of the job, depending on which applications are being GRT processed.
After installing the latest hotfixes 199866 and 200433 (to get orphan snapshot deletion functionality) everything has turned sour. I can backup hundreds of gigabytes and all looks good then randomely performance takes a nosedive. I have one system that has been updating catalogs for 18 hours and seems to process around 25 files per second - normal performance would see 1000's of files per second cataloged.
Hardware is not the issue as both source and destinations are high performance and low utilisation. I suspect some sort of memory leak or buffer issue that is throttling performance but I can't see anything useful in the debug logs.
The only fix seems to be to cancel running jobs and restart BUE services then all work fine again for a random time before slowing right down again.
I am currently waiting on Symantec responding to my call and will update if I get anywhere. Very frustrating and not much in the way of successful backups for over a week now.
Hi _grimm
I have exactly the same issue.
I am running 7 BE2012 Servers worldwide and had the issue on two of them. On one server I solved the issue by completely delete and recreate the Dedup Store. Everything runs fine now for one week. On the other server I did exactly the same without any luck. every night, Full & incremental backups, take about 24 to 48 hours, because of a slow catalog job of one server (it is always the same server).
Did you already get an answer of Symantec Support?
Hello,
I am having the same issue. Backups to dedupe folder are slow specifically in the catalog process. Job use to run in 6 hours, now runs in 24+ hours.
I have an open case with Symantec which is now going on its' 4th week (case opened 1/23). To this date, I have not been provided with any explanation or fix. All we have done is recreated the catalogs folder and generated some debug logs. Issue I have now, is that a catalog of the dedupe folder will not even complete. I am also finding that catalog jobs for tapes are running very slow as well.
Our next steps are to remove the dedupe folder and then readd it. I am told this process will not remove the backup contents in the dedupe folder. We are performing some test restores with tape to be sure we can deal with the removal or potential loss of the dedupe folder.
Anyone else have any recommendations or possible fix for the issues?
Thanks,
Nick
Hello,
I have the same problem here. Please let me know if you have answer from symantec support.
Thanks,
Kareene
Please run a backup of the file server VM to Backup to Disk folder. We have a case reported so we will come out with a tech note shortly.
How is that a viable solution to this issue? That compromises the customer's entire backup plan and severely cuts the retention time for backup data.
This is truly an issue since hotfix 199866. The performance issue is specifically with the catalog process of a VMware Virtual Machine when using a dedupe folder as the backup destination.
We're experiencing the same issue. But we have the slow updating catalog problem on every Disk target, not just only for the deduplication folder. Our problem came up with the installation of the latest hotfixes 199866 and 200433.
We uninstalled the Hotfix 200433, but the problem persists. The Hotfix 199866 cannot be uninstalled, so we opened a support case and we're still waiting for a solution.
Regards,
Matthias
The problem has been there since the beginning of January. It is very disappointing that Symantec has not found a solution yet. We have been forced to convert our backup routine for file-based backup. Vmware backup with GRT sucks. Symantec fix your software.
Hopes Symantec finds a solution as quickly as possible.
We have the same issue here. All the latest patches applied and it started around January. I ran patching last week to hopefully fix the issue but no love yet from Symantec.
We have to say that we are really disappointed about the current support. We have open the ticket now nearly 4 weeks and in the last week we didn't received any status updates about our case.
My case was opened on 1/23/13. I have also worked with their SMB support which appear to be in the mode of a response or recommendation every 2-3 days. I find it very disheartening that this issue has been dealt with so poorly.
Same problem here guys.
Someone know when Symantec plan to solve it? It's a big issue that compromise all the company backup strategy
We too are having the same issue! Waiting for an update/fix
We too are having the same issue! Waiting for an update/fix. Sorry not sure why it duplicated my message
Hi, we are having the same issue this week.
Is it possible to set a full backup with GRT and incremental without in Backup Exec 2012.
Can anyone confirm if this is working as instructed by Colin Weaver here:
https://www-secure.symantec.com/connect/forums/exc...
Still no update from Symantec.
Latest update not working either.
Will keep updating as soon as I have an update.
I solved my issue (at least for now). For the past few weeks I did not tuch the backup exec. Everything is backing up smoothly.
First I defragmented the mail database since it had 50% white space.
The support team advised me to take the following steps in reagrds to the Backup Exec and Exchange 2010:
- full backup should be scheduled every week
- incremental backup throughout the week (every day currently)
- system state and Microsoft information store should be run in separate jobs and should not overlap
- full and incremental backup should be targeted on the same storage
- in Backup Exec under 'Advanced Open File' - everythins switched off
- under 'Microsoft Exchange' - everything switched on
- install EMC on BE Server
- the circular logging for the databases should be disabled
- maintenance for exchange databases should run at a different schedule from the Backup Exec jobs
Hope this helps anyone.
I just face with same issue.
VMware backup using SAN connection to Dedupe storage.
The guest is file server with 1.2TB sized and many files in it.
The backup is still updating catalog more than one days.
I just cancel it but it's still updating catalog and status is cancel pending.
Would you like to reply?
Login or Register to post your comment.