Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Backup Exec 2012 backup very slow (catalog problem?)

Created: 28 Jan 2013 • Updated: 29 May 2013 | 37 comments
This issue has been solved. See solution.

We've been having some problems with the speed of our backup. The setup is the following:

physical BE2012 server coonected to disk storage via fibre and connected to VmWare enviroment via Fibre as well. When backup up our file and print server to DeDupe location on our disk storage it takes a very long time (+19hr) for 900GB of data. It seems to hang on updating catalog.

History: Everything working fine and backupjob of all servers (1.4TB) runnning at speeds of between 4000MB/min to 6000MB/min. Then from one day to the other (no windows or BE updates done) the speed slowed down to between 600MB/min and 900MB/min.

Things allready tried: delete and recreate the backup job. update to SP1a. Thick all the boxes under the catalog tab and set to trucate after 4 weeks. run database maintenance at a time when no backup jobs are running. split up the job into different jobs (this alleviated the problem a bit)

After splitting the job up so that the exchange server, the SQL server and the file and print server had different jobs I noticed that the exchange and SQL were backing up at normal speeds (+4000MB/min) but the file and print server was still taking 18Hrs to backup. a bit more digging and a couple of days later I noticed that the data of file and print was backing up at speeds of +4000MB/min itself but it was hanging (going verrry slow) on updating the catalog at the end of the job while the size stayed unchanged.

My counterpart in our other location the other side of Australia is having the same problem for the exact same setup. (his problem started about 2 weeks before mine) He has been testing with going from full daily backups to incremental but the same problem stays.

We do not have a bottleneck problem with our disks, speed of our servers or any other hardware but it seems to come backup to the updating of the catalog for the file and print server. (my counterpart has also tried to uninstall and reinstall BE2012 but no sollution there either)

Does anyone have an idea how to clean, compact or do anything else with the catalog to get it back to normal speed? (catalog folder is currently 9GB with 16GB available)

Thanks in advance

Comments 37 CommentsJump to latest comment

CraigV's picture

Hi,

Is BE 2012 loaded with SP1a and all other available patches?

Are you using SAN Transport mode for your backups as well? If not, check the TN below:

http://www.symantec.com/business/support/index?pag...

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

SaWa's picture

As mentioned in the first post sp1a is installed. We have indeed loaded all other available patches.

We have it configured exactly as the link shows.

It has always been working fine up untill one day where it decided to go slow. it hangs on the "changing catalog" at the end of the job. which makes me think that there ight be a problem with the catalogs or the database (but I have not found any info yet on how to troubleshoot the db problems.)

Cheers.

pkh's picture

The "updating catalog" phase is when BE enumerates the individual items for GRT restores.  If you have Exchange and files on the same VM, then each mail item and each file/folder needs to be enumerated and this can take some time.

If you suspect a corrupt catalog, you can try creating a new catalog

1) stop all BE services

2) rename the catalog directory under the BE directory to catalog.old

3) create a new catalog directory

4) start all BE services

5) try your job again.

SaWa's picture

still the same problem, I've renamed the folder, restarted the server, ran the backup again. It backed up all data to dedupe at a rate of 6000MB/min and once the updating catalogs starts nothing happens anymore and you see the Job rate going down steadily. files in the catalog folder are populating very very slowly.

Any other idea's would be welcome.

Lejre's picture

I have the same problem. Suddenly, updating catalogs is very slow. I think it happened after a microsoft update. Do you know if there is added microsoft updates to your backup exec 2012 server in January?

Sorry I forgot to read, that you havent applyed any new windows updates.

I have tried to rename the catalog folder, with no luck. Perhaps you are in luck...... Looking forward for your result.

SaWa's picture

I've noticed something else, When I go into the Job history, the start time and end time are showing in US date format instead of AU date format. (MM/DD/YYYY) all the rest in Backup exec and windows is showing AU date format. Has anyone has an idea if that might cause an issue? (grasping straws here).

cheers.

SaWa's picture

My counterpart in the other location has tried to format the Dedupe storage and backup exec hard drive as well with no positive outcome. the problem stays the same.

cheers

_grimm's picture

I have exactly the same issue,

VMware backups via fibre channel to a media server with local dedupe store (9TB). Usual performance is 4000-6000MB/s with catalog updates taking anywhere between 1-20 mins at the end of the job, depending on which applications are being GRT processed.

After installing the latest hotfixes 199866 and 200433 (to get orphan snapshot deletion functionality) everything has turned sour. I can backup hundreds of gigabytes and all looks good then randomely performance takes a nosedive. I have one system that has been updating catalogs for 18 hours and seems to process around 25 files per second - normal performance would see 1000's of files per second cataloged.

Hardware is not the issue as both source and destinations are high performance and low utilisation. I suspect some sort of memory leak or buffer issue that is throttling performance but I can't see anything useful in the debug logs.

The only fix seems to be to cancel running jobs and restart BUE services then all work fine again for a random time before slowing right down again.

I am currently waiting on Symantec responding to my call and will update if I get anywhere. Very frustrating and not much in the way of successful backups for over a week now.

JBAS's picture

Hi _grimm

I have exactly the same issue.

I am running 7 BE2012 Servers worldwide and had the issue on two of them. On one server I solved the issue by completely delete and recreate the Dedup Store. Everything runs fine now for one week. On the other server I did exactly the same without any luck. every night, Full & incremental backups, take about 24 to 48 hours, because of a slow catalog job of one server (it is always the same server).

Did you already get an answer of Symantec Support?

Nick Maxwell's picture

Hello,

I am having the same issue.  Backups to dedupe folder are slow specifically in the catalog process.  Job use to run in 6 hours, now runs in 24+ hours.

I have an open case with Symantec which is now going on its' 4th week (case opened 1/23).  To this date, I have not been provided with any explanation or fix.  All we have done is recreated the catalogs folder and generated some debug logs.  Issue I have now, is that a catalog of the dedupe folder will not even complete.  I am also finding that catalog jobs for tapes are running very slow as well.

Our next steps are to remove the dedupe folder and then readd it.  I am told this process will not remove the backup contents in the dedupe folder.  We are performing some test restores with tape to be sure we can deal with the removal or potential loss of the dedupe folder.

Anyone else have any recommendations or possible fix for the issues?

Thanks,

Nick

KK80's picture

Hello,

I have the same problem here. Please let me know if you have answer from symantec support.

Thanks,

Kareene

Gurvinder Rait's picture

Please run a backup of the file server VM to Backup to Disk folder. We have a case reported so we will come out with a tech note shortly.

Nick Maxwell's picture

How is that a viable solution to this issue?  That compromises the customer's entire backup plan and severely cuts the retention time for backup data.

This is truly an issue since hotfix 199866.  The performance issue is specifically with the catalog process of a VMware Virtual Machine when using a dedupe folder as the backup destination.

m.rieder's picture

We're experiencing the same issue. But we have the slow updating catalog problem on every Disk target, not just only for the deduplication folder. Our problem came up with the installation of the latest hotfixes 199866 and 200433.

We uninstalled the Hotfix 200433, but the problem persists. The Hotfix 199866 cannot be uninstalled, so we opened a support case and we're still waiting for a solution.

Regards,
Matthias

Lejre's picture

The problem has been there since the beginning of January. It is very disappointing that Symantec has not found a solution yet. We have been forced to convert our backup routine for file-based backup. Vmware backup with GRT sucks. Symantec fix your software.

Hopes Symantec finds a solution as quickly as possible.

SGSCIT's picture

We have the same issue here.  All the latest patches applied and it started around January.  I ran patching last week to hopefully fix the issue but no love yet from Symantec.

m.rieder's picture

We have to say that we are really disappointed about the current support. We have open the ticket now nearly 4 weeks and in the last week we didn't received any status updates about our case.

Nick Maxwell's picture

My case was opened on 1/23/13.  I have also worked with their SMB support which appear to be in the mode of a response or recommendation every 2-3 days.  I find it very disheartening that this issue has been dealt with so poorly.

Mauro Cerutti's picture

Same problem here guys.

Someone know when Symantec plan to solve it? It's a big issue that compromise all the company backup strategy

LorettoHealth's picture

We too are having the same issue! Waiting for an update/fix

LorettoHealth's picture

We too are having the same issue! Waiting for an update/fix. Sorry not sure why it duplicated my message

v.todorov's picture

Hi, we are having the same issue this week. 

Is it possible to set a full backup with GRT and incremental without in Backup Exec 2012. 

Can anyone confirm if this is working as instructed by Colin Weaver here:

https://www-secure.symantec.com/connect/forums/exc...

Back to your original problem.

Symantec are aware of an issue where the updating catalogs phase can take extended periods to run and are working to try an minimize the effect.
In the meantime the options are run your Exchange Full backups slightly more often (which you already know works from your accidental fix) or setup your Incremental template with GRT not enabled - but leave GRT enabled on the full template and then make sure that your Exchange deleted message retention (Microsoft setting not a Backup Exec setting) is longer than the interval between your full backups so that your restore options are
1) Use the full set with GRT
2) Use the Outlook mechanism, to restore deleted message for the incremental dates
3) Use an RSG process if the above 2 options can't be used
4) Just restore the full and incrementals without using Granular Recovery if you have a disaster that affects the whole server.

SaWa's picture

Still no update from Symantec.

Latest update not working either.

Will keep updating as soon as I have an update.

v.todorov's picture

I solved my issue (at least for now). For the past few weeks I did not tuch the backup exec. Everything is backing up smoothly. 

First I defragmented the mail database since it had 50% white space.

The support team advised me to take the following steps in reagrds to the Backup Exec and Exchange 2010:

- full backup should be scheduled every week

- incremental backup throughout the week (every day currently)

- system state and Microsoft information store should be run in separate jobs and should not overlap

- full and incremental backup should be targeted on the same storage 

- in Backup Exec under 'Advanced Open File' - everythins switched off

- under 'Microsoft Exchange' - everything switched on

- install EMC on BE Server

- the circular logging for the databases should be disabled 

- maintenance for exchange databases should run at a different schedule from the Backup Exec jobs

Hope this helps anyone.

SONGYOD.KH's picture

I just face with same issue.

VMware backup using SAN connection to Dedupe storage.

The guest is file server with 1.2TB sized and many files in it.

The backup is still updating catalog more than one days.

I just cancel it but it's still updating catalog and status is cancel pending.

robnicholson's picture

Same issue here after installing latest hot fixes except we're still on BE2010. Full Exchange backup at the weekend backed up 65GB @ 908MB/min. Incremental backup last night backed up just 1.7GB at 9MB/min. It's always the incremental backups that sit at the updating catalogs prompt for hours & hours.

Counting down the days until our new non-Symantec backup system goes live...

Rob.

SaWa's picture

We finaly have a temporary solution from Symantec.

It seems that there is a problem with a dll (pdvfsprotocol.dll) they installed a new dll and now the speed is to an acceptable 3.5GB/min backup to de-dupe (still not the 6GB/min I used to have but a lot better) They said they are still finalising the dll for general public and I'm not allowed to install any Symantec updates untill they bring out the final update and contact me again.

So for anyone out there with this issue I would advise to create a ticket with Symantec and let it roll from there.

Lets hope the final fix takes it back to original speeds and it won't take to long to get it.

Hope the rest of you get there as well.

G.

SOLUTION
Sapphy's picture

I am experiencing the exact same issue.

Any word on when this fix will be included an update?

Thanks,

Sapphy..

cfrancopy's picture

Hi,

Same issue here! did anyone get the fix?

Thanks,

Carlos Franco

Mikkel Andreasen's picture

HI,

We are also experiencing this exact issue (extremely slow "updating catalogs" on GRT based backups on fileservers).

Are there any news of availability of this hotfix or who to contact to get the updated DLL?

Sincerely

Mikkel Andreasen

csotoc's picture

Hello,

We need a solution from Symantec. 

now We are reinstalling VMware tools and Symantec Agent, then we´ll try to do backup job again. 

let's see what happen.. 

please, if anyone have a solution let my know. 

thanls in advance,

Carlos

Meindert's picture

Addendum for other users experiencing slow backups.

I have noticed slow backups with BE2010sp3 combined with (backups of) servers with databases (SQL, Exchange) and/or Active Directory, usually in combination with deduplication. After a long analysis, logging and such, I noticed that the BE server had rather long wait periods, in which it would wait for responses from the client. Some further googling led me to the culprit: Microsoft Forefront Endpoint Protection. You would expect them to know how to get the exclusions right on their own anti virus software, but forget it. The on-access scan engine slows backups to a crawl. 

The following article finally put me on the right path:

https://kc.mcafee.com/corporate/index?page=content...

Solution

Add the following exclusions to your antivirus. I found that it works for FEP, but it might well also work for Sophos, NOD32, Norman, Panda, F-Secure, Kaspersky, Symantec and others. 

Add Low Risk Processes policies and exclusions for Backup Exec.

Step 1 - Add exclusions to AV on your Backup Exec Server

  1. Open your antivirus scanner and go to the Exclusions part of the settings.
  2. Add exclusions for the following processes:

    C:\Program Files\Symantec\Backup Exec\beremote.exe 
    C:\Program Files\Symantec\Backup Exec\beserver.exe 
    C:\Program Files\Symantec\Backup Exec\bengine.exe 
    C:\Program Files\Symantec\Backup Exec\benetns.exe  
    C:\Program Files\Symantec\Backup Exec\pvlsvr.exe 
    C:\Program Files\Symantec\Backup Exec\BkUpexec.exe

    NOTE: Change the path as appropriate, depending on which root volume the Media Server or Remote Agent has been installed to.

Step 2 - Exclude Backup Exec paths from scanning

  1. Open your antivirus scanner and add the following path to the Exclusions (including subfolders):

    C:\Program Files\Symantec\Backup Exec\
    C:\Program Files (x86)\Symantec\Backup Exec\

When configuring your settings, keep your eye open for other settings that could complement the exclusions. For example, McAfee has an option to ignore files that are opened for backup.

Step 3 - Add exclusions based on the used product

Active Directory, Exchange, SQL - they all have their own required exclusions. Configure these exclusions using the following articles:

https://kc.mcafee.com/corporate/index?page=content...

Once you've set these, there's a good chance things will improve drastically.

Symantec, you would do well to make a dedicated article explaining what exclusions to add - not just for your own products, but also other products that BackupExec has to work with. 

cfrancopy's picture

I also have McAfee and my google searches lead me to the same KB articule. I would not know if that solved the problem as I also did a clean installation on my backup media and afterwards I finally got the backup jobs running well again.

csotoc's picture

In our case, antivirus is fully disabled.

Deduplication backup’s delays more than 48 hours, the same backups to normal disk or tape takes no more than 10 hous.

Symantec recommended me to disable GRT (http://www.symantec.com/business/support/index?pag...) but what we really need is to have fast restore from disk.

Hope to have a solution soon!

Any help?

OP's picture

Hi csotoc,

we've had the same problem and gave up after 5 month of timewasting with symantecsupport.

They all tried to explain, that slow duplicate FROM a dedup folder or restore from the same folder is not a problem - it's the way it's designed.

I've got the impression that Symantec is no longer recommending dedup storage as a primary backuptarget - only as the second or third stage.

So we changed back to 1. stage backup2disk and we are duplicating from there to tape or other external media.

I'm totally disappointed from the quality of the dedup and even more from the support.