Video Screencast Help

BackupExec 2012 SP4 problems

Created: 26 Mar 2014 • Updated: 08 Apr 2014 | 60 comments

Hello,

after installing SP4 for BE2012 the server runs poorly. The jobs cannot access the devices properly. Restarting the services work for a short time.  I think we have to open a call. We have the caso options installed... Anyone the same problems?

Comments 60 CommentsJump to latest comment

CraigV's picture

Hi,

 

In a CASO environment, the update should be:

1. CASO...

2. All MMS servers.

Have you made sure all MMS servers (and their remote servers) have been updated? Have you tried to reapply SP4 on the server/s giving you issues?

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Dieter14's picture

We are having similar issues after the upgrade to SP4.  Anyother suggestions other than a wipe and reload?

CraigV's picture

...logging a call with Symantec MIGHT help. The OP on this post simply ended up looking elsewhere. If you can't uninstall SP4 you are left with reinstalling from scratch (BE that is) or getting help from support.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

HNeumann's picture

Hi,

yes, thats right. We always apply the patches on the CASO first. After reapply SP4 nothing happend. After a while of backing up all our jobs in the deduplication store hang around in the queue saying they cannot access the deduplication store.
Than we have to restart the services, wait 3 hours than Backup will run until the next error ist happening. We now will open a case and we will look for an other product. Backup Exec is annoying me :-( 

Can we uninstall the SP4?

HN

CraigV's picture

Hi HN,

 

No, the latest releases are uninstallable.

The only other suggestions I have would be to do a repair of the BEDB using beutility.exe, and also check the Known Issues section on Connect to see if your particular error is listed there.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

HNeumann's picture

Hi CraigV,

can i do a repair installation of BE? Do you think than can help? Or first, i will try to repair the DB. Thanks for your ideas.

HN

CraigV's picture

Yes you can try that too. Do the repair of BE first, and then the repair of the BEDB.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

HNeumann's picture

Hello Craig,

i have repaired the DB an made a repair installation. I hope it works now. I will report. But i think the problem is the deduplication store. This technique makes me crazy from day one we installed the deduplication option...

Hanging jobs on the dedup isn´t quite new to me. After rebooting the system it will work for a week or two, but there are always problems...

HN

HNeumann's picture

This is it. We will uninstall the program and evaluate another solution. BE is the worst software out there. All jobs are hanging around saying waitung for device etc. Its not true the software makes us crazy. Since SP4 there no dys without rebooting the caso server hoping the Engine can access the Dedup folder...

There is not a week without problems. Backup is so important but with this software you cannot backup important files.
Nothing works its so ridiculous

Greetings

HNeumann's picture

We opened a call with symantec, the engeneer is clicking around, he has no ideas where the error is coming from. Tomorrow we are getting another phone call :-( It´s more than annoying...

CraigV's picture

...you're entited to ask for an escalation to a higher level of support! If this hasn't been done, do so and see how this pans out.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Syntex's picture

Wow! I had the exact same issue after SP4 and I thought I was alone. Communication errors accessing the deduplication folder (0xe000fe30 - A communications failure has occurred). I spent a huge amount of time looking thru all the TECH notes, but found no joy and it all happened after SP4. Tech support also sent me some unrelated Tech notes which had me believing this was new.

This is a CASO/MMS environment where I followed the SP4 instructions to apply to CASO first and then to the MMS's servers.

To make a long story short I tried; Repairing and re applying sp4. that did not work. Tried dumping my Deduplication Folder and creating a new one from scratch, that didnt work. Tried dumping my jobs and recreated from scratch, that didnt work, Retargeting all the Jobs to B2D, Success!

Now I was in a pickle because without deduplication the 12TB of storage would not last long with the retention I needed. So, biting the bullet and throwing a hale mary I backed up the data and catalogs just incase and reinstalled everything from scratch applying SP4 before anything else.

Everything is communicating again so, I guess it was not a network issue as tech support suggest which is not completely there fault as this may be a new issue directly related to SP4. I will spend the next week getting all the jobs in order and back to normal.

 

 

sbora's picture

Hello HNeumann, Dieter14, 

If you have open up support cases, can you plesae send me the case id.

Thanks,
Sandesh

Dieter14's picture

06385613 &  06182939  are the two open tickets with Symantec.  It appears that the upgrade destabilized the Backup Exec  Job engine service.  After working with support yesterday and uploading logs, I restarted the server.  Confirmed all backup exec services that are required were running. 

  • Backup Exec Agent Browser
  • Backup Exec Device & Media Service
  • Backup Exec Job Engine
  • Backup Exec Management Service
  • Backup Exec Remote Agent For Windows
  • Backup Exec Server

 

Kicked off a backup job that was running when I logged off.  Sometime between 5:30 PM EST and 9:30 PM EST the backup exec job engine system service stopped as at 9:31 PM EST is when I see in the log that the service was no longer responding.  

Restarted all the services again this morning and kicked off a job.  We'll see how it goes.    

sbora's picture

Hello Dieter14,

I will get these cases escalated to further investigate the issue.

Thanks,

sbora's picture

We investigated the issues reported by Dieter14 and they are not related to SP4. 

HNeumann, if you got a chance to create case with our support please send me the case id so that we can investigate the cause.

Thanks,

jwoppert's picture

Everything was working GREAT before service pack 4 was installed. A typical duplicate job used to take 5 minutes to complete, now sits in 'queued' status for 15 minutes or longer. Service Pack seems to have messed everything up. DO NOT INSTALL SERVICE PACK 4 for Symantec Backup Exec. 

I have noticed that the first initial backup job which I have going to an iSCSI NAS usually completes at normal time. The duplicate jobs that I do (which go to a removable disk via eSATA) now take forever to complete. This was never the case before the installation of service pack 4. 

Someone please help.

marcodoy's picture

I have the exact same problem. Before SP4 everything was working before and the jobs queued and stuck... The engine seems to have been broken by SP4. Restarting all services is the only way to kick new jobs and may fail again by freeezing,.

Artegic's picture

SP4 did not cause any additional breakage for me. It even fixed one issue. (RALUS crashing on kernel 3.x) Apart from that, no change, neither for better nor for worse.

JaapB's picture

Same over here. After installing BE 2012 SP4 jobs got stuck and queued. Restarting the services temporarily solves the problem. Since upgrading to SP4 we had to restart the services every day.

We are running BE 2012 SP4 on a windows 2008 R2 server.

 

VJware's picture

Hello JaapB, marcodoy & jwoppert,

Would you pls log formal support cases for us to investigate as currently there aren't any known issues as mentioned above caused by installing SP4. (in fact almost all cases which we investigated led to a different cause than SP4)

Pls do PM/email me the case numbers once they have been logged. Thanks.

jwoppert's picture

Just sent you the case number that I have filed with Symantec.

marcodoy's picture

I sent you the case tooo.

jwoppert's picture

Still have the same issues. Seems like the backup engine keeps failing. I can restart the services and it works for a little bit and then back to not working.

LSW's picture

Same story for me.  Pre SP4 everything was working quite well in comparison to a year ago.  Now performance very slow, jobs queuing when idle devices are available.  Duplicate jobs to tape sometimes run but then the tape stays in the drive rather than returning to the library with BE believing a job is still running when it isn't.  Only option is to restart the services so all the other jobs that are running then fail.  Job Engine failing on a daily basis.

I don't see how we have any other option but to look at another product.  It took us a long time to get BE 2012 stable and it feels like all that work has been undone.  I cant keep spending all day looking after BE 2012.  Case open with support.

 

 

Mike55's picture

Had the same problem with my library.

Try running C:\Program Files\Symantec\Backup Exec\tapeinst.exe through to the end.

I know it doesn't make any sense.

3sprovider's picture

Last Sunday (5/4/2014) we roll SP4 to about a dozen different customers.   Exact Same problem, jobs are hanging.  We can restart the Job Services but that is getting old.   We have 4 different cases open and the symantec engineers are attempt to gather logs. 

LSW's picture

We were backing up to disk, duplicating the backup sets to tape and then duplicating the original backup set to dedupe.  After a week of very unreliable backups I recreated all our jobs to backup to tape only.  Since then not one failure.  It's not what we want to do but to get some reliable backups is a step in the right direction.

So I don't know if I'm alone with this but the issue would appear to be with the backup to disk or duplicating of backup sets from what I've seen so far with SP4.  Support have been trying to get hold of me so I'll be dropping them a line shortly.  I'm away from the office next week so I'm reluctant to let them do too much whilst I'm getting some backups, even if it is a step backwards.

lukeb123's picture

I see it's another successful release from Symantec.....

I WAS going to apply this to try resolve my issue of not being able to backup Hyper V machines with 'writer errors' (of course, there are none..) but I think I'll give it a miss.

I've found with 2012 that it's better to live with the smaller problems rather than install a SP and create 10 more major ones.  What on earth is going on over there?  It should be illegal to release such junk.

DaveyMG's picture

I've had similar problems since installing SP4 with jobs staying in queued state for a long time. Also had a lot of random "lost communication" failures on jobs but that seems to have cleared since I updated all of the remote agents.

In my experience BE2012 works well when everything is working properly but it becomes a nightmare to manage when things go wrong. There is little control available on the job queue. After a failure I end up with a lot of jobs being ready, waiting for resources, but I can't control which ones run next. I can't cancel them until they start to run and I can't reschedule or hold them to run later. When the backup schedule is disrupted by a failure I need to be able to control the queue so I can ensure that critical backups take priority once things are running again.

DD1984's picture

We have exactly the same issues after upgrading to SP4
- Backup stay queued forever
- Services restart works for half a day.
- DB repair helped for a couple of days. (case closed)
- New case made and now they lost all of the other logs (so lets start from the beginning)

Any solutions known?

Tony77's picture

We have also exactly same problems after installing SP4 to backup exec 2012.

I spent lot of time to tracking the problem and finally found some kind of "solution". Of course it´s possible that the problem in your case is different.

 

I made only following thing:

-I stopped Backup Exec Agent Browser service and disabled it.

-Restart the backup server

 

Now backup jobs are running again. (We cannot made any backups before that)

I know that there are other coin side to stop that service, but the backups are more important.

We aren´t opened support case to Symantec yet.

 

 

 

 

Oleg_Oz's picture

We have the same problem..

Waiting for support(case opened). Waiting for solution..

jwoppert's picture

Have not heard anything back from Symantec. Sent in jobs logs and Backup Exec diagnostic logs. Still having the same issue. WIll restart services, will work for a little bit.

One job on Friday which was a full backup was held running for over 2 days and stuck in the same position. Because this job was stuck, it failed to backup other servers over the weekend. Symantec keeps telling me that it is nothign to do with Service Pack 4, but I say they are wrong. ALL of these problems happened at the end of the installation of service pack 4. I am about to uninstall Backup Exec and reinstall it and upgrade to service pack 3 again.

Anyone else have any input?

Robert H.'s picture

Please send your open case number and we will review it.  You can do this through private message or e-mail.

Thanks

Mike55's picture

This software is just awful.

I ran on SP1 and SP2 for a long time, finally getting them somewhere close to stable.

I upgraded to SP4 and a week or so later my jobs just started queuing.  The deduplication resource just cycled from active to online without any jobs getting processed.  The usual solutions, restarting services and rebooting, didn’t help for long.

I built an entirely new BE2012 SP4 server and identical hardware starting with the SP4 media and remade all of the jobs again, but severely limited deduplication and GRT usage.  That ran okay in testing, but once I ran all of the production jobs, they started queuing.  At first I assumed that it was the deduplication that was holding things up, but then it seemed like it might be the tape library.  I ran tapeinst.exe as I saw suggested.  It didn’t make sense to me and it didn’t seem to make any changes, but jobs started running once more.

I then developed problems with deduplication jobs.  “Please insert overwritable media"  I don’t have any control over that.  I can’t manage deduplication media in this version.  After looking around online, I saw that some people had luck with running a deduplication reclaim space sequence (ref TECH130103) which I’ve done since 2010.  That resolved the problem with all but 1 job.  From what I remember, I deleted many of my backup sets related to that job and more.  Again, I have no direct control over media in this version.  I ran another reclaim.  I deleted and remade the failing job.  That seemed to get me past that problem.

Recently, the backup exec deduplication engine crashes repeatedly.  Something I haven’t seen before.  Usually it’s the BEengine that crashes.  I can’t resolve it and I’m giving up.  I don’t feel like going back to SP2 or SP3, so I abandoned deduplication today.

Deduplication and backup to disk storage seriously needs cleanup utilities.  When the BEengine crashes or something else fails, material is left in those resources that can’t be removed.  This material wastes space and causes cataloging to fail.  Cataloging job times should be measured in minutes, not hours or days.  I have very powerful hardware, use it.

Those of you out there that have more than just a couple of TBs of seasoned storage resources, just try running some index and catalog jobs.  If you have storage that has been used for a good while, it’s likely that after the unbelievably long job execution it will terminate with an error.  The error will usually be that it didn’t like some of the media.  Something you really can’t do much about in this version.  Do this with caution for those with old deduplication storage, since the job will likely run for days.  Keep in mind, that this is what you would have to do to recover if you lost the catalog.

Mark Stephens's picture

I'm having issues at ALPA as well, have opened 2 cases with support, problems persyst.

Had run BE2012 SP2 level for a long time, had issue with BEDB db & 4GB limit.  Noticed that SP3 was to correct that runtime SQL issue for db size.  Went to apply SP3, noticed SP4 was out, so applied it  (it was early April) and the problems have been ongoing since.

Today when checking I have 5 jobs active but ZERO activity, see attached.    I am thankful for finding this article.  Why, when I take the effort to open several cases (see below, after alot of troubleshooting on my own) aren't I directed to this forum article by Support?  Instead I've been through several iterations of check this job, recreate this job, change these options.  Had initial case open for post SP4 issues, somehow morphed into Exchange issue.  Currently have cases open for both SQL and Hyper-V and I've mentioned that I didn't think its related to any of these agents/backup jobs/efforts it's something post SP4.....no reply on that.  There will be now & this will be escallated.

 

Case history- Initial Case May 19-

Case # 06606201 - Since applying SP4, BE having a lot of issues. Jobs getting cancelled. Previous jobs not running. has been created.  This morphed into Exchange issue (the job was suffering from other issues BE)  This was closed, however issues persyst so opened another case.

Case # 06632888 - SQL & Hyper-V related issues. Current case.

Seems of late the reply from support is that a new product (BE2014) is due to be released in June and that I'll love that!  Perhaps they should focus on fixing what has been broken by SERVICE PACK 4- one they released with what looks like little testing in real world environment.

Come on MAN, like many others have said, recreating jobs (that previously ran well) and Terminating BE Services is getting old, hopefully a solution is found soon.

Frustrated with better things to do in Northern VA / DC

Mark

 

BE2012-SP4.jpg
marcodoy's picture

Same happened for me.

The initial case was closed and told it it is a Hyper-v issue and they need to open the case with a different team.

I cannot believe how your story is same as mine.

DaveyMG's picture

On my BE 2012 server I've noticed that bengine.exe is always running at 50% CPU on task manager. Also, jobs that appear to have hung do perform some work if left long enough. I suspect that bengine.exe may be spinning its wheels for some reason and being prevented from performing actual work.

Ken Johnson II's picture

DaveyMG,

I've noticed something similar since SP4. The bengine.exe process is using 25% of the CPU even though no jobs are running. Even if I put the job queue on hold it continues to use right around 25% of the quad-core CPU. We have never waited around long enough to see if all the jobs eventually complete.

Has anyone tried uninstalling BE completely and installing just up to SP3? Did you have to restore a pre-SP4 copy of the database or was it able to use the post-SP4 database? I'd really hate to scrap it all and start from scratch, but that's what I'm looking at as the next step. I'd open a case, but this thread so far seems to indicate that opening a case with Symantec doesn't help.

Ken Johnson II's picture

Well, we went ahead with uninstalling without any guidance. This is what worked in our case.

  1. Uninstall Backup Exec choosing the option to only uninstall Backup Exec Program files.
  2. Reboot
  3. Install Backup Exec 2012 from media or downloaded installation.
  4. Download SP3 from here: http://www.symantec.com/business/support/index?page=content&id=TECH205351&profileURL=https%3A%2F%2Fsymaccount-profile.symantec.com%2FSSO%2Findex.jsp%3FssoID%3D1401345892983cayT195O3QT4WE55JKR0R41hjE5pXiwls68JOfile and then install it.
  5. We got a whole bunch of ODBC errors if we tried to do anything in Backup Exec. We found that running this command cleared up those errors: C:\Program Files\Symantec\Backup Exec\catrebuildindex -r (See this as a reference: http://www.symantec.com/business/support/index?page=content&id=TECH67271

That was it. No re-creating jobs, no inventory or catalog jobs. We are currently running backups again and the Job Engine is not unresponsive and is not consuming lots of CPU anymore. We intend to stay at SP3 for the time being. It was much more stable for us. Hope this helps.

DaveyMG's picture

Hello Ken, I'm going to follow this course of action as well. Did you have to roll back the windows agents on other machines after you went back to SP3?

Ken Johnson II's picture

Just to be safe we uninstalled the agents from the servers and then push installed them again from the Backup Exec server after installing Service Pack 3. Most of the servers did not require a reboot for the uninstall, but one older 2003 server did. Your mileage may vary.

FYI: All the backups ran successfully last night for the first time since we installed SP4.

DaveyMG's picture

The uninstall/reinstall went OK, also had to do the catalog rebuild. Duplicate tape job started straight away and bengine.exe using up to around 10% cpu. I'm going to let my backlog of dupe to tape jobs clear before doing a couple of test backups as I don't want to have to reinstall the agents if I can avoid it.

R2D2_C3PO's picture

FWIW for anyone else going through this, ahem, inconvenience - this solution worked word-for-word.

I installed SP4 and the Hotfix that came thereafter.

Our primary job workflow is a primary backup to a NAS that duplicates the backup set to a secondary external drive once the primary is made.

Primaries to the NAS were unaffected. The duplication backup job - or usually, the verify portion of the duplication backup job - would get stuck. As a result BE would come to a screeching halt with all jobs held up until intervention. The only resolve was to reboot. It would work again for a while but absolutely no jobs would complete on the duplication task after the primary backup.

The exact above steps worked for me. The only thing I had to do was reconfigure the visuals of BE; no lost jobs, no lost data, etc. Even down to the point of the ODBC errors ... ran that command line and that resolved that.

Everything is back to a way more significantly stable BE with all jobs completing. Will NOT be updating 2012 anymore ... will stay status quo until jumping up to 2014.

Thanks, Ken Johnson, for the write up!

marcodoy's picture

Symantec? Are you serious with your product and your support?

 

WE NEED TO UNINSTALL everything and go back to SP3. I havecase opened and the support do everything to do not escalate to advanced. This is crazy on things take long time to be fix.

 

There is no resolution for this case. The support close the case and open a different one  06668941

Mark Stephens's picture

Spoke to the BackupExec CORE group who advised doing a repair of the install (add/remove...repair) and that seems to have worked for us.  Scheduler is much improved, jobs ran over weekend and although a couple had issues others that were active completed and those scheduled to kick-off later ran and completed.

With that said, I'm looking forward to taking a look at 2014 (released today) and seeing how that does.

Mark

Wes Stewart's picture

Off-shore support had me install SP4 for another issue I was having, which was not being able to restore within random VM backups.  That issue was not solved by SP4, but by rebuilding the catalogs.

Since the install of SP4 I have spent hours upon hours dealing with the issues. On-shore support has been working with me, and we thought we had it licked last week after a repair, with two solid days of complete backups.  But this past weekend it came back with a vengeance. 

My first entry on my support ticket after the SP4 install says it all: “The installation of service pack 4 has create utter havoc. I have failures all over the place. Jobs running over each other and essentially very little being backed up. I don't have time to think of the original issue of restoring from VMs now because I am having a tough time getting complete backups to run."

I now have an open support ticket for 2 months on the issues of SP4.  Major issue is jobs hanging at queued, or verifying.

My tech support rep calls almost every day, but I really have run out of time to work on this, and I feel we have hit a brick wall.

BTW twice at the request of support I have run sgmon overnight to capture logging information.  It would suck up all of the RAM from the BE server which would cause all of the jobs to fail.

Is BE 2014 the fix or another whole set of headaches?

 

DaveyMG's picture

The solution seems to be to uninstall/reinstall BE2012, install SP3 and stay there. It worked for me and a few others judging by other comments on this thread.

I'm considering waiting for BE2014 SP1 before going there.

Mike55's picture

For what it’s worth, SP4 seems to be working since I dropped deduplication completely.  So that’s with backup to disk and duplication to a tape library only now.  I’ve had no more queuing issues since running tapeinst.exe and dropping deduplication.  This server was a SP4 media install.  I have older posts here.

I may start testing soon on my abandoned server that was upgraded to SP4.

Wes Stewart's picture

If I have to uninstall and reinstall, what do I have to lose by trying 2014? I am under maintenance so it is no additional software cost to me.  If it is a mess I go back to step 1.  I'm downloading the 2+gb trial right now.

 

DaveyMG's picture

I guess you have the uncertainty of not knowing whether 2014 will fix your current set of problems or become "another headache" and, if it doesn't fix your problems, the anxiety of an ever increasing window of failed backups while you do yet another uninstall reinstall to go back to 2012 (providing you can go back as 2014 may change the catalog database structure).

annettelloyd's picture

I have getting cases with the same issue.  I know this is frustrating, believe me, I understand.  I would just like to let you know as long as you save out (to desktop, or documents) the data and catalogs folder from the 2012 install, you can always roll back safely by simply removing 2014 and putting 2012 back in, confirm it's functionality, then stop the sql instance service and be services, drop those folders back in on top of the newly created ones (make copies of these too just in case), you should be able to restart the services and have 2012 back to prior to the upgrade.

jan.villeres's picture

There is a recent hotfix 217347 for Backup Exec 2012 Revision 1798 SP4.
http://www.symantec.com/docs/TECH217347

Has anyone tried installing this and resolved the deduplication issue?

Mike55's picture

Sorry, I’m done with Symantec deduplication.

Let me put it this way though:
SP4 was released 2014-03-13
Hotfix 217347 was released 2014-05-27, 75 days later

Even if this does fix the queuing problems, if it’s cool at your company to have your backup system crippled for 75 days, please post.  I want to know what business you’re in.

HNeumann's picture

75 days :-) better than 365 days :-) no its cool Symantec is the best company a very big global player for Backup ;-)

SkipStPierre's picture

This should no longer be of an issue for our Backup Exec 2012 customers.  However; for anyone that might be experiencing any of the symptoms from above, we invite you to upgrade to Backup Exec 2014

In Backup Exec 2014 we have made many improvements and you should no longer experience this issue. 

You are welcome to PM me directly if you have any questions.

Thank you,

Skip St. Pierre

Product Management

Backup and Recovery Team!

jan.villeres's picture

We have an environment with Backup Exec 3600 R2 appliances configured as MMS. Is the new Backup Exec 2014 compatible with our existing applances?

Thank you.

DSAI's picture

Hi,

 

Can someone confirm the problems associated with SP4 have been resolved.

 

thank you

fwolf's picture

Hi DSAI,

In my case hotfix 217347 doesn't work. I'm currently creating a new case for my trouble.

Regards