Video Screencast Help

Backing up server ends in recovered or stalled

Created: 12 Oct 2013 • Updated: 21 Oct 2013 | 30 comments
This issue has been solved. See solution.

Hi,

I am running BE2012 with latest SP and hotfixes. One of my servers refuses to finish a full backup.

It runs and at some point it always get to stalled or recovered 0% complated.

I have run a defrag on the server.

Restart it.

Try to backup just a volume with data.

Try to backup system state and C:

Nothing has worked, any ideas?

Thanks,

Operating Systems:

Comments 30 CommentsJump to latest comment

pkh's picture

Are there any error messages in the Windows eventlog on the remote machine or the media server?

*EDIT*

Have you tried pushing out the remote agent again to this problem server?

porress's picture

Hi,

 

Nothing on the remote server aplication or system logs. Nothing as yet on the BE server.

I have reinstalled the agent manually on the remote server and restart.

I have just selected the system state to backup and it looks like it is all working fine and then it gets to operation = Completed on the job details and status running. Then after a few minutes it always gets to stalled status and that is it.

Any other ideas?

porress's picture

Hi,

I have just tried to backup just 1 file that was 340 bytes in a c:\test folder and the it backuped up the file. The same issue still operation = completed and the status = running. Then Stalled.

It looks like a glitch when BE2012 is trying to finish the job or something.

Nothing on the remote server event logs.

CraigV's picture

...any AV installed? If so, turn it off and try the backup again.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

pkh's picture

Is there any outstanding alerts in BE?

Are you using a stand alone tape drive and you have checked the eject after job completes option? If so, there will be an alert asking you to remove the tape. You need to respond to the alert before the job can complete.

porress's picture

I have turned the antivirus off and try again and the same issue.

I have cleared the alerts as well and all of the other servers are backing up fine. I am backing up to Disk storage.

pkh's picture

When the job stalls, look for any alerts.

porress's picture

I have cleared all the alerts and restarted all the services.

Run the one file backup again and same issue, stalled. I had cancelled the backup but no alerts were showing on BE interface.

 I have this log from previous attemps.

Job ended: 13 October 2013 at 09:00:15 Completed status: Recovered Final error: 0xe00081d9 - The Backup Exec job engine system service is not responding. Final error category: Job Errors

I used to get this error randomly with the servers but since I did the upgrade to SP2 and the hotfix after all the servers are fine apart from this one.

What do you think I shall try now?

porress's picture

I am running the job again and see if it gives me something in the job history later.

Do you have any other tests/changes I can do?

CraigV's picture

Eoes a backup of the local media server go through?

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

porress's picture

I have run the job and it goes to job history as recovered 0% with the same issue.

It looks like in the job history the job has tried to run 3 times with the same outcome. Shall I try to backup the local media server and see if there is an issue with as well CraigV?

 

Job ended: 13 October 2013 at 11:31:45

Completed status: Recovered

Final error: 0xe00081d9 - The Backup Exec job engine system service is not responding.

Final error category: Job Errors

 

For additional information regarding this error refer to link V-79-57344-33241

CraigV's picture

Yes i would. Reason for this is that it could prove to be an issue there. So check the AV on this server as well. Check the event viewer logs to make surenthe BE services aren't stopping either.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

porress's picture

You were right!. The BE server can not backup itself either. It was backing up very slow at 150Mb/min. It has failed as well. This is so frustrating...

Could it be something wrong with the Backup exec database? I am running another backup of the BE server again without the AV on. I have run a consistansy check and it is fine. I dont understand how one remote server can not backup and the BE server either but the rest are backing up fine.

Shall I run a repair database and reindex or something like that?

Thanks again,

porress's picture

I done everything in this TN and see if that helps.

http://www.symantec.com/business/support/index?pag...

I have found the  - Adamm Database Event: Handle Failure! on the application log when backing up the BE server. It might be the database why I am having issues with that remote server and the backups of the BE server?

What do you think?

I am running another full backup of the BE server and see if this database repaired worked.

 

 

porress's picture

This is what I am getting on the BE server when It tries to backup itself.

EVENT LOG:

Adamm Database Event: Database Connection Lost!

Error = E_PVL_DB_LOST_CONNECTION

 

porress's picture

I can see lots of event logs as information on this during the backup and before the crash:

 

Database Exception Context:CBESession::CreateDBCmd Error:-536837662: :: -2147467259:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80004005 Timeout expired

I have entered on google and it takes me to the hotfix for BE2012 I have installed a few weeks ago. Could it be that the hotfix is not working for the BE server still or that I have to re-run it?

http://www.symantec.com/business/support/index?page=content&pmv=print&viewlocale=&id=TECH195183

What do you think?

 

 

CraigV's picture

...reinstall the HF, and for good measure, open up BEutility.exe and run a repair of the BEDB.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

porress's picture

I am reinstalling SP2 for BE2012 and see if that solves the issue. I will let you know.

Thanks,

porress's picture

I have raised a case with Symantec just in case this does not work.

 

Jaydeep S's picture

Could you provide me the case number.

porress's picture

Same issues again. 

-This before the failed job on the log:

Database Exception Context:CBESession::CreateDBCmd Error:-536837662: :: -2147467259:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80004005 Timeout expired

-Then this on the application log:

Adamm Database Event: Database Connection Lost!

Error = E_PVL_DB_LOST_CONNECTION

 

 

CraigV's picture

...I suspect that if you dump your database and reload from base you should be OK as it could be some sort of database corruption.

However, read the TN below on how to comprehensively repair your BEDB:

http://www.symantec.com/business/support/index?page=content&id=TECH67239

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

pkh's picture

What time is the job running each day?  Make sure that it does not clash with the BEDB maintenance which by default is at 4 a.m. each day. If need be, change the time of the BEDB maintenance by clicking on the BE button ---> Configuration and Settings ---> Backup Exec Settings ---> Database Maintenance

porress's picture

I have been on the phone to symantec and we have done a repaired of the BE2012 installation.

Disable the removal storage service first.

Add exclusions to the antivirus:

c:\program files\symantec\  ---- I have turned it off before completely for testing as well.

Reboot.

They have asked to try again to run a backup and see if I get this errors again and call them.

 

We will see what happens.

Thanks,

 

Stefan_F's picture

Same problem with BE2012 SP2, including hotfix 209149:

Completed status: Canceled
The job was canceled by user Recovery.

Event id 57345 - Backup Exec
Database Exception Context:CBESession::CreateDBCmd Error:-536837662: :: -2147467259:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80004005 Timeout expired

I found the next link with the same problem: http://pastebin.com/gMcUhW6J.
I will exclude the Symantec folder to the antivirus.

porress's picture

Hi,

It has not worked as yet.

Symantec support has taken just about 1 hour ago some log files using SymHelp utility and Symantec Gather Utility for the advance support team to investigate.

In 24 hours they will call me back.

 

Stefan - That is exactly the same I have installed SP2 with that hotfix.

I hope they come back with a solution.

Thanks.

porress's picture

Hi,

Symantec advance team is still looking into it. They asked me to take off the VSS shapshots on the server and run it again. The same issue is still there.

I dont understands what they are looking for now but tomorrow they are going to call back. They asked me as some point not to reference the system log and only go for what it says on the BE log history at the end of the backup. We will see what happens.

Thanks,

 

Stefan_F's picture

Our daily jobs are succesfully.
Once a month we are running a backup. After that job we get the problem with the daily jobs and get the errors as shown in my previous post.

For now, I have re-created the monthly job. Hope this will solve the problem.

porress's picture

Symantec support advance team sorted my issue.

http://www.symantec.com/business/support/index?page=content&id=TECH188880

Thanks guys for this one. It was not easy to find out exactly what the issue was.

Appreciated.

 

 

SOLUTION