Video Screencast Help

Enterprise Vault Agent

Created: 17 Jul 2012 | 4 comments

Environment:

BE2012 SP1a

EV 10.0

SQL 2008 R2

Problem:

The BackupExec service hangs (and the job says "stalled") after the databases for the vault stores are backed up.  The actual vaults are on a NAS so they are de-selected.

I created a job to JUST back up the vault store databases - this is where I have isolated the issue.

So, this is what happens:

Vault stores are put in backup mode properly (from EV server event logs)

Vault store database are backed up (from SQL server event logs)

Vault stores exit backup mode properly (from EV server event logs)

The job hangs as soon as the vault store exits backup mode, goes to stall, tries to recover, stalls, etc, etc.  If there were other jobs running when this happens they are all failed (bad).

Spoke with Symantec, got tossed between EV and BE engineers repeatedly.  Was told my tape rotation was not optimal and that we needed to turn off antivirus (neither of which did anything).

So, is anyone running the EV agent successfully?  Anyone running the EV agent with the vault stores on a NAS?  The actual vault stores get backed up via NDMP (in backup mode with some powershell scripting to put them into backup mode) successfully.  I could work around this with backing up the SQL DB's directly but since the EV agent is installed and licensed it doesn't let you select the databases directly on the SQL server.  Just about ready to back to BE2010

Comments 4 CommentsJump to latest comment

Russ Perry's picture

from your description the EV agent has done everything it is supposed to do but the mention of the job trying to recover sounds like a Backup Exec service stopping.  Check on this and also to see if any of the service processes are using excessive resources. 

Are there any problems backing up the system databases on the SQL instance you have your VDBs stored on?

Russ

DanJ71's picture

Not sure what you mean by excessive resources - I'm assuming you mean I need to see if it is ramping to 100% somewhere.  The BE server is a quad-core dual proc 2008 R2 server with 8GB of RAM. Never see CPU high or memory high.

Backing up the system databases (and any databases on any other SQL server) works without a hitch.  The only problem (that I'm aware) is related to the end of the EV agent.  After the "stall" it produces the message "The job was canceled by user Recovery"

So, I created a job to JUST backup the audit database - when it is part of a larger job it finishes and moves to the next step but the job in total "stalls".  When it is the only thing in an EV job, bang the thing stalls AFTER the database is backed up successfully.  I back up other database servers and I back up this same SQL server (other databases) without a problem.  It seems like it is something after the agent is done that is either not exiting doing something nasty to the job engine.  I'll try Symantec tech support again but I was hoping that someone else was having this problem ... perhaps I'm the only poor soul that has upgraded to BE2012 running the EV agent ;)

 

Abhijit_D's picture

To work around EV agent and back-up the SQL DB's through remote agent, one needs to rename file "bedsev.dll" like 'bedsev.dll.old' etc on both Backup Exec server and Enterprise Vault server. Restart BE services on both servers after rename. This will enable SQL database selection of Enterprise Vault databases in backup job.

EV agent performs only DB consistency check after backup and that should not stall any running job. Was this backup operation working fine with BE 2010?

It will be helpful if you can email or share you support ticket id logged with BE support.

DanJ71's picture

Thanks for the tip on the DLLs - this would have been nice to know 8 weeks ago when I asked support if there was a way to disable the EV agent so I could get a backup. 

The DB's actually back up but afterwards it goes into "stall" - you can see that they were backed up on the SQL end (as well as the consistency check completing successfully).  It even happens when you back up something that doesn't require anything going into backup mode (like the audit DB).  If you back up the other databases on the same SQL server it works without a hitch.  Nothing ever crashes so all attempts at getting a crash dump produced nothing as there was no crash dump to analyze.

I installed another copy of BE to see if it was the installation that was a problem (it was an upgrade from BE2010 which worked fine) and there are no problems with the fresh installation.  The ticket number is 418-809-203 but over time I believe I was assigned multiple tickets to no avail. 

At this point, I believe we are going to do a complete rebuild and import the catalogs.  This is a fairly painful process (fiber attached tape library, multiple B2D locations, a year of catalogs, etc) and I would prefer a solution that would make it work (everything else works except the EV backup).