Video Screencast Help

BE 2010 R3 hanging on system state

Created: 22 Oct 2012 • Updated: 30 Oct 2012 | 17 comments
This issue has been solved. See solution.

We were running BE 2010, not sure exactly which revision and randomly starting having our entire backup job hanging on the system state of a few remote servers. I installed all the latest updates and updated the agents, same thing. I can do system state backups on these systems individually, but as part of the larger full backup, they hang to the point that I can't cancel the job. I have to restart services. I've done the vssadmin checks on the remote servers and they come back clean and the backup jobs are running to a B2D folder. Any suggestions?

Comments 17 CommentsJump to latest comment

hrishi_111's picture

< I installed all the latest updates and updated the agents, same thing. >

Good move, have you had an opportunity to reboot the remote servers wherre the agents are installed?

The remote servers needs to be restarted once an update being installed/pushed to the Agents.

Jason Casteel's picture

Yes, they were restarted. Some requested it, some didn't, but I restarted all of them.

Vivek Jain's picture

See if the Backup Exec service restarts during the full backup or not...Try re-creating the backup job or maybe change the resource order & see if it hangs at the same time or same resource.. Anything in the Event Viewer ?

 

Jason Casteel's picture

Nothing in the event log and none of the BE services are restarting.

 

I haven't tried changing teh order yet, but i'll give that a go. So far i've just been tweaking settings and such. Thanks.

Jason Casteel's picture

Ok so I tried changing the order around and it still hangs up when it gets to either of two particular servers. One is a 2008 R2 server, the other 2003 R2 64bit. Both have the correct, latest agent installed. Other things I've also tried:

I've broken the backups out into system state only backups and those complete without issue. 

I have a file only backup and that one still appears to hang when connecting to the above servers. 

 

Any new suggestions/tests I can run with here? 

Vivek Jain's picture

Check if any AV scans running or not... And is AOFO enabled in the backup job ?

Run a file-only backup for one of the servers...double-click the active job on the job monitor tab & check on which particular file is the backup hanging...

Jason Casteel's picture

arg, sorry, I left that part out. 

 

There's no a/v installed on these servers and while we previously did have AOFO enabled, I disabled it for the file only backup run to see if it would have any positive impact. 

 

Oddly enough it doesn't list any particular file that it's hanging on. It shows the following:

 

Source: \\servername\c:

destination, current directory, current file are all blank. This is the same thing I would see when it hung on the  system states as well. 

 

Oh and one more thing to mention, it still will not let me cancel these jobs when they hang. It continues to show the state as "running" and never changes to cancel pending. I've checked the event log on the server that it's trying to back up but hanging on and there's nothing at all in there related to the agent, shadow copy or network errors. 

Vivek Jain's picture

Is Backup Exec & the agents full patched ? And are you able to take a backup though the native windows backup ?

CraigV's picture

...OP mentioned that everything was patched in the original opening query.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Vivek Jain's picture

Maybe I did miss the part about the BE being fully patched, but the OP isn't sure about which release of BE...neither has he mentioned anything whatsoever about my second query....I think this doesn't warrant a thumbs down when a valid query is being asked.

CraigV's picture

...can't give a half thumb's down. wink

Also, the first part of your question above wasn't targeted at the version; it was whether or not he had patches installed...thumbed up your comment below.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Jason Casteel's picture

2010 R3, fully patched. 

 

Haven't tested running a local backup, but as mentioned below i'm going to uninstall and repush my agents and see what that gets me. If that goes nowhere I'll try a local backup and go from there. Thanks. 

Jason Casteel's picture

Ok I have more to add. Last night I decided to break the two servers that were hanging previously into individual backups with system state. Both were successful. My remaining backup of servers that previously had no issues hung up on a different server this time. The odd thing that I've noticed, which I have not noticed before, is that they all seem to hang at around 210-250GB. These are B2D backups that get offloaded to tape later and there's plenty of free space there, upwards of a TB or more. For the the backup that hung again last night I created a new B2D folder before starting it, just to make sure there wasn't something damaged with that one. 

 

At this point I think I'm going to uninstall the agent on all of my servers and push it out again. There's no logical reason I should have to do this, but it's software... you never know. 

 

If anyone has any new suggestions they would be appreciated, just in case. 

CraigV's picture

Software does become corrupt, and it wouldn't be the first time that a RAWS agent somehow got corrupted. I haven't had it happen on many occassions myself, but it has happened to me.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Jason Casteel's picture

Ok so my plight continues. I uninstalled and repushed all agents and also did a repair on media server install. 

I created an entirely new selection list, created a new B2D folder and a new media set, created the backup job, files only, and it hangs at queued or snapshot processing. I can't cancel the jobs and have to restart services to get out of that. There's plenty of space on the B2d device. I've also tried a different B2D device with the same results. 

 

one thing I noticed is that before starting the job is showed the device it's going to be backing up to, but hwne you start the job that field goes blank. 

 

Very Frustrating, any suggestions? 

CraigV's picture

Hi Jason,

 

Might be time to open up a case with Symantec if you have a support agreement with them. If they help you find a solution, post back here & close this query off.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Jason Casteel's picture

I decided to completely remove BE and reinstall it and so far everything is running successfully. Thanks to everyone here for your assistance. 

SOLUTION