Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

Never ending job

Created: 12 Jan 2013 | 14 comments

From time to time I have jobs that start well and progress to backup varying amounts of data then the job stops writing.  No error, no message of any kind just nothing else is written.  We only detect it when looking in activity monitor and find a few hour job is still running 26 hours later but no increase in write activity or file being backed up.  I have little doubt that everyone has experienced this.  What do you do with these jobs?  Very frustrating.  please help.

Comments 14 CommentsJump to latest comment

Marianne's picture

You will need to share details of one of these jobs:

All text in Details tab of specific job
Media server logs: bpbrm and bptm
Client logs: bpcd and relevant client backup log, e.g. bpbkar if filesystem backup.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

trs06's picture

I moved past the job that prompted this post and re-ran the job.  The next time it happens I will gather the requested information.

thx

bills's picture

I have seen these several times.  Generally I just kill and restart.  Moderately annoying....

Bill

William Jansen van Nieuwenhuizen's picture

Hi

 

I've even seen jobs that fail in NBU that you cant even delete without cycling the NBU processes(but they had a specific error code). Like Mariaan said, we would need some additional info on these jobs. Like the backup type, detailed job logs and possibly a screenshot showing the job to help jump start memory :-P

Mark_Solutions's picture

If you look at the job PID in its details you should find that it relates to the bpbrm process on the Media Server - usually if you kill that process of the job will die (you may need to kill off its associated bptm process also but that normally closes when bpbrm is killed off)

But it is usually caused by an unresponsive client so you need to take a look at the client involved to see why it is hanging - often low disk space etc

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Carlos Abreu's picture

I have this same issue..

My job starts normally but after some days running it not end, no error displayed, no log message and other jobs from the queue fails becouse they don't get resources.

Someone have a troubleshoot line that i can follow?

Stumpr2's picture

I see this sometimes. I suspend and resume jobs.

If I can't suspend then I cancel the job.

If I cant cancel the job then I log into the media server and kill bptm

VERITAS ain't it the truth?

captain jack sparrow's picture

in past i got dumped in same situation , and had big nightmare solving it

we observed trend which was causing this.

After support analyzed behaviour of jobs being in stalled state, it turned out to be bug with 7.5.0.3

Strange you never highlighted information about your NBU domain and affected clients

Support had asked to increase time outs and to apply EEB to only 7.5.0.3 clients which were affected

If we fall back on 7.5 it went smooth

Check if it's same with  you. Better sync with Support to check if any EEB exists and can address it or require backline to intervene

 

Hope this helps

 Cheers !!!

CJS

 

Carlos Abreu's picture

 

My problem seems to be with multiple data stream.
I have Netbackup 7.5.0.4 on Linux RedHat Enterprise 5.4 (Tikanga)
 
I have a policy with multiple data stream, and I observed that if any stream is bigger than 1TB the job will into the stalled state.
Mark_Solutions's picture

popular thread - shame trs06 hasn't been back on here for 2 weeks!

I have found that a combination of processing limites and timeouts can cause these

There were some early bugs in NBU itself but if patched up to date wityh all available EEBs it shouldn't really happen

On Windows it can be caused on busy systems due to the DesktopHeap Setting being used - difficult one to track down as it doesn't always even log to the event logs due to being too low to be able to even log to itself.

So make sure you memory is always tuned - paging, paged pool memory, desktopheap etc. and it will really help

It is often associated with nbrb so if you are not on 7.5.0.2 or above take a look at this tech note:

http://www.symantec.com/docs/TECH187645

lways worth tuning nbrb to get the best out of you system:

http://www.symantec.com/docs/HOWTO43779

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

trs06's picture

I have followed all posts.  Please remember that I get each of your posts via eMail.  Also this is not the only matter I have to deal with.

I appreciate all of the information I get from Symantec Connect.  I've found it a very valuable resource.

Thank You.

Mark_Solutions's picture

Thats great - sorrry if you took my comment badly but we get so many people come on here and make a post and then dissapear - glad you have been following things and that the forum is of help.

If there is further troubleshooting we can do on this one for you please get back to us

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

trs06's picture

I wrote up our environment then forgot to hit the save button.  Duh

Netbackup 7.1.0.4

Master Server is RHEL 6.2 VMware  7.1.0.4

(2) Media Servers Windows 2008 R2 Standard SP1, Physical, 7.1.0.4

Clients are a mix of Solaris 9 and 10, Global and Local Zones, RHEL and Windows 2003 - 2008 and lots of MSSQL, DB2 and Informix.  Mostly 7.1.0.4 a few 7.1.0.1

We are budgeted for 7.5x in the fall 2013.  I hope by that time many EEB's, bugs and fixes are rolled up in the version we install.

Mark_Solutions's picture

7.5.0.5 or 7.6 should be out by then hopefully so should be good timing!

I dont see anything in terms of rollups or EEB's for 7.1.0.4 but it may be worth asking Symantec - many EEBs never make it to public view.

Also worth looking at the nbrb tuning links i gave in my earlier post and the first link as that affects all version of NetBackup below 7.5.0.2 - so does apply to your setup

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.