Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

I need a command to kill a respawning DB2 backup...

Created: 18 Jul 2014 • Updated: 25 Aug 2014 | 9 comments
This issue has been solved. See solution.

Master - 7.6.0.2 on 2k8R2

I've got a couple of AIX boxes running DB2 that I back up.  And every week when they're rebooted the policy for the log files (just an User Backup) attempts to back up the final log but then the client software gets shut down before it finishes.  So I get a series of retries with code 58...eventually the box comes back up, that backup works, and everything is then clear.

But what worries me is if we ever have to shut that machine down for any maintenance, I'll get a eternal string of code 58 failures until it comes back up.

So...short of cycling the services on the Master, is there any command I can execute at the CMD level to stop the eternal respawning?  Or is there a process that gets spawned on the Master that I'm not seeing that I can kill?

Thanks,

Operating Systems:

Comments 9 CommentsJump to latest comment

INT_RND's picture

It sounds like you want to disable a policy temporarily. If the server is reset regularly then you want to use the backup window to stop the policy from spawning during the scheduled reboot.

Here is some instruction on using backup windows:

http://clientui-kb.symantec.com/resources/sites/BUSINESS/content/live/TECHNICAL_SOLUTION/67000/TECH67319/en_US/318347.pdf

If you have maintenance coming up that is not part of your regular schedule all you need to do is deactivate the policy prior to shutdown. You must remember to reactivate the policy once the maintenance period is over.

You can accomplish this via the GUI by using right-click on the policy name. 

You can also use the command "bpplinfo" a script:

/usr/openv/netbackup/bin/admincmd/bpplinfo <policyname> -modify -inactive

/usr/openv/netbackup/bin/admincmd/bpplinfo <policyname> -modify -active

Command guide:

http://www.symantec.com/business/support/index?page=content&id=HOWTO43678

D.Flood's picture

The problem with establishing a window is that it won't allow any log backups to happen so if I guess wrong then we have missing logs.

I guess the only thing I can do is add a 'sleep 120' to the start of the bp_kill_all script to try to give things a chance to finish before NetBackup gets shut down.

INT_RND's picture

You are planning on taking down Netbackup to stop one policy from running? Is this the only policy in your backup environment?

Marianne's picture

You say this is 'just an User Backup'? 

This statement implies that the backup is kicked off on the client, since a User backup cannot be kicked off from the master. 
Or maybe I am misunderstanding what you meant since a client cannot exactly kick off a backup while being rebooted, right?

To disable backup attempts for this client, you can 'offline' the client.

In Host Properties -> Master -> Client Attrtibutes, add/select this client name, then select 'Offline Until'.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

D.Flood's picture

Yes, this is configured as a User Backup.  The DB2 system is set up to dump its logs to NetBackup as needed so the policy only contains a 24x7 schedule that's configured as a User Backup.

'Offline Until' won't stop the backup attempts, once started.  It just changes the error code.  And unless I install Media Server on that box and configure the shutdown scripts to remotely edit the "offline until", I may not get that last backup as it is shutting down.  Same thing with editing the time window.

Since there doesn't appear to be any way (short of cycling the Master, and yes, I would cycle the Master if I needed to in order to stop a new code 58 from showing up every minute) to tell NetBackup to "stop trying that backup that was requested", I'm resorting to trying to slow down the shutdown process on the box to give NetBackup time to finish what was started.

INT_RND's picture

If the DB2 server is the source of the backup job and it is also the source of the backup interruption then you need to correlate the two. 

Don't allow the DB2 server to shut down while a backup operation is in progress. Edit the "rc.shutdown" script to check for backup jobs. This will stop the shutdown process until it is safe to reboot. You can also put in commands to gracefully take down the DB2 server in there.

"/etc/rc.shutdown script. This file is the opposite of rc.local. Here, commands are placed that are to be run when the system is issued with a shutdown operation."

You would check for the running process with something like this:

#!/bin/sh
SERVICE='NAME OF PROCESS THAT INDICATES BACKUP JOB IS ACTIVE'
 
while [ ps ax | grep -v grep | grep $SERVICE > /dev/null ]
do
    echo "$SERVICE service running, don't shut down yet"
done

That will fill your screen with text in an endless loop if the process is hung or otherwise still running. Edit to fit your needs. Use at your own risk. Not an official Symantec solution. You get the idea.

Marianne's picture

I agree.

You need to speak to db2 dba and server owner to check and edit shutdown scripts.

Since this is a User backup, there is nothing you can do from NBU to stop or disable the user requests.

If shutdown cannot be postponed, the backup processes will have to be killed as part of shutdown script.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

D.Flood's picture

Update:

The Application Owners decided that they didn't want either the development or production boxes rebooted for the last few weeks.

That means I got lucky and the situation didn't occur.  But it also means that I haven't tested the sleep delay yet.

We have some maintenance going on later today that will require both boxes be shut down so I'll finally get to see if the sleep works or if I need to get the AIX Admin involved and add the official shutdown script to the shutdown process.

D.Flood's picture

Well, the sleep helped but further research shows that the issue was actually on startup.

The NetBackup startup script was after the DB2 startup script.  So when DB2 tried to fire off a backup, there was nothing to connect to until NetBackup got started.

A reshuffle of the scripts and everything should work nicely this Weekend's reboots.

SOLUTION