Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Notifications after switchover

Created: 25 Oct 2013 • Updated: 25 Nov 2013 | 6 comments
This issue has been solved. See solution.

Hi

Does anyone know how to get VCS to send a notification when it automatically fails over an application?

Thanks,

Laura

Operating Systems:
Discussion Filed Under:

Comments 6 CommentsJump to latest comment

kjbss's picture

Look in your VCS Admin guide for sections on configuring the Notifier resource.  In the Linux vcs_admin_60_lin.pdf guide at:

https://sort.symantec.com/public/documents/sfha/6....

...on page 118 it covers "Notifier Resource Configuration wizard" which then points you to page 166 on "Setting up VCS event notification by using the Notifier wizard".

Page numbers will likely vary depending upon your version of VCS.

It is pretty straight forward.

-HTH

Kevin

SOLUTION
lahenrich's picture

Thanks!

Unfortunately I am not familiar with running the gui and would rather just modify the postonline and postoffline files under $VCS_HOME/bin/triggers with a simple mailx command.  Doesn't seem to work.  Is there something else I need to set or update in another configuration file to get this to work?

kjbss's picture

I do not recommend messing with the postonline and postoffline trigger scripts for what you say you wanted, which was to "get VCS to send a notification when it automatically fails over an application"

To me, the above means you want to have VCS send you a notification when a  service group is failedover to another node in the cluster.  VCS will only automatically fail a service group when one of it's critical resources fault.  

Whereas the postonline trigger script is invoked when the service group faults or is taken offline manually.  IE: you will be sending emails when the service group didn't even fault, and that is likely not what you want to do.  Of course you can put intelligence in the trigger script to determine if it is a switchover (no fault), or simply a normal service group being asked to go online, and only send your email when the service group is marked faulted somewhere in the cluster.  But that is also problematic in a multi-node cluster.  

It is not very hard to set up the normal Notifier resource manually, so I would give that a go first.  Look in the bundled agents guide for how to configure it and also in the Admin guide. 

As far as why you cannot get your postonline/postoffline triggers to work...

1.  Make sure it is in the trigger directory and is executable by root (it probably also should be only rwx by root, group and other permissions should be disabled (aka:  postonline owned by root and have permissions -rwx------)

2.  Just in case standard out is not captured by the bit of VCS software running the trigger, make sure that you send some test and/or status information from it to a log in a reasonable place, like /var/VRTSvcs/log/postonline.log.

3.  Make sure the script exists on all nodes in the cluster (or at the very least, on the node it will run on during the test, which should be to node that has just onlined the service group (be careful here, the postonline script will be run on the system where the group went online from an offline state; whereas the postoffline trigger is invoked on the system where the group went offline from a partial or fully online state).

4.  Manually move any service group over and look for your log entries in the file you decided to log to.

Then you should have a good idea if your desired trigger script is running...

It might help you to review the sample trigger scripts located in the  $VCS_HOME/bin/sample_triggers/VRTSvcs directory.

-HTH
 

AlexeyL's picture

You can change notifier settings in CLI but I believe it can only accept severity levels for notifications. So if you failover is successful it would be just "Informational" severity level that most of standard VCS activities fall under. So you probably would be swamped with all kinds of emails if you do anything with VCS.

Maybe monitoring engine_a.log file (maybe custom script every few mins or some serious product like splunk) would be one of the options. You could search for either of these two:

VCS NOTICE V-16-1-10208 Initiating switch of group ClusterService from system sys1 to system sys2.

VCS NOTICE V-16-1-10447 Group ClusterService is online on system sys1

Rather than playing with VCS itself you could try this way.

You can also install Veritas Operations Manager. It has a lot of flexibility in setting up the alerting via emails/custom scripts.

kjbss's picture

Well, there are certainly more than one way to do things...

However, I would recommend using the intended and integrated tool, and setting the level to 'Error', because as AlexeyL points out, you do not want hundrends of emails sent out every time you do anything on your service groups like switch them from one host to another, or bringing them up (online) or down (offline).

Here's a sample configuration I have, which works very well for me:

        NotifierMngr AlertNotifier (
                SmtpServer = "xxx.10.8.xxx"
                SmtpServerVrfyOff = 1
                SmtpReturnPath = "VCS_AlertNotifier@yourDomain.co.uk"
                SmtpFromPath = "VCS_AlertNotifier@yourDomain.co.uk"
                SmtpRecipients = { "root@localhost" = SevereError,
                         "Kevin.Barth@myDomain.com" = Error }
                )

With that NotifierMngr resource configured, I get an email whenever something bad is happening on my customer's system, and I mostly end up fixing it before they get around to addressing it themselves.... (keeps them very happy...)

mikebounds's picture

It depends on that you are trying to achieve.  If you want an email when an application has failed, then add a Notifier resource to the ClusterService service group using "hares -add" and modify attributes such as SmtpRecipients using "hares -modify".

However if you want to know what node your application has failed to, then this is an information message and you probably don't want an email about all information messages, so then the postonline trigger is more appropiate.  

You do not need to modify any configuration for postonline to work - the only trigger that requires this is the preonline trigger.  Use "halog" command in your trigger to check whether it is being called and of course check it is executable and in the /opt/VRTSvcs/bin/triggers directory.

Note you can use the passed 4th and 5th arguments:

whyonlining:  FAULT or MANUAL
system_where_group_faulted: Name of the system on which the group has faulted or switched
 
in your script to determine when to send an email and if you want to include what node the application failed on in your message

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

SOLUTION