Symantec, SNMP, and OpenNMS
One of the challenges of administrators is gathering statistics and notifications. Symantec provides a lot of great resources to help manage events and notifications. I'm able to set up traps from my OpsCenter server to go to my Veritas Operations Manager, where I can create custom notifications for certain events. In Command Central, I can also create notifications. Additionally, I can also create notifications for the same system services that are tied in with Command Central within VOM. Within VOM, I can create notifcations and send notices to other Symantec systems to handle those. It seems, though, that VOM is my centralized server for handling events.
But I don't just have my Symantec services; I also have an F5 Big-IP load balancer, Solaris systems with ASR and consoles that complain about hardware. With all these events, there has to be a way to handle notifications so that I don't have to log into 20 different systems to manage this (e.g., adding someone to notifications; and yes, I know there are other ways to handle this). SNMP can handle notifcations and traps. It can also help you monitor and chart activity, but we'll focus on just the notifications (and some monitoring) for now.
In this example, I'm only going to focus on one event: a NetBackup failure event. I started out having the backup_exit_notify scripts in place to notify me when the backup failed/exited, but I wasn't getting enough information. I then created notifications from OpsCenter, (Manage -> Alert Policies ; Job Finalized). I like these notifications because I can filter them and I get more information about the failure (such as Policy Name, what the exit status means, etc). Using this policy, I can also only notify on the status of the last attempt, as backup_exit_notify gets called after each attempt. This way, if the first attempt fails, and I don't want to hear about it, I can use OpsCenter.
I created a policy called "Job Failed", selecting "Job Finalized" as my Alert Condition. As an administrator, I want to get notified for all job types, so I checked the box labeled "All" and kept "Include all exit status:" checked. I don't want to hear about successful jobs because I have about 200 or so of those a day, so I set "Exit Status to Exclude" at "0, 1". Now, the only thing left was to set my recipients (email, trap), severity, and activate the policy. I included my email in the email recipients so that I can be sure that the policy is working by getting an email and I'll remove it later. I put my SNMP host under the trap recipients, select the proper severity, activated the policy and I was good to go.
Finally, the SNMP server needs to know what the MIB looks like when it receives the notification. Symantec provides the MIB for OpsCenter, which is based on the Command Central MIB, and located in /opt/SYMCOpsCenterServer/config/snmp/VRTS-cc.mib. Symantec's VOM is located in /opt/VRTSsfmcs/config/snmp/VRTSsfm.mib. Because I use OpenNMS, I have to convert MIBs, so I'm not sure if this is so in the regular MIB format, but within the xml for OpenNMS, I did have to change the "generic" maskelement value to "6", instead of "0" ( 0 matches the Command Central notifications). Once installed on the network monitoring server (SNMP recipient), I configure notifications and whatever else I want to do with those. In OpenNMS, these create events, where I can create rules with what to do with those events (notify, set an alarm, etc).
If you use OpenNMS, like me, I will upload my XMLs to the download section. Download them and place them in your opennms's etc directory and put them in the eventconf.xml.