Symantec Management Platform (Notification Server)

 View Only
Expand all | Collapse all

EventQueue SMP v7.1

  • 1.  EventQueue SMP v7.1

    Posted Jul 18, 2011 01:39 PM

    My EventQueue seems to have gotten messed up. Some .NSE files aren't going to the proper location/EvtQxxxx folder. In trying to troubleshoot extreme slowness, I found that I have thousands of .NSE files (239,076 and growing, to be exact) directly in my EventQueue folder. Shouldn't they be in one of the EvtQxxxxx folders to be processed? They are all sizes, from 8MB down to 1KB. I know some .NSE files are being processed, but these obviously aren't. What can I do?



  • 2.  RE: EventQueue SMP v7.1

    Posted Jul 18, 2011 06:36 PM

    If some .NSE's are being processed, but the EvtQueue is this full, the most likely cause is performance.  What does CPU look like on your NS?

    How often are you running a full inventory?  If it's more frequently than monthly, I would change your inventory schedule to monthly as the first step.

    How many nodes in your environment?  How many NS's?  How many site servers (and what services)?  Task interval used by clients?  Policy interval used by clients?

    Are you seeing anything at the Error or Warning level in the Altiris Log Viewer on the server?

    Is EvtQueueFast processing?

    How is SQL server performance?  If you aren't the DBA, buy your DBA a beer and ask him or her.  Any SQL deadlocks reported in the Altiris Log Viewer?

    Essentially what we're looking at doing is increasing performance (SQL or NS) or reducing load (task interval, policy interval, inventory and patch schedules, site server deployment for offloading).

    A lot of things to look at, I know, but some of them will hold the answers for us.



  • 3.  RE: EventQueue SMP v7.1

    Posted Jul 20, 2011 09:36 AM

    CPU was fine until this started happening. When this started, it pegged the CPU most of the time.

    Full Inventory is set to Monthly, deltas set to daily, configuration requests set to hourly. We have about 1400 clients in 6 locations. One NS, with 7 Site Servers, one in each location with an extra Task Server at HQ to take some of the load off the primary NS.

    I'm not sure about policy processing or task interval. The website has quit working now, so I can't check. I've had to shutdown all the services as well because this is overloading the server.

    SQL performance is normally good, but is now overloaded with all this inventory processing.

    Errors and warnings aplenty in the log viewer. So many I don't know where to start looking. Most say inventory file couldn't be processed for various reasons, deadlock being one of the reasons. Another being "the data class table cannot be empty". Doesn't say which one, but it can't be empty. I take that back. Two errors before that it says data class 986fa1db-628f-412a-9ed9-bf105816a3af can't be empty. I searched the DataClass table, and several others, and that GUID doesn't exist. Whatever happened to names? :-)

    Everything appeared to be working fine until Monday, or so it seemed. Performance was a little slow, so it could have just been building up to this, or maybe I tweaked something wrong. My guess is I may have done something with policy processing or task intervals. I was poking around in the settings trying to improve performance, and may have set something wrong. Can't check now unless there is somewhere in the database I can check.

    NSEs were not being processed until I moved the top level EventQueue folder over to the D: drive where it belongs(by stopping the Altiris Service and editing the registry). However, the other EvtQxxxx folders on the C: drive still get the files once EventQueue processes them. I'd like to get them moved over to the D: drive as well - without using any external utilities.

    All EvtQxxxx folders were being processed until this morning when I stopped all Altiris services. I did this because the "bad" folders filled up the C: drive until there was no room left.



  • 4.  RE: EventQueue SMP v7.1
    Best Answer

    Posted Jul 20, 2011 10:48 AM

    Well, certainly this is why you want to have some sort of change record, so that you can tell what changed, and when, so that when something like this happens you can back up to see what it might have been.

    Did you use this process to move the Evt folders to a drive other than C?

    http://www.symantec.com/business/support/index?page=content&id=TECH161370&actp=search&viewlocale=en_US&searchid=1311172906984

    I'd recommend the following:

    • Delete all .NSE's in Bad folders
    • Repair SMP
    • Use the approved process to move the Evt folders to a non-C drive (so that the OS volume doesn't fill up)
    • Start any stopped Altiris/Symantec services
    • When you regain console access, reduce delta inventory schedule to weekly to alleviate load on your NS

    If your NS is not processing NSE's but your clients are still uploading them, you're certainly going to have a processing nightmare.  Consider deleting all NSE's, even valid ones, in favor of the current ones they'll upload once your server is brought online.  And remember to buy your DBA a beer and ask about potential SQL tuning issues.  The deadlock messages almost certainly indicate a performance issue with SQL.



  • 5.  RE: EventQueue SMP v7.1

    Posted Jul 20, 2011 11:53 AM

    That "approved" process is not approved by us. No external utilities to redirect folders in the background. That's like using the SUBST command back in the DOS days.

    Been deleting NSEs that don't process. A lot did, but most didn't. With the services stopped, that shouldn't be a problem now.

    I'll try the repair/apply SP1 to try and get things back to normal.

    Deadlock messages weren't a problem until there were so many NSEs to process. Once we find out why so many NSEs are being generated, I'm hoping that the deadlock messages will go away.



  • 6.  RE: EventQueue SMP v7.1

    Posted Jul 20, 2011 12:24 PM

    Microsoft tools are external tools in your environment?

    It's definitely something to investigate: are the deadlock errors (which can point to a SQL performance issue) a result of slow processing of NSE's (because of a SQL performance issue)?  Or are the deadlock errors (which can point to a SQL performance issue) the result of an abnormal number of NSE's (because the NS is improperly tuned and agents have been asked to send back a ridiculous number of NSE's)?

    Keep us updated.



  • 7.  RE: EventQueue SMP v7.1

    Posted Jul 20, 2011 01:24 PM

    External was a poor choice of words. What we don't like is workarounds for poor application design. I wouldn't have used the SUBST command in DOS, and don't like the idea of having to use Junction in Windows to compensate for poor application design.

    SQL performance was seemingly good prior to this happening. We don't have the luxury of a SQL farm, so we use what we have. What we have is a pretty beefy server with SQL already on it, to which we added the Symantec CMDB. Quad processors, 32GB RAM, Server 2008 R2, and 64-bit SQL Server 2008. I don't know what the IOPS is supposed to be for that server, and I've read some of the articles here that IOPS is pretty important, but sometimes you have to make do with what you have and suffer the consequences of slow performance.

    I'll update as soon as I can get SMP repaired.



  • 8.  RE: EventQueue SMP v7.1

    Posted Jul 20, 2011 01:50 PM

    Sounds good.  If you have spare time while the SMP repair runs, head out to the KnowledgeBase and search for SQL tuning within the SMP product family.  This should return several SQL tuning and SQL optimization and SQL performance results which could help you determine what could be improved from a SQL performance side.  Good to check out even when things are working properly.



  • 9.  RE: EventQueue SMP v7.1

    Posted Jul 21, 2011 01:33 PM

    Running a repair now.

    Finally got the number of NSEs under control. Inventories were already set to the recommended levels. Task Interval and Policy Interval were set to daily.

    Once I slowed down the number of NSEs appearing, I began investigating why there were so many bad ones. Still don't know for sure. However, for one I did find a kb that said to reduce the number of reported events - kinda defeats the purpose of having SWD events, but I removed those.

    I'm guessing that the upgrade/migration didn't go very well for some clients.

    Most of my deadlock NSEs are from LegacyInventory. These agents are reporting to the new server, but are not upgrading for some unknown reason. Guess I'll have to check them out one by one. Unfortunately, only the GUID is displayed in the NSE, so it will take some time to research. I hate GUIDs, or rather, the way some vendors use GUIDs.

    Some of the deadlock NSEs are from Active Directory imports. Why I'm getting so many NSEs for that, I don't know. My AD imports are set to run hourly for deltas and monthly for full imports. After checking several, I found that most were from the same import rule. They are all from last month, too. Is that import file stored on the SMP server somewhere? As soon as the repair completes, I'm going to delete and then re-create that import rule.



  • 10.  RE: EventQueue SMP v7.1

    Posted Jul 21, 2011 03:20 PM

    Repair got stuck on step 2 for 1-1/2 hours and then failed. I'm going to try again in the morning.



  • 11.  RE: EventQueue SMP v7.1

    Posted Jul 22, 2011 03:02 PM

    Repairing the SMP and reducing the amount of data appears to have worked. Deadlocks were partly from "LegacyInventory" and the amount of data being processed. Even though I haven't changed the AD import settings, those NSEs have stopped appearing. No idea where they came from either.

    Thanks for your help.