Notification Server busy or overloaded
|Article:TECH14180|||||Created: 2005-09-19|||||Updated: 2008-09-24|||||Article URL http://www.symantec.com/docs/TECH14180|
Notification Server CPU utilization is pegged at 100 percent. Evtqueue and Evtqfast queues are filled with .TMP files.
Notification Server 6.0 SPx
The Notification Server cannot handle the workload either because of configuration issues on the server side, or poor client configuration that create an excess of work for the server.
One of the client computers with the Altiris Agent is causing the problem by sending in bad XML to the Notification Server which is in turn creating a huge number of temp files and and bad folders. Also, in some cases, antivirus software doing a "realtime" live scan on the "\program files\altiris\notification server\nscap\" folder and it's subdirectories slows the server down from processing files.
Verifying the Issue
The following items will help identify the extent of the issue:
- Go to the queues, found at <installpath>\Program Files\Altiris\Notification Server\NSCap\.
- How many files are in the Evtqueue folder?
- How many files are in the Evtqfast folder?
- Do you have subdirectories found at <installpath>\Program Files\Altiris\Notification Server\NSCap\Evtqueue\Bad\, and <installpath>\Program Files\Altiris\Notification Server\NSCap\Evtqfast\Bad\?
- What are the names of the subfolders, and how many files are under each folder?
General Items to Bring the Server under Control
First, Dissable Antivirus from scanning the following subdirectory and it's subdirectories:
"\program files\altiris\notification server\nscap\"
The number of threads running can make a significant impact on server performance. he following registry keys control the number of threads being used:
- customSetting key="MaxConcurrentFastMsgs" type="local" value="5"
- customSetting key="MaxConcurrentSlowMsgs" type="local" value="2"
- HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Notification server\MaxDispatchThreads REG_DWORD 5
Other settings that could be adjusted are
- SlowMsgDispatcherThrottleDelay = 1000
- Basic Inventory interval
There is a maximum amount of data that any given system can process. High file queue counts are to be expected under certain conditions (monthly full inventory, for example) but a constant full queue indicates that too many events are being sent to the server. Either reduce the number of agents reporting to that Notification Server, decrease the number of events those Agents are sending to the Notification Server, or increase the hardware specifications of the Notification Server itself.
The following items can be deactivated or scaled-back to provide relief for the server and an opportunity for the server to catch up.
- Disable Status Events on the Server, found under the Configuration tab, Server Settings, Notification Server Settings, and select Advanced Settings.
- If using Application Metering, disable AeX Application Start and AeX Application Stop events. This will lower traffic significantly. This will prevent the capturing of concurrent application usage, but will still allow the monthly trending data from app metering to be captured.
- Disable the 3 Software Delivery events (used for every package in the Notification Server, from core and all solutions). These are AeX SWD Execution (Keep Execution enabled, it is the best event type for troubleshooting agent problems), AeX SWD Package, and AeX SWD Status.
- Scale-back Configuration Requests events for the Altiris Agent. This is located under Configuration, Altiris Agent, Altiris Agent Configuration, and each individual policy. Under Agent Basic Settings, Change the Request new configuration information every: to 12 hours (This can be scaled back according to needs).
- Set Send Basic Inventory to every 2 days.
- If using full Inventory Solution, change the daily policies to run once every three days (twice a week) and stagger them so that no two inventory policies run at the same time.
- Change the Shared Schedules to run a different times at night. Many of them run at 2:00am. If they are spaced out, the server will be more capable of handling the even load.
- Deleting packages on the Notification Server. This generates undue traffic on the Notification Server. See article 22496, "Queues flooded with ‘Itemnotfoundexception,'" for more information.
Other Items for Consideration
- Patch Management—Disable all Bulletins not required. This will lower traffic in general.
- Package Servers—If you have a large number of package servers, either use site maintenance (found under Configuration, Server Settings, Notification Server Infrastructure), or remove those that are secondary or unnecessary.
Altiris Support Services can analyze large numbers of NSE files to see what items/task types are hitting the server the hardest.
Article URL http://www.symantec.com/docs/TECH14180