How to increase Queue processing performance in NS 7.1 SP2 (Queue was Full)

Article:TECH183347  |  Created: 2012-03-08  |  Updated: 2014-12-12  |  Article URL http://www.symantec.com/docs/TECH183347
Article Type
Technical Solution


Environment

Issue



Logs full of "Queue was full" warnings.


Error



Module: w3wp.exe
Source: Altiris.NS.EventRouter.ProcessAgentHttpPostEvent
Description: Event not processed. IP: -.-.-.-; Error description: Altiris.NS.EventRouter+ServerBusyException: Queue was full.
at Altiris.NS.EventRouter.<>c__DisplayClass3.<RegisterDelivery>b__0(IDatabaseContext ctx)
at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry)
at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, IsolationLevel isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)
at Altiris.NS.EventRouter.RegisterDelivery(Boolean fileInLine, String tempFile, Int64 fileSize, Guid source, Int32 priority)
at Altiris.NS.EventRouter.DeliverStream(Stream input, Guid source, Int32 priority)
at Altiris.NS.EventRouter.DeliverPostedDataImpl()
at Altiris.NS.EventRouter.ProcessAgentHttpPostEvent()

 


Environment



SMP 7.1 SP2


Solution



1. Stop the Altiris services:

a. Altiris Support Service
b. Altiris Client Message Dispatcher
c. Altiris File Receiver
d. Altiris Event Receiver
e. Altiris Service
f. Altiris Object Host Service
g. Altiris Service Hoster
h. Altiris Event Engine
 

2. Stop IIS (W3WP)
 

3. Move NSE out of eventqueue (C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue) and into a temp folder.
 

4. If a larger queue size is desired, make a change to the Windows Registry "HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Notification Server" Increase the "MaxFileQSize(KB)" to the new desired queue size.  Default is 512000. SP2 design can benefit from larger queue sizes of between 2 to 4 GB of harddisk space, a value 4000000 is acceptable.
 

5. Run the SQL in the two files attached to this article (spMapMergedResourceWithIntentToMerge.sql, spResourceDelete.sql) against the Symantec_CMDB database in order to alter them for efficiency.
Note: This step can be skipped if the server is at 7.1 SP2 MP1 or later. 

NOTE: These store procedures do not need to be installed if MP1/MP1.1 is installed. If the stored procedures were applied after MP1 was installed and before 7/17/2013, please reapply the attached stored procedures . The old stored procedure 'spMapMergedResourceWithIntentToMerge.sql' was missing code that would prevent resource merging from occurring. The currently attached file has the correct code.
 

6. Consider increasing these coresettings threadcounts and queue thresholds. To determine if changing the default configuration will be helpful, run this select query in SQL to see how many concurrent NSE files are being processed.

SELECT 
case
when QueueId = 0 then 'Priority (0)'
when QueueId = 1 then 'Fast     (1)'
when QueueId = 2 then 'Normal   (2)'
when QueueId = 3 then 'Slow     (3)'
when QueueId = 4 then 'Large    (4)'
end AS [QUEUE ID]
,COUNT(Id) AS [THREADS IN USE] 
FROM EventQueueEntry where ProcessingState = 2 
GROUP BY ProcessingState, QueueId order by queueid, ProcessingState

Expect to see around 10 or more threads working on a full queue.  If the results of this select statement show 1-4 threads running concurrently and there are plenty of NSE files to process, try adjusting these thresholds in the "C:\ProgramData\Symantec\SMP\Settings\coresettings.config" file to increase the threads available for each queue and balance the load across more queues by modifying these thresholds to the values below:

<customSetting key="FastQueueThreshold" type="local" value="50000" />   (default is 15,000)
<customSetting key="SlowQueueThreshold" type="local" value="3000000" /> (default is 1,048,576)
<customSetting key="MaxConcurrentFastMsgsThreadPoolSize" type="local" value="20" /> (default is 10)
<customSetting key="MaxConcurrentSlowMsgsThreadPoolSize" type="local" value="10" /> (default is 3)

In SMP 7.1 SP2, Increasing the MaxConcurrentPriorityMsgsThreadPoolSize setting will give zero performance gain.  Since more threads are allocated but are never assigned work, increasing this setting can lead to slightly lower performance.

The LargeQueueThreshold does not need to be modified.

Notes: You must do steps 7 and 10 for this threshold settings to have effect on your NSE backlog.  Coresettings does not have a value for MaxConcurrentLargeMsgsThreadPoolSize.  It is hardcoded to 1 thread. 
 

7. Clearing these event and merge tables with this SQL:

truncate table EventQueue
truncate table EventQueueEntry
 

8. Restart the SMP services:

a. Altiris Support Service
b. Altiris Client Message Dispatcher
c. Altiris File Receiver
d. Altiris Event Receiver
e. Altiris Service
f. Altiris Object Host Service
g. Altiris Service Hoster
h. Altiris Event Engine
 

9. Start IIS (W3WP)
 

10. Move saved NSEs to the EvtInbox or delete them depending on retention requirements.
 

11. Make an Antivirus exclusion rule for the .nse text files or the EvtQueue directory. 


Note:  NS trace logging adds overhead to NSE processing.  Please turn trace logging off during normal usage.  Trace logging is off by default but can be changed in the SMP console under Settings>All Settings>Nofication Server>Nofication Server Settings>Logging.

 

For similar information on ITMS 7.5, please refer to TECH211788 "How to increase Queue processing performance in SMP 7.5"


Supplemental Materials

SourceETrack
Value2661853,2895624,2704466



Article URL http://www.symantec.com/docs/TECH183347


Terms of use for this information are found in Legal Notices