Client Management Suite

Hi All

Please forgive this if it doesn't have all the info, first post.  We are runing CMS 7.1 sp2.  The guy who used to manage it left the company at the beginning of the year and it has been left to stagnate.  Long story short we are starting to see problems with different part of the system and we have traced them back to inventory not being updated correctly.  We initially found @ 50k NSE files in the event queue, when asked to look into this.  tried pushing some back through but were getting more erors than success so took the hit and decided to leave them unprocessed so we had a clean slate.

We are now seeing loads of Warning\errors same as in the link below.  but the suggested settings have already been implemented and our DBA assures me the DB tables are clear.  I am seeing no files in the EvtQueue folder to process but streams of the warnings going through log viewer.

http://www.symantec.com/business/support/index?page=content&id=TECH122190&actp=search&viewlocale=en_US&searchid=1287083339917

Any ideas where these failed entries could be stored? and how to clear them.  Half of the IP addresses in the warnings cannot be pinged and the names resolved by them are not in the CMS

I have set up folders to catch failed bad NSE files but only seeing couple of entries that are not related.

Any help would be appreciated

Hi MW67,

kindly share the log file so that we can provide some resolution.

Thanks,
Siva

Hi Siva

Thanks for reponding, here is the log entry as requested.  I am seeing several of these a minute.  Several IP addresses are reapeated constantly but there are plenty that are not repeats.

Log File Name: C:\ProgramData\Symantec\SMP\Logs\a.log
Priority: 2
Date: 06/08/2014 16:02:47
Tick Count: -1159665786
Host Name: CMS-01
Process: w3wp (22876)
Thread ID: 160
Module: w3wp.exe
Source: Altiris.NS.EventRouter.ProcessAgentHttpPostEvent
Description: Event not processed. IP: 10.xxx.xxx.160; Error description: Altiris.NS.EventRouter+ServerBusyException: Queue was full.
   at Altiris.NS.EventRouter.<>c__DisplayClass3.<RegisterDelivery>b__0(IDatabaseContext ctx)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, IsolationLevel isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)
   at Altiris.NS.EventRouter.RegisterDelivery(Boolean fileInLine, String tempFile, Int64 fileSize, Guid source, Int32 priority)
   at Altiris.NS.EventRouter.DeliverStream(Stream input, Guid source, Int32 priority)
   at Altiris.NS.EventRouter.DeliverPostedDataImpl()
   at Altiris.NS.EventRouter.ProcessAgentHttpPostEvent()

Hi,

Try this out --> https://www-secure.symantec.com/connect/forums/queue-was-full-cms-71-sp-2

Yes,  Saw that one.  The SQL queries in the reply marked as the solution don't point to anything so we are looking at the response below the one marked as the solution and going to try the DB repair.  Not sure how long that will take though?

Still totally confused by the error though as the event queue is empty most of the time so where the queue full part is coming from I don't know.

1. Are these 'bad' folders also are empty on NS server?

• \\localhost\NSCap\EvtInbox\Bad\
• \EvtQFast\
• \EvtQLarge\
• \EvtQPriority\
• \EvtQSlow\
• \EvtQueue\

C:\ProgramData\Symantec\SMP\EventQueue\

2. How often these errors appears in Log? After some time or non-stop?

3. Did you try to restart Altiris Services?

Hi MW67,

You will be able to see eventqueue in two different places,

1. NScap
2. C:\ProgramData\Symantec\SMP\EventQueue\

You have to confirm where you are seeing as empty.

Hi All

Firstly thanks for all the responses.  Secondly let me appologise the link in the original post was incorrect and should have been http://www.symantec.com/business/support/index?page=content&id=TECH183347

I have checked all legacy and default event\bad\process folders and they were all empty.  Due to the number of NSE files sat in the EvtQueue folder, when I started looking at this, I moved the location to a non OS partition and added a seperate location for BAD files.  These folders were also empty

The errors\warnings we were seeing were constant with 20 -30 per minute.  We would also see numerous files remain in EvtQueue not being processed.  I did another service restart this morning and as soon as the services were back up just under 600 nse files dropped into EvtQueue I waited a while and the Queue got over 750 without anything processing so I moved these to temp folder and started dropping these back in through the EvtInbox 10 at a time and they all processed without any errors.

Been monitoring the event log all morning and although the odd deadlock error goes thorugh I am no longer getting the warnings.  For the first time since looking at this the event queue is processing all files that enter it.  I have been coming into hundreds of files that have failed to process over night.  I have even seen the queue shoot up to 150 files and all get processed 

Hopefully this means the system is working as it should be and the issues we were having, that caused me to check things in the first place, should start to clear. 

Again thanks for all the help

There were significant changes to queue processing in 7.1 SP2.   All NSE events are now registered in the database.  Smaller NSEs never go into the disk queue, and instead are queued directly in SQL.   If you have manually manipulated the disk queues you may have gotten out of sync with the SQL queue table.  That could explain why the disk queues are empty but the server is reporting the queue full.

More reading:

Event Queues in ITMS 7.1 SP2
http://www.symantec.com/business/support/index?page=content&id=DOC5480

Q/A regarding changes in Event Queues processing in SMP 7.1 SP2
http://www.symantec.com/business/support/index?page=content&id=HOWTO65758

Note, to see the NSEs in SQL, check the EventQueueEntry table.  select * from EventQueueEntry

Where you see DataInLine = 0, it will show the name of the corresponding NSE file.  Where DataInLine = 1 you will see the content of the NSE instead and there is no file.  

Hi Joe

Thanks for that.  Makes a lot of sense.  I guess with the service restart this morning the DB queue would have been resynched.  Only explanation for everything starting to work as expected.