Symantec Management Platform (Notification Server)

"10/11/2010 8:44:54 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/SoftwareManagementSolution-4-129337383420909185] Global mutex timed out for: 47b8423b7dea4e57ad2c228f36d13e5c","Altiris.NS.Server.GroupMessaging_Trace","w3wp","52"<

 
 
"10/11/2010 8:44:54 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AeXMonitor/Agent-9-129337383744667473] Global mutex timed out for: b0e86430e895480c863a3f45258ea1b3","Altiris.NS.Server.GroupMessaging_Trace","w3wp","132"
 
 
"10/11/2010 8:44:49 AM","[EventEngine.exe:EventEngine.exe] Global mutex timed out for: b0e86430e895480c863a3f45258ea1b3","Altiris.NS.Server.GroupMessaging_Trace","EventEngine","18"
 
 
"10/11/2010 8:44:42 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/ei.PowerAgent-13-129337384935479207] <_lm_w3svc_1_root_altiris_ei_70e345b777f143a28ddb7e78432fe087>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.Stream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.PushbackStream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","224"
 
 
"10/11/2010 8:44:40 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/Reporting-14-129337435070356377] <_lm_w3svc_1_root_altiris_reporting_14_129337435070356377_93d979614b674041ace84ca8455f43df>:<ReliableMMF> Packet processing failed. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.ProcessInboundPacket(Byte[] packet)
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","345"
 
 
"10/11/2010 8:44:40 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AeXMonitor-18-129337496937252629] <_lm_w3svc_1_root_altiris_aexmonitor_18_129337496937252629_eca26bf9583c4577bba4db6048cd65dc>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.Stream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.PushbackStream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","538"
 
 
"10/11/2010 8:44:36 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AltirisNSAutomation-17-129337448975556091] <_lm_w3svc_1_root_altiris_altirisnsautomation_17_129337448975556091_59cf0bbe6ee84eed843467abe78ed889>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","19"

Has anyone ever experienced this? We are running v7.0.7416.

... but I can see that the limitations you are currently hitting relate to the maximum amount of memory .Net can get within a single process.

I would suggest you implement custom application pools to isolate the various solutions running on your server to see which web-applications is consuming most memory.

I have written a blog entry on how to do this:

https://www-secure.symantec.com/connect/blogs/creating-new-iis-application-pools-magic-solution-smp-7-slow-console-operations

Once you have implemented a number of pools (I would suggest you start with the pools used in the blog) the problem may disappear or take longer to come back (simply because we split the memory consumption which gives the system some headroom and also makes a better use of available resources).

In all cases, let us know if this helps.

That's some really helpful information.

I'll give it a try in our test environment and see how we go.

Obviously it's different for everyone, but do you have any general suggestions as to which solutions would benefit the most from a separate application pool?

Cheers,

Rhys

We've got a heap of RAM on this box (although it is only Enterprise 32-bit) and it seems to cap around 1.1GB.

Is there any way to allow it to utilise more?

There are a number of discussions and entries  in MSDN or Stackoverflow if you search for "max .net memory consumption".

One such thread of interested from social.MSDNl is here: http://social.msdn.microsoft.com/forums/en-US/clr/thread/b4839085-772b-4925-a0f6-2fe21096c376/

Depending on the objects allocated the max memory used by a .net process could be around 1.1~1.4 GiB.

If you want to see where the memory is gone they point to the .net profiler on the aforementioned discussion.

All in all we are back to the beginning: where does the memory goes in you w3wp process and is this nromal or not (memory leaks...)?

I've moved Arellia, InventoryRuleManagement and PatchManagement solutions to different application pools and things are now running much better. We have multiple W3WP.exe processes and the web console is now running in a usable state!

The DefaultAppPool process is now sitting at around 755MB, with the two others at 210MB and 10MB. We're not really using Patch Management at the moment so I assume it's Arellia that's running at around 210MB.

I'd like to play around with this some more to find out which solutions consume the most memory (and then separate them) as it definitely makes a difference. I would recommend this to anyone who is having similar issues.

I'd also like to note that hunting for memory leaks shouldn't be something we need to deal with. But I do very much appreciate your help Ludivoc.

Cheers,

Rhys

http://support.microsoft.com/kb/919790

According to MS: "an IIS 6.0, an IIS process uses up to 1.3 gigabytes (GB) of virtual memory."

But I understand that it's hard finding this type of problems and solutions in the field.

I got some feedback internally on my blog post and findings, and the problem and solutions are now with the engineering team for review.