Video Screencast Help
Scheduled Maintenance: Symantec Connect is scheduled to be down Saturday, April 19 from 10am to 2pm Pacific Standard Time (GMT: 5pm to 9pm) for server migration and upgrades.
Please accept our apologies in advance for any inconvenience this might cause.

NS7 - System.OutOfMemoryException - Global Mutex Timed Out

Created: 09 Nov 2010 • Updated: 22 Nov 2010 | 7 comments
This issue has been solved. See solution.

 

Hey team,
 
Pretty regularly we are seeing System.OutOfMemoryException errors in our NS7 console.
 
We have a separate, 64-bit SQL server w/ 32GB of RAM and 8 cores that hosts only the NS database.
 
The NS7 server is an Enterprise Edition 2k3 server also w/ 32GB of RAM and 8 cores.
 
But still, every now and then, the w3wp.exe process hits around 1.2GB and then logs the below errors, and then restarts.
 
"10/11/2010 8:44:54 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/SoftwareManagementSolution-4-129337383420909185] Global mutex timed out for: 47b8423b7dea4e57ad2c228f36d13e5c","Altiris.NS.Server.GroupMessaging_Trace","w3wp","52"<
 
 
"10/11/2010 8:44:54 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AeXMonitor/Agent-9-129337383744667473] Global mutex timed out for: b0e86430e895480c863a3f45258ea1b3","Altiris.NS.Server.GroupMessaging_Trace","w3wp","132"
 
 
"10/11/2010 8:44:49 AM","[EventEngine.exe:EventEngine.exe] Global mutex timed out for: b0e86430e895480c863a3f45258ea1b3","Altiris.NS.Server.GroupMessaging_Trace","EventEngine","18"
 
 
"10/11/2010 8:44:42 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/ei.PowerAgent-13-129337384935479207] <_lm_w3svc_1_root_altiris_ei_70e345b777f143a28ddb7e78432fe087>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.Stream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.PushbackStream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","224"
 
 
"10/11/2010 8:44:40 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/Reporting-14-129337435070356377] <_lm_w3svc_1_root_altiris_reporting_14_129337435070356377_93d979614b674041ace84ca8455f43df>:<ReliableMMF> Packet processing failed. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.ProcessInboundPacket(Byte[] packet)
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","345"
 
 
"10/11/2010 8:44:40 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AeXMonitor-18-129337496937252629] <_lm_w3svc_1_root_altiris_aexmonitor_18_129337496937252629_eca26bf9583c4577bba4db6048cd65dc>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.Stream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.PushbackStream.ReadByte()
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","538"
 
 
"10/11/2010 8:44:36 AM","[w3wp.exe:/LM/W3SVC/1/Root/Altiris/AltirisNSAutomation-17-129337448975556091] <_lm_w3svc_1_root_altiris_altirisnsautomation_17_129337448975556091_59cf0bbe6ee84eed843467abe78ed889>:<ReliableMMF> Forced to close consumer stream: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Altiris.GroupCommunications.Protocols.ReliableMmfProtocol.MmfReceiverThreadRun()","Altiris.NS.Server.GroupMessaging_Trace","w3wp","19"

Has anyone ever experienced this? We are running v7.0.7416.

Comments 7 CommentsJump to latest comment

Ludovic Ferre's picture

... but I can see that the limitations you are currently hitting relate to the maximum amount of memory .Net can get within a single process.

I would suggest you implement custom application pools to isolate the various solutions running on your server to see which web-applications is consuming most memory.

I have written a blog entry on how to do this:

https://www-secure.symantec.com/connect/blogs/crea...

Once you have implemented a number of pools (I would suggest you start with the pools used in the blog) the problem may disappear or take longer to come back (simply because we split the memory consumption which gives the system some headroom and also makes a better use of available resources).

In all cases, let us know if this helps.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

SOLUTION
Rhys Paterson's picture

That's some really helpful information.

I'll give it a try in our test environment and see how we go.

Obviously it's different for everyone, but do you have any general suggestions as to which solutions would benefit the most from a separate application pool?

Cheers,

Rhys

Rhys Paterson's picture

We've got a heap of RAM on this box (although it is only Enterprise 32-bit) and it seems to cap around 1.1GB.

Is there any way to allow it to utilise more?

Ludovic Ferre's picture

There are a number of discussions and entries  in MSDN or Stackoverflow if you search for "max .net memory consumption".

One such thread of interested from social.MSDNl is here: http://social.msdn.microsoft.com/forums/en-US/clr/thread/b4839085-772b-4925-a0f6-2fe21096c376/

Depending on the objects allocated the max memory used by a .net process could be around 1.1~1.4 GiB.

If you want to see where the memory is gone they point to the .net profiler on the aforementioned discussion.

All in all we are back to the beginning: where does the memory goes in you w3wp process and is this nromal or not (memory leaks...)?

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Rhys Paterson's picture

I've moved Arellia, InventoryRuleManagement and PatchManagement solutions to different application pools and things are now running much better. We have multiple W3WP.exe processes and the web console is now running in a usable state!

The DefaultAppPool process is now sitting at around 755MB, with the two others at 210MB and 10MB. We're not really using Patch Management at the moment so I assume it's Arellia that's running at around 210MB.

I'd like to play around with this some more to find out which solutions consume the most memory (and then separate them) as it definitely makes a difference. I would recommend this to anyone who is having similar issues.

I'd also like to note that hunting for memory leaks shouldn't be something we need to deal with. But I do very much appreciate your help Ludivoc.

Cheers,

Rhys

Ludovic Ferre's picture

But I understand that it's hard finding this type of problems and solutions in the field.

I got some feedback internally on my blog post and findings, and the problem and solutions are now with the engineering team for review.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Rhys Paterson's picture

http://support.microsoft.com/kb/919790

According to MS: "an IIS 6.0, an IIS process uses up to 1.3 gigabytes (GB) of virtual memory."