Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Endpoint Management Community Blog

Getting deep down in the Altiris-NS-Agent w3wp process with WinDbg

Created: 05 Feb 2013 • Updated: 05 Feb 2013
Ludovic Ferre's picture
0 0 Votes
Login to vote

I have a customer that has some problems with one of three child servers, with some serious memory consumption there going from 2 GiB for TaskManagement, ~2.5GiB for AeXSvc and anything above that (sometimes 5GiB+) for the Altiris-NS-Agent.

This is causing some serious troubles obviously, as the CPU are all clogged up (95~100% on 24 real cores). So first we reduced the overall impact by limiting the number of CPU's the process is allowed to on (using advanced setting and processor affinity mask).

And then we took a few dumps to see how the memory gets up to 4GiB or more...

I'll detail the WinDbg commands (and how to load mscorewks via sos.dll) but here's the over view running DumpHeap -stat at various stages.

Note! Here are the field names used below in order, Memory address, object count, total size, object type. The listing is sorted by size ascending. So the interesting data is at the end.

0x000007ff004afdc0  683,955   54,716,400 Altiris.Common.Collections.Generic.Set`1[[System.Guid, mscorlib]]
0x000007ff006a2bb0 1,492,090   59,683,600 System.Collections.Generic.List`1[[Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+Codebase, Altiris.NS.StandardItems]]
0x000007fef8ccfce0  261,210   69,515,376 System.Byte[]
0x000007ff006aa630      312   77,468,544 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+CodebaseCollection, Altiris.NS.StandardItems]][]
0x000007ff004e5fb8  684,296   99,544,848 Altiris.Common.Collections.Generic.Set`1+Entry`1[[System.Guid, mscorlib],[System.Guid, mscorlib]][]
0x000007ff00565a88 2,933,741  140,819,568 Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+Codebase
0x000007fef8cb5a90 1,940,036  166,768,280 System.Object[]
0x000007ff006a48a8 2,932,309  211,126,248 Altiris.NS.StandardItems.SiteServer.PackageServer.DataAccessLayer.PackageCodebaseRow
0x0000000003070550   33,363  693,892,248      Free
0x000007fef8ccedc0  995,994  737,485,592 System.Int32[]
0x000007fef8cc7ca0 6,301,118 1,477,344,960 System.String
Total 23,854,409 objects, Total size: 4,219,133,592

Now here's a simpler view (of the same data), as the above is not necessarily a very digest read:     

Type Size Count
(total) 4,219,133,592 23,854,409
System.String 1,477,344,960 6,301,118
System.Int32[] 737,485,592 995,994
(free) 693,892,248 33,363
Altiris.NS....PackageServer.DataAccessLayer.PackageCodebaseRow 211,126,248 2,932,309
System.Object[] 166,768,280  1,940,036
Altiris.NS...PackageServer.PackageInfoProvider+Codebase 140,819,568 2,933,741
Altiris..Collections.Generic.Set`1+Entry`1[[Guid],[Guid]][] 99,544,848  684,296
System...Dictionary`2+Entry[[...],CodebaseCollection]][]  77,468,544 312
System...List`1[[PackageInfoProvider+Codebase]] 54,716,400 683,955

Now, we're researching what strings are taking 1GiB+ of memory, and some of the remaining information (like the PackageInfo related data structures that are totaling 6,550,317 objects for 461MiB)...