Getting deep down in the Altiris-NS-Agent w3wp process with WinDbg
I have a customer that has some problems with one of three child servers, with some serious memory consumption there going from 2 GiB for TaskManagement, ~2.5GiB for AeXSvc and anything above that (sometimes 5GiB+) for the Altiris-NS-Agent.
This is causing some serious troubles obviously, as the CPU are all clogged up (95~100% on 24 real cores). So first we reduced the overall impact by limiting the number of CPU's the process is allowed to on (using advanced setting and processor affinity mask).
And then we took a few dumps to see how the memory gets up to 4GiB or more...
I'll detail the WinDbg commands (and how to load mscorewks via sos.dll) but here's the over view running DumpHeap -stat at various stages.
Note! Here are the field names used below in order, Memory address, object count, total size, object type. The listing is sorted by size ascending. So the interesting data is at the end.
0x000007ff004afdc0 683,955 54,716,400 Altiris.Common.Collections.Generic.Set`1[[System.Guid, mscorlib]] 0x000007ff006a2bb0 1,492,090 59,683,600 System.Collections.Generic.List`1[[Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+Codebase, Altiris.NS.StandardItems]] 0x000007fef8ccfce0 261,210 69,515,376 System.Byte[] 0x000007ff006aa630 312 77,468,544 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+CodebaseCollection, Altiris.NS.StandardItems]][] 0x000007ff004e5fb8 684,296 99,544,848 Altiris.Common.Collections.Generic.Set`1+Entry`1[[System.Guid, mscorlib],[System.Guid, mscorlib]][] 0x000007ff00565a88 2,933,741 140,819,568 Altiris.NS.StandardItems.SiteServer.PackageServer.PackageInfoProvider+Codebase 0x000007fef8cb5a90 1,940,036 166,768,280 System.Object[] 0x000007ff006a48a8 2,932,309 211,126,248 Altiris.NS.StandardItems.SiteServer.PackageServer.DataAccessLayer.PackageCodebaseRow 0x0000000003070550 33,363 693,892,248 Free 0x000007fef8ccedc0 995,994 737,485,592 System.Int32[] 0x000007fef8cc7ca0 6,301,118 1,477,344,960 System.String Total 23,854,409 objects, Total size: 4,219,133,592
Now here's a simpler view (of the same data), as the above is not necessarily a very digest read:
| Type | Size | Count |
| (total) | 4,219,133,592 | 23,854,409 |
| System.String | 1,477,344,960 | 6,301,118 |
| System.Int32[] | 737,485,592 | 995,994 |
| (free) | 693,892,248 | 33,363 |
| Altiris.NS....PackageServer.DataAccessLayer.PackageCodebaseRow | 211,126,248 | 2,932,309 |
| System.Object[] | 166,768,280 | 1,940,036 |
| Altiris.NS...PackageServer.PackageInfoProvider+Codebase | 140,819,568 | 2,933,741 |
| Altiris..Collections.Generic.Set`1+Entry`1[[Guid],[Guid]][] | 99,544,848 | 684,296 |
| System...Dictionary`2+Entry[[...],CodebaseCollection]][] | 77,468,544 | 312 |
| System...List`1[[PackageInfoProvider+Codebase]] | 54,716,400 | 683,955 |
Now, we're researching what strings are taking 1GiB+ of memory, and some of the remaining information (like the PackageInfo related data structures that are totaling 6,550,317 objects for 461MiB)...
The Endpoint Management Community Blog is the perfect place to share short, timely insights including product tips, news and other information relevant to the Endpoint Management community. Any authenticated Connect member can contribute to this blog.