Client Management Suite

We are starting to see a wide number of our agents crash when they update configuration.  

We are upgrading to SP1 tonight so hoping an agent upgrade will help with this, but has anyone seen this happen?

This has seemed to start when we started to use SSL recently and implimented our CEM, could be just timing but not sure.  We are above 200+ agents with this issue now.

This is what I am seeing in the event logs

Log Name:      Application
Source:        Application Error
Date:          5/28/2014 9:46:23 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:     mypc
Description:
Faulting application name: AeXNSAgent.exe, version: 7.5.1673.0, time stamp: 0x52d7b637
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000374
Fault offset: 0x00000000000c4102
Faulting process id: 0x740
Faulting application start time: 0x01cf7a83941d8bc8
Faulting application path: C:\Program Files\Altiris\Altiris Agent\AeXNSAgent.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: d5e93f7a-e676-11e3-bd08-0027132f1745
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-05-28T14:46:23.000000000Z" />
    <EventRecordID>22508</EventRecordID>
    <Channel>Application</Channel>
    <Computer>2CE942HT59.us.grainger.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>AeXNSAgent.exe</Data>
    <Data>7.5.1673.0</Data>
    <Data>52d7b637</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7601.18247</Data>
    <Data>521eaf24</Data>
    <Data>c0000374</Data>
    <Data>00000000000c4102</Data>
    <Data>740</Data>
    <Data>01cf7a83941d8bc8</Data>
    <Data>C:\Program Files\Altiris\Altiris Agent\AeXNSAgent.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>d5e93f7a-e676-11e3-bd08-0027132f1745</Data>
  </EventData>
</Event>

You can try to use procdump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx and open the dump file to see what excaclty is failing.

Teh ntdll.dll is a Windows file. Did you install any updates recently ?

Since it is happening over 200+ clients I would open a case with Symantec Support.

I am upgrading our servers to SP1 as I type.  If the problem still persists after all the agents are upgraded I will be opening a case with Symantec.

As a test on a machine where it is crashing I changed the client to communicate with just HTTP vs HTTPS and now it no longer crashes   ( I don't have my server requiring SSL, it allows for either 80 or 443)

We are seeing the crashes on both CEM agents and Non CEM that are utilizing SSL.

I'll post here after SP1 gets mostly populated and let people know what my results are, they will either be fixed or open case with Symantec.

Hi SonicGT,

If you may upload dmp and logs to the NZ account that I have sent to you by private message then dev will take a look at the problem. Very similar issue has been resolved in 7.5.Sp1, but it's always  possibility what this is something different.

Thank you,

Alex.

SonicGT, could you please check if throttling is enabled in the agent policy? It is configured in Targeted Agent Setting -> Downloads -> Throttling Periods.

If throttling is ON, try turning it OFF and see if the problem goes away.

We fixed the very nasty bug in 7.5 SP1 when certain throttling values cause any HTTPS call to cause the heap corruption. It sounds like you hit this problem.

Throttling is enabled since we use it.  

I installed SP1 on our production system this weekend we have a 1x2 heirarchy

The agent upgrades are in progress now, so I will know in a day or two if the issue is still persisting.  I'll keep this thread updated one way or the other.  Hoping it was something like the throttling or other thing that may have been resolved with SP1.  I'll know more in a few days

How have your agents been behaving since your SP1 upgrade? We're in a precarious situation where we're seeing this issue a lot, but really need DS up and running for deployments we currently have going on. We'd have to know for a fact that things are better with SP1.

The agent itself is behaving better.  We have not had any reported crashes with the new agent.  Getting many of our older agents that were or are crashing upgraded appears to be an issue since many of them are sellers and use the CEM so we can't do any remediation on broken old agents without user contact to get them to a branch.

We are having other issues with SP1 (SWV bluescreens, Tasks during our image process timing out, and a couple minor security related changes)  Have tickets open with symantec on all of those things and are working through them.  But the agent itself crashing like this seems to be mitigated by the machines getting the new agent itself.

If it's not one thing it's another. Are the tasks that are timing out during the image process really messing with your ability to use DS?

We have to weigh our options. Stable agents vs functional imaging.

For us yes.  Not necissarily DS but our image process.  

Once in windows (post official DS tasks) we can't get our job to finish.  We have it set for 5 hours and are seeing timeouts so it just gets stuck in the middle of installing our apps.  Our normal image process takes 90 minutes.  We have no apps in our base ghost since we have aprox 30 different roles with different application sets and use image jobs to install the OS then all the role specific image applications.

Hi SonicGT,

Might it be that your DS job contains "Task Server Script task" within?

We have a problem with such type of tasks in 7.5.SP1 - they timed-out if running more than 5 minutes.

Thank you,

Alex.